SpectralPlasmaSolver: a Spectral Code for Multiscale Simulations of Collisionless, Magnetized Plasmas

Transcription

1 Journal of Phyic: Conference Serie PAPER OPEN ACCESS SpectralPlamaSolver: a Spectral Code for Multicale Simulation of Colliionle, Magnetized Plama To cite thi article: Juri Vencel et al 2016 J. Phy.: Conf. Ser View the article online for update and enhancement. Related content - Multicale imulation of ion beam impact on a graphene urface K B Dybypayeva, A Zhuldaov, A Ainabayev et al. - Multicale imulation of thermal diruption in reitance witching proce in amorphou carbon A M Popov, G N Shumkin and N G Nikihin - Multicale Simulation of Porou Ceramic Baed on Movable Cellular Automaton Method A Smolin, I Smolin, G Eremina et al. Recent citation - Dicontinuou Galerkin algorithm for fully kinetic plama J. Juno et al - Convergence of Spectral Dicretization of the Vlaov--Poion Sytem G. Manzini et al - Stefano Markidi et al Thi content wa downloaded from IP addre on 03/01/2019 at 10:38

2 SpectralPlamaSolver: a Spectral Code for Multicale Simulation of Colliionle, Magnetized Plama Juri Vencel 1, Gian Luca Delzanno 1, Gianmarco Manzini 1, Stefano Markidi 2, Ivy Bo Peng 2, Vadim Royterhteyn 3 1 Lo Alamo National Laboratory, Lo Alamo, NM, USA 2 KTH Royal Intitute of Technology, Stockholm, Sweden 3 Space Science Intitute, Boulder, CO, USA {vencel,delzanno,manzini}@lanl.gov,{markidi, bopeng}@kth.e, vroyterhteyn@pacecience.org Abtract. We preent the deign and implementation of a pectral code, called SpectralPlamaSolver (SPS, for the olution of the multi-dimenional Vlaov-Maxwell equation. The method i baed on a Hermite-Fourier decompoition of the particle ditribution function. The code i written in Fortran and ue the PETSc library for olving the non-linear equation and preconditioning and the FFTW library for the convolution. SPS i parallelized for haredmemory machine uing OpenMP. A a verification example, we dicu imulation of the two-dimenional Orzag-Tang vortex problem and uccefully compare them againt a fully kinetic Particle-In-Cell imulation. An aement of the performance of the code i preented, howing a ignificant improvement in the code running-time achieved by preconditioning, while trong caling tet how a factor of 10 peed-up uing 16 thread. 1. Introduction Colliionle, magnetized plama are ubiquitou in a variety of laboratory, pace and atrophyical environment. They are decribed by kinetic theory and are characterized by highdimenionality, ince the particle ditribution function i defined in the ix-dimenional phae pace obtained by the three patial and three velocity coordinate, and trong aniotropy, ince tranport propertie can be very different along and acro the magnetic field [1]. In addition, colliionle plama are characterized by extreme eparation of patial and temporal cale. Thi occur already at the kinetic level due to the large ma difference between electron and ion. However, the cale eparation widen even further if one compare the kinetic cale with typical macrocopic cale of the ytem: everal order of magnitude in patial and temporal cale eparation are common for intance in magnetic fuion energy experiment or in the Earth magnetophere. Even with the mot advanced upercomputer, fully-kinetic imulation of thee ytem are unfeaible today and in the foreeeable future. On the other hand, fluid model treat the plama a a macrocopic fluid and are therefore uitable for ytem-cale imulation. They are obtained from kinetic theory by taking moment of the ditribution function. A kinetic cloure i neceary to cloe the moment equation but it can be rigorouly jutified only in very retrictive limit [2], which are often not met in practice. Indeed, performing ytem-cale imulation of colliionle (or weakly colliional plama that properly include the kinetic/microcopic phyic i a major challenge of computational plama Content from thi work may be ued under the term of the Creative Common Attribution 3.0 licence. Any further ditribution of thi work mut maintain attribution to the author( and the title of the work, journal citation and DOI. Publihed under licence by Ltd 1

3 phyic. One of the recent approache to thi problem i to embed a kinetic olver in a mall part of the computational domain of a fluid imulation [3, 4, 5]. The mot common approache for the numerical olution of the kinetic equation handle the dicretization of phae pace by either introducing macro-particle (a in the Particle-In-Cell (PIC approach [6, 7] or a computational grid (Eulerian-Vlaov [8, 9] and therefore treat the full ditribution function everywhere in the computational domain. Adaptive meh refinement technique can be ued effectively to reduce the number of degree of freedom for thee kinetic olver [10, 11, 12]. A third approach i pectral, namely the velocity part of the ditribution function i expanded in bai function (typically Hermite or Fourier [13]. In certain cae, uch a for a proper Hermite bai [14, 15] or the Legendre bai [16], it can be hown that the low-order moment of the expanion correpond to the fluid moment and the kinetic phyic i recovered by imply adding more moment to the expanion. Thu, for a uitable pectral bai the fluid-kinetic coupling i built-in. In recent year there ha been a renewed interet on pectral method for the kinetic equation [15, 17, 18, 19, 20, 21]. A comparion between the one-dimenional, electrotatic, Hermite pectral method and PIC on tandard kinetic theory problem uch a Landau damping, twotream intabilitie and ion acoutic wave wa preented in Ref. [19], where it wa concluded that, at leat on thee problem, the pectral method wa order of magnitude more accurate or fater than PIC. Vencel et al. [15] ued the ame method and performed kinetic imulation where the number of Hermite mode could be changed dynamically at run-time, thu providing the tranition between fluid and kinetic regime. An extenion of the pectral method to the multi-dimenional, fully electromagnetic cae i preented in [20], where exact conervation law in the dicrete are derived and ome preconditioning technique are preented to peed-up the convergence of the Newton-Krylov olver adopted for the nonlinear equation. Thee previou work alo point out that the optimization of the Hermite bai (in order to reduce the number of mode and improve the computation time and robutne of the method, parallelization and efficient preconditioning technique are critical need for thee method. In thi paper we preent the Hermite-baed pectral olver, called SpectralPlamaSolver (SPS, for the multi-dimenional Vlaov-Maxwell equation and dicu the deign and implementation of the code. SPS i written in Fortran and parallelized with OpenMP for hared-memory ytem. It ue open-ource librarie uch a PETSc [22] for nonlinear olver and preconditioning and FFTW [27] for the convolution. The paper i organized a follow. Section 2 preent the governing equation and the Hermite- Fourier method. Section 3 dicue the deign and implementation of the SPS code. Section 4 focue on the application of the method to a two-dimenional Orzag-Tang vortex problem, often ued a a paradigm to tudy plama turbulence. Scaling tet on a hared-memory ytem are preented in Section 5, while concluion are drawn in Section The Hermite-Fourier Spectral Method We model a colliionle, magnetized plama by olving the Vlaov-Maxwell equation. We conider a Carteian ytem of coordinate with the patial domain defined by [0, L x ] [0, L y ] [0, L z ] (L x, L y, L z are the domain length in each patial direction and aume periodic boundary condition. We ue the following normalization: time i normalized to the electron plama frequency ω pe = e 2 n 0 ε 0 m e (where e i the elementary charge, m e i the electron ma, ε 0 i vacuum permittivity, and n 0 i a reference denity, velocity to the peed of light c, the patial coordinate to the electron inertial length d e = c/ω pe, the magnetic field to a reference magnetic field B 0, and the electric field to cb 0. The ditribution function f i normalized a fc 3 /n 0. 2

4 Accordingly, the dimenionle Vlaov equation read: f t + v xf + q ω c (E + v B v f = 0, (1 e ω pe where label the plama pecie ( = e, i for electron and ion, the cyclotron frequency i ω c = q B 0 /m and E (B i the electric (magnetic field. Equation (1 i coupled to Maxwell equation via the electromagnetic field: B t = E, E E = ω pe ω ce ( + t = B ω pe f i dv + ω ce ( + vf i dv + vf e dv, (2 f e dv, B = 0. (3 We expand the particle ditribution function in Hermite bai function, f = N n 1 n=0 N m 1 m=0 N p 1 p=0 C n,m,p(x, tψ n (ξ xψ m (ξ yψ p (ξ z (4 (where N n, N m and N p are the total number of Hermite mode in the v x, v y and v z direction, repectively, to obtain a et of non-linear partial differential equation (PDE for the coefficient of the expanion. We conider the aymmetrically-weigthed (AW bai function, Ψ n (x = (π2 n n! 1/2 H n (xe x2, where H n i the Hermite polynomial of the n-th order [23]. The /αβ, where u β and α β are (contant argument of the bai function i defined a ξβ (v = β u β hift and caling parameter that have to be upplied by the uer and β = x, y, z. We further decompoe the patial part of the particle ditribution function into Fourier bai function, C n,m,p(x, t = N x/2 N y/2 N z/2 k x= N x/2 k y= N y/2 k z= N z/2 [ ( Cn,m,p kx,ky,kz, kx x (t exp 2πi + k yy + k ] zz, (5 L x L y L z where N β + 1, β = x, y, z, i the total number of Fourier mode in each patial direction. Uing the orthogonality propertie of the Hermite and Fourier bai function, the Vlaov- Maxwell equation can be rewritten a a et of ordinary differential equation (ODE [20]: dc kx,ky,kz, n,m,p dt 2πik y αy L y 2πik z αz L z q e ω c ω pe q ω c e ω pe = 2πik x ( m ( p + 1 L x α x ( n m C kx,ky,kz, n,m+1,p + n,m,p Ckx,ky,kz, { 2n [E x C n 1,m,p ] kx,ky,k z + α x [ B x (m + 1p α y α z { ( mp α z αy α y αz C n,m+1,p 1 + 2m u z α y C kx,ky,kz, n n+1,m,p + 2 Ckx,ky,kz, n 1,m,p 2 Ckx,ky,kz, n,m 1,p p 2 Ckx,ky,kz, n,m,p 1 2m α y + u y αy Cn,m,p kx,ky,kz, + u z αz Cn,m,p kx,ky,kz, + u x αx + C kx,ky,kz, n,m,p } 2p [E y C n,m 1,p ] kx,ky,k z + [E z C n,m,p 1 ] kx,ky,k z + α z Cn,m 1,p 1 + m(p + 1 α z αy Cn,m 1,p+1 }] Cn,m 1,p 2p u y αz Cn,m,p 1 k x,k y,k z + 3

5 q ω c e ω pe [ { ( np α B y x αz α z αx α n(p + 1 z Cn 1,m,p+1 + 2p u x q ω c e ω pe α x [ B z (n + 1m α x α y { nm ( α y α x for the Vlaov equation, and α x α y C n 1,m,p 1 + (n + 1p α x α z α z C n+1,m 1,p + 2n u y α x Cn,m,p 1 2n u z αx Cn 1,m,p C n+1,m,p 1 }] k x,k y,k z + Cn 1,m 1,p + n(m + 1 α y αx Cn 1,m+1,p }] Cn 1,m,p 2m u x αy Cn,m 1,p k x,k y,k z, (6 db kx,ky,kz δ dt de kx,ky,kz δ dt = 2πiε βγδ k β L β E kx,ky,kz γ, k β = 2πiε βγδ Bγ kx,ky,kz ω pe L β ω ce =e, i ( q α e α xαyα z 2 δ C kx,ky,kz, a,b,c + u δ Ckx,ky,kz, 0,0,0 (7 for Maxwell equation (written in index notation with ummation over repeated indice implied. Here ε βγδ i the Levi-Civita tenor and (a, b, c = (1, 0, 0, (0, 1, 0, (0, 0, 1 for δ = x, y, z, repectively. In Eq. (6, the convolution operator i defined a [H C n,m,p ] kx,k y,k z = N x/2 N y/2 N z/2 H kx k x,ky k y,kz k zc k x,k y,k z, k x = Nx/2 k y = Ny/2 k z = Nz/2 n,m,p, (8 where H i an arbitrary function and the ytem i truncated by impoing Cn,m,p = 0 for n N n, m N m, p N p. It i worth nothing that the Fourier patial dicretization preerve the divergence contrain (3 of Maxwell equation: if thee contraint are atified at t = 0, they are atified at all time. In matrix form, Eq. (6 and (7 become dc dt = L 1C + N (C, F, df dt = L 2C + L 3 F, (9 with C repreenting the coefficient of the Hermite expanion and F the electromagnetic field. The matrix L 1 correpond to the linear advection operator on the right hand ide of Eq. (6, while N i the non-linear operator aociated with the convolution. Finally, the matrice L 2 and L 3 are defined by the right hand ide of Eq. (7. A common apect of colliionle plama i the filamentation of phae pace [24], i.e. the development of increaingly maller phae-pace tructure. Thi implie that any numerical method would eventually run out of reolution. In the context of the Hermite bai, lack of reolution can lead to the well-known recurrence effect [25], and thi i typically addreed by adding a colliional term to the right hand ide of Eq. (6. We chooe [ C[Cn,m,p kx,ky,kz, ] = ν p(p 1(p 2 (N p 1(N p 2(N p 3 n(n 1(n 2 (N n 1(N n 2(N n 3 + m(m 1(m 2 (N m 1(N m 2(N m 3 + ] Cn,m,p kx,ky,kz,, (10 4

6 eentially to damp the high-order mode of the Hermite erie. Note that the operator in Eq. (10 doe not act on the firt three Hermite mode o that the conervation of total ma, momentum and energy remain valid in the dicrete, a hown in Ref. [20]. Equation (6 and (7 are dicretized in time by a fully-implicit, econd-order accurate Crank- Nicolon cheme. In reidual form, the reulting et of equation become { ( R1 C θ+1, F θ+1 = C θ+1 C θ t [ L 1 C θ+1/2 + N ( ( C θ+1/2, F θ+1/2] = 0 R 2 C θ+1, F θ+1 = F θ+1 F θ tl 2 C θ+1/2 tl 3 F θ+1/2 R(X = 0, = 0 (11 where we have introduced the time tep t, ued upercript θ to label time C(t θ = C(θ t = C θ, C θ+1/2 = ( [ [ ] C θ+1 + C θ C R1 /2 (and imilarly for F, X = and R =. F] R Jacobian-Free Newton-Krylov olver The mot important part of the algorithm i the olution of the nonlinear Eq. (11. For thi, we ue a Jacobian-Free Newton-Krylov (JFNK method [26]. At each time tep the initial gue X 0 i iteratively updated with the Newton method, X n+1 = X n + X n, where n i the nonlinear iteration number and X n i the olution update, which i obtained by olving the linear ytem J(X n X n = R(X n (12 with the Jacobian matrix J defined a J ij = R i(x, where R i label the i-th row of R. X j The linear ytem (12 i olved with the Generalized Minimal RESidual (GMRES iterative method. It approximate the exact olution by minimizing the Euclidean norm of the reidual J(X n X n +R(X n. The method require the computation of matrix-vector product J X n, which can be approximated by a directional derivative with a mall tep increment without actually toring J and computing the product Preconditioning It i well known that the convergence of the linear part of the JFNK olver trongly depend on the problem eigenvalue pectrum, and that GMRES typically doe not cale well a the problem ize increae. Hence, applying a uitable preconditioner can ignificantly accelerate the convergence rate and reduce the imulation time. By default PETSc ue left preconditioning, i.e. the preconditioner matrix P 1 i applied to the left of ytem (12: P 1 L J(Xn X n = P 1 L R(Xn. (13 An efficient preconditioner can effectively reduce the number of linear iteration without being too cotly to compute. In SPS, we follow [20] and contruct the preconditioner by accounting for the convolution in Eq. (11 via linearization relative to ome reference olution P 1 L [ I = t 2 L 1 t 2 N (, F ref t 2 N ( C ref ] 1, t 2 L 2 I t 2 L, (14 3 where I i the identity matrix. In principle, the preconditioner can be applied at every time tep, where the reference olution correpond to the olution at the previou time tep. In practice, we typically compute it only once at the beginning of the imulation (i.e. the reference olution i given by the initial condition of the ytem and apply it at all time tep. While thi can lead to a deterioration of the performance of the preconditioner a the imulation progree, in the example dicued below thi trategy wa quite efficient. L 5

7 3. Implementation of the SPS code The SPS code ha been implemented in Fortran to olve problem in a two-dimenional patial geometry with three velocity component. Fortran wa choen for it native upport for complex number operation that are widely ued in the code. The SPS code conit of approximately 4,000 line of code and i baed on the PETSc [22] and FFTW [27] librarie, and on OpenMP. We ue the PETSc library to olve the dicretized governing equation with the JFNK method. PETSc i an open-ource library for the numerical olution of partial differential equation. It i written in C and ha an interface to Fortran. Even though PETSc ha been primarily deigned to upport programming on ditributed ytem, current PETSc verion ue OpenMP and pthread for matrix and vector operation. The FFTW library [27] i ued to perform the convolution in Eq. (6 by mean of two Fat Fourier Tranform. The convolution i carried out by firt Fourier-tranforming the input vector from pectral pace to real pace, multiplying them in real pace and tranforming their reult back to the Fourier pace. The complexity of the FFT-baed convolution i O(M log M, where M i the input vector ize, a oppoed to the O(M 2 complexity of the direct calculation. The OpenMP API i ued to parallelize the SPS code on hared-memory ytem. A profiling of the SPS performance revealed that the mot compute-intenive part of the code i the evaluation of the reidual function during the JFNK olver iteration. For thi reaon, we mainly ue OpenMP to parallelize loop in thi part of the calculation. 4. Verification of SPS: Orzag-Tang Vortex Tet The SPS code ha been benchmarked on everal problem, including Landau damping, twotream intabilitie, beam-plama intabilitie and ion acoutic wave. Here we briefly dicu a complex tet cae: imulation of the two-dimenional Orzag-Tang vortex. The Orzag-Tang vortex i an initial configuration often ued in tudie of 2D plama turbulence (e.g. [28, 29]. Evolution of the ytem lead to the formation of current heet, which are narrow region of intene current denity correponding to large gradient of the magnetic field. If the ytem ize i large enough, the ytem eventually tranition to turbulence. We focu on colliionle magnetized plama with moderate ratio of thermal to magnetic preure β e β i 1, where the characteritic thickne of the current heet i expected to be of the order of d e ρ e. Here β = 8πn T /B0 2, ρ e = v te /ω ce, v t = T /m /c, n e n i are the electron and ion denitie, and T i the effective temperature of pecie. We conider a domain of ize L x L y, with the initial ditribution function given by 1 f (x, v, t = 0 = (2π 3/2 exp ( [v U (x] 2 vt 3 2vt 2 (15 and the initial magnetic field by B = e z + δb(x, where U = U 0 [ in(k y ye x + in(k x xe y ] + U,z e z and δb = δb B 0 [ in(k y ye x + in(2k x xe y ], with e β the unit vector along β. Here δb U 0 = V A B 0, V A = B 0 / 4πn 0 m i /c, k x = 2π/L x, k y = 2π/L y. To atify Ampere law, the initial configuration include the electron flow U e,z = δb ω ce B 0 ω pe [2k x co(2k x x + k y co(k y y]. The ion flow i U i,z = 0. The following phyical parameter were ued: domain ize L x = L y = 10d i, m i /m e = 25, T e = T i, ω pe /ω ce = 2, β e = β i = 0.25, δb/b 0 = 0.2. For the SPS imulation, the configuration (15 i expanded in Hermite-Fourier bai function numerically. We chooe αβ = 2v t and u β = 0. Our repreentation with u β = cont cannot capture Eq. (15 exactly with only one polynomial, therefore run conducted with different value of N = N n = N m = N p correpond to lightly different approximation of the initial condition. Other parameter ued for the imulation are: N x = N y = 128, ω pe t = 1 and ν = 1, while N will be varied parametrically. The reult of the SPS imulation are compared againt a reference olution obtained uing a fully kinetic particle-in-cell (PIC imulation of the ame 6

8 initial condition. The PIC imulation wa conducted uing the explicit relativitic imulation code VPIC [30]. The VPIC imulation with the parameter pecified above had cell with 4000 particle per cell per pecie at t = 0. The PIC time tep wa ω pe t Figure 1 compare the current denity j z normalized to a characteritic value en 0 v te at ω pe t = L x /V A in two SPS imulation obtained with N = 4 and N = 6, againt the reference PIC imulation. At thi time the ytem ha not tranitioned to fully developed turbulence yet. Depite a relatively mall number of Hermite polynomial, the tet cae uccefully reproduce both the large-cale tructure oberved in the PIC imulation and the mall, intene current heet. Some of the econdary feature are arguably reproduced better in the cae with larger number of polynomial in the bai. The multi-cale nature of fluctuation excited in the ytem i apparent from an etimate of the omnidirectional power pectrum of magnetic fluctuation S B preented in Fig. 2. The pectrum alo clearly demontrate potential advantage offered by noie-free pectral method for turbulence imulation. While the pectrum at very high k i eventually dominated by dicrete particle noie even in thi highly reolved PIC imulation, SPS can provide clean etimate of power in large-k mode. It i intereting to note that both PIC and SPS imulation ugget that power pectra extend to relatively mall cale kd e > 1, with a poible tranition to another power law at kλ D 1. A detailed invetigation of energy diipation and fluctuation pectra in thee imulation will be preented elewhere. 0 PIC 128x4 128x6 0.3 x/d e y/d e 0 50 y/d e 0 50 y/d e -0.3 Figure 1. Normalized current denity ĵ z = j z /(en 0 v te in the Orzag-Tang imulation at ω pe t = L x /V A. Left: Reference PIC imulation; Middle: SPS imulation with N x = N y = 128 and N=4; Right: SPS imulation with N x = N y = 128 and N=6. The global energy balance in the Orzag-Tang vortex imulation i illutrated in Fig. 3. SPS reproduce well the change in the different energy channel meaured in the reference PIC imulation. We note that in contrat to fluid and MHD turbulence, diipation of fluctuating energy in weakly colliional plama i due to collective interaction of particle with the electromagnetic field, even though the ultimate diipation in the ene of entropy generation i provided by colliion. Undertanding of the turbulent energy diipation i thu quite challenging and ha been a focu of intene reearch effort. The methodology preented here may make a ignificant contribution to the field by enabling accurate imulation of kinetic phenomena in turbulence, free of inherent limitation of other currently available method. 5. Performance Reult In thi ection we dicu the performance of SPS. The imulation were performed on a node of the Wolf upercomputer at the Lo Alamo National Laboratory. Each Wolf node ha two 8-core Intel Xeon E proceor with 2.6 GHz CPU and 64 GB of memory. 7

9 10 5 SPS,128x6 SPS,128x4 PIC PIC thermal 10 0 S B kd e =1 kλ D = kd i Figure 2. Omnidirectional power pectrum of magnetic fluctuation in the Orzag-Tang vortex imulation at the ame time a in Fig. 1. The black dotted line how the pectrum of thermal noie in the PIC imulation. The pectrum i normalized according to B 2 dxdy = S B (kdk e, SPS i, SPS B, SPS e, PIC i, PIC B, PIC δw/w t/t A Figure 3. Change in the kinetic energy of electron (e, ion (i, and of the magnetic field energy (B in the Orzag-Tang vortex imulation. Here δw (t = W (t W (t = 0, W kin, = (m v 2 /2f d 3 vd 2 x, W B = (B 2 /2 d 2 x, and ω pe t = L x /V A. The energie are normalized to the energy of the initial perturbation W 0 = [ (m i Ui 2/2 + m eue 2 /2 + δb 2 /2 ] d 2 x. SPS imulation had N x = N y = 128 and N = 6. The SPS code ha been compiled with the Intel Fortran compiler and linked to PETSc verion and to FFTW verion The relative and abolute tolerance of JFNK are et to PETSc Incomplete LU (ILU factorization i ued to invert matrix P L for preconditioning. For preconditioned and unpreconditioned imulation, we et the PETSc option -ne mf operator and -ne mf, repectively Performance of the preconditioner Table 1 how the running time and the average number of nonlinear and linear iteration per time tep for the Orzag-Tang problem. The imulation parameter are decribed in Section 4, except that N x = N y = 32 and the imulation time correpond to 100 time tep. Let u focu on the unpreconditioned cae. For N = 4 the average number of linear iteration per time tep i 14 and the running time i 297. A we double N, the problem ize increae by a factor 7.7 but the running time i 14 time longer ince the average number of linear iteration double. 8

10 Going from N = 8 to N = 12 increae the problem ize by a factor 3.4 while the running time increae by a factor 5.3, again becaue of the higher number of linear iteration (now 52 required for convergence. Let u now look at the preconditioned cae. For N = 4, the average number of linear iteration i 5, about a third of thoe for the unpreconditioned cae, tranlating into a 40% reduction of the running time. Up to N = 16 the number of linear iteration remain fairly contant and, conequently, the running time how a fairly good linear caling with the number of unknown of the ytem. For N = 12 there i a factor of 4 peed-up relative to the unpreconditioned cae and the gain will increae for larger problem. The preconditioner loe performance for N = 20, where the average number of linear iteration rie to 8.5 with a 20% increae in the running time relative to a linear caling. Thi could arie from the problem not being exactly the ame a N change, due to the approximation of the initial condition and the nature of Eq. (10. Finally, the number of nonlinear iteration i inenitive to preconditioning. Table 1. Running time and average number of nonlinear and linear iteration per time tep for the Orzag-Tang problem with N x = N y = 32 and 16 thread. Preconditioned Unpreconditioned N Unknown Time, Nonlin./ Nonlin./ Time, Lin. iter. Lin. iter , / / , 121, 670 1, 475 2/5.4 4, 138 2/ , 770, 118 5, 165 2/5.6 22, 120 2/ , 927, , 130 2/ , 430, , 200 2/ Parallel performance on hared-memory ytem A trong caling tet, where we kept the problem ize fixed and increaed the number of thread, ha been carried out for the Orzag-Tang problem. The parameter are the ame a thoe ued for the preconditioning tudie in the previou ubection and we focu on the cae with N = 8. Table 2 how the parallel peed-up varying the number of thread in ue: the trong caling tet how a factor of 10 peed-up with 16 thread. In addition, Amdahl law [31]: Speed-up = 1 S + 1 S p (16 i ued to evaluate the performance of SPS. In Eq. (16, S i the fraction of time in the code that i not parallelized and p i the number of thread. For the Orzag-Tang tet, S = wa calculated by running the code with 1 thread and uing Intel Vtune for profiling. Table 2 how that there i good agreement between the theoretical and meaured peed-up. 6. Concluion We have preented the deign and implementation of the SPS code, which olve the Vlaov- Maxwell equation for a plama in multiple dimenion. The code i baed on a Hermite-Fourier pectral decompoition of the particle ditribution function. The reulting et of non-linear ODE i dicretized in time with a econd-order accurate Crank-Nicolon cheme and i olved numerically with a preconditioned JFNK olver. SPS i written in Fortran and ue the PETSc library for olving the nonlinear equation and the FFTW library to handle the convolution operation efficiently. The code i parallelized for 9

11 Table 2. Strong caling tet: parallel peed-up varying the number of thread for the Orzag- Tang problem with N x = N y = 32, N = 8 and with preconditioning. Thread Speed-up Amdahl law hared-memory machine uing OpenMP. A comparion againt a fully kinetic PIC imulation for a Orzag-Tang vortex problem how that SPS can capture reaonably well the large-cale and mall-cale current tructure of the ytem, depite uing a relative low number of Hermite mode. It alo highlight the trength of the method and it ability to capture the high-k tail of the fluctuation pectrum which i normally obcured by noie in PIC imulation. Performance tet have alo been preented on the ame problem, howing a ignificant reduction of the number of linear iteration and of the running time obtained by preconditioning, and a factor of 10 peed-up for a trong caling tet with 16 thread. Acknowledgment Thi work wa funded by the Laboratory Directed Reearch and Development (LDRD program, under the aupice of the National Nuclear Security Adminitration of the U.S. Department of Energy by Lo Alamo National Laboratory, operated by Lo Alamo National Security LLC under contract DE-AC52-06NA Thi reearch ued reource provided by the Lo Alamo National Laboratory Intitutional Computing Program. VR acknowledge upport from NASA grant NNX15AR16G. PIC imulation utilized reource provided by the NASA High-End Computing Program through the NASA Advanced Supercomputing Diviion at Ame Reearch Center. Reference [1] Goldton R J and Rutherford P H 1995 Introduction to Plama Phyic Plama Phyic Serie (Intitute of Phyic Publication [2] Fitzpatrick R 2014 Plama Phyic (CRC Pre [3] Kolobov V and Arlanbekov R 2012 Journal of Computational Phyic [4] Markidi S, Henri P, Lapenta J, Rönnmark K, Hamrin M, Meliani Z and Laure E 2014 Journal of Computational Phyic frontier in Computational Phyic Modeling the Earth Sytem [5] Daldorff L K S, Tóth G, Gomboi T I, Lapenta G, Amaya J, Markidi S and Brackbill J U 2014 Journal of Computational Phyic [6] Birdall C K and Langdon A B 2004 Plama Phyic Via Computer Simulation (Taylor & Franci [7] Hockney R and Eatwood J 1988 Computer Simulation Uing Particle (Taylor & Franci [8] Cheng C Z and Knorr G 1976 Journal of Computational Phyic [9] Filbet F, Sonnendrücker E and Bertrand P 2001 Journal of Computational Phyic [10] Vay J L, Colella P, Kwan J W, McCorquodale P, Serafini D B, Friedman A, Grote D P, Wetenkow G, Adam J C, Hron A and Haber I 2004 Phyic of Plama 11 [11] Fujimoto K and Machida S 2006 Journal of Computational Phyic [12] Arlanbekov R R, Kolobov V I and Frolova A A 2013 Phy. Rev. E 88( [13] Armtrong T P, Harding R C, Knorr G and Montgomery D 1970 Method Comput. Phy [14] Camporeale E, Delzanno G L, Lapenta G and Daughton W 2006 Phyic of Plama [15] Vencel J, Delzanno G L, Johnon A, Peng I B, Laure E and Markidi S 2015 Procedia Computer Science [16] Manzini G, Delzanno G L, Vencel J and Markidi S 2015 J. Comput. Phy. (Submitted [17] Parker J T and Dellar P J 2015 Journal of Plama Phyic 81(02 [18] Loureiro N F, Schekochihin A A and Zocco A 2013 Phyical Review Letter 111(

12 [19] Camporeale E, Delzanno G, Bergen B and Moulton J 2016 Computer Phyic Communication [20] Delzanno G L 2015 Journal of Computational Phyic [21] Loureiro N F, Dorland W, Fazendeiro L, Kanekar A, Mallet A, Vilela M S and Zocco A 2015 arxiv: [22] Balay S, Abhyankar S, Adam M F, Brown J, Brune P, Buchelman K, Dalcin L, Eijkhout V, Gropp W D, Kauhik D, Knepley M G, McInne L C, Rupp K, Smith B F, Zampini S and Zhang H 2015 PETSc Web page URL [23] Holloway J P 1996 Tranport Theory and Statitical Phyic [24] Klima A J 1987 Journal of Computational Phyic [25] Joyce G, Knorr G and Meier H K 1971 Journal of Computational Phyic [26] Knoll D A and Keye D E 2004 Journal of Computational Phyic [27] Frigo M and Johnon S G 2005 Proceeding of the IEEE [28] Orzag S A and Tang C M 1979 Journal of Fluid Mechanic [29] Bikamp D and Welter H 1989 Phyic of Fluid B [30] Bower K J, Albright B J, Yin L, Bergen B and Kwan T J T 2008 Phyic of Plama [31] Amdahl G M 1967 Validity of the ingle proceor approach to achieving large cale computing capabilitie Proceeding of the April 18-20, 1967, pring joint computer conference (ACM pp