Large Scale Parallel Wake Field Computations with PBCI E. Gjonaj, X. Dong, R. Hampel, M. Kärkkäinen, T. Lau, W.F.O. Müller and T. Weiland ICAP `06 Chamonix, 2-6 October 2006 Technische Universität Darmstadt, Fachbereich Elektrotechnik und Informationstechnik Schloßgartenstr. 8, 64289 Darmstadt, Germany - URL: www.temf.de
Outline Introduction Numerical Method Parallelization Strategy Modal Termination of Beam Pipes PBCI Simulation Examples Conclusions 1
Introduction Motivation for PBCI: 1. A new generation of LINACs with ultra-short electron bunches a. bunch size for ILC: 300 μm 2 b. bunch size for LCLS: 20 μm 2. Geometry of tapers, collimators far from rotational a. 8 rectangular collimators at ILC-ESA in the design process b. 30 rectangular-to to-round transitions in the undulator of LCLS 3. Many (semi-) analytical approximations become invalid a. based on rotationally symmetric geometry b. low frequency assumptions (Yokoya, Stupakov) c. detailed physics needed for high frequency wakes (Bane)
Introduction beam view side view 15.05mm 38.1mm 3 2.75mm 38.05mm 17.65mm 132.54mm Courtesy: N. Watson, Birmingham 1 m 38.05mm ILC-ESA collimator #3 ILC-ESA collimator #8 bunch length 300μm collimator length ~1.2m catch-up distance ~2.4m
Introduction vacuum vessel tube shielding PITZ diagnostics double cross gap outgoing pipe step bunch length bunch width 2.5mm 2.5mm bunch length taper length 1cm 50cm structure length 325mm Tapered transition @PETRA III vacuum slots elliptic pipe 4 Courtesy: R. Wanzenberg, DESY
Introduction There is an actual demand for: 1. Wake field simulations in arbitrary 3D-geometry 3D-codes 2. Accurate numerical solutions for high frequency fields (quasi-) dispersionless codes 3. Utilizing large computational resources for ultra-short bunches parallelized codes 4. Specialized algorithms for long accelerator structures moving window codes 5
An (incomplete) survey of available codes Introduction Dimensions Nondispersive Parallelized Moving window 1982 BCI / TBCI 2.5D No No Yes Time 2002 2006 NOVO ABCI MAFIA GdfidL ECHO PBCI 2.5D 2.5D 2.5/3D 3D 2.5/3D 3D Yes No No No Yes Yes No No No Yes No Yes No Yes No No Yes Yes 6
Outline Introduction Numerical Method Parallelization Strategy Modal Termination of Beam Pipes PBCI Simulation Examples Conclusions 7
Numerical Method The FIT discretization 8 A A V V E ds = μ H da t A H ds = ε E + J da t A μ H da= 0 εe da= Topology of FIT: V ρ dv FIT Ce = d dt M h μ d Ch = M ε e + j dt SM e q ε = SMμ h = C T = C semidiscrete energy conservation SC = SC = 0 semidiscrete charge conservation 0
Numerical Method Using the conventional leapfrog time integration n+ 1/2 1 T n 1/2 n e Δt e 1 M 1 ε C t ε n+ 1 = Δ 1 2 1 1 T n M j Δt μ Δt h M C 1 MμCMε C h 0 9 180 150 210 Behavior of numerical phase velocity vs. propagation angle 90 120 60 0.75 0.50 0.25 240 270 300 cδt 1 σ : = Δz 3 30 0 330 Stable but large dispersion z 90 120 60 1.00 150 0.75 0.50 0.25 180 210 240 270 300 cδt σ : = = 1 Δz 30 0 330 No dispersion but unstable z
Numerical Method Idea: Reduce the integration of 3D equations to a sequence of 1D / 2D integrations along the coordinate axes Longitudinal-Transversal (LT) splitting: 10 H H t l 1 T 0 Mε C t = 1 Mμ Ct 0 does not affect longitudinal waves 1 T 0 M ε Cl = 1 Mμ Cl 0 integrable in 1D without numerical dispersion n+ 1 n n Δt Δt t t t Δ 2 lδt 2 3 e = e e H H = e e e e H H + O( Δt ) modified time evolution h h h Second order Strang splitting
Numerical Method Idea: Reduce the integration of 3D equations to a sequence of 1D / 2D integrations along the coordinate axes Replace each evolution operator with Verlet-leapfrog propagators: 11 G 2 3 Δt 1 T 1 1 T Δt 1 T 1 1 T 1 Mε Clt ; MμClt ; ΔtMε Clt ; Mε Clt ; MμClt ; Mε Clt ; 2 4 Δ = 2 ( t) lt ; 2 1 Δ t 1 1 T ΔtMμClt ; 1 MμClt ; Mε Clt ; Time discrete update: n+ 1 e Δt Δt e = Gt igl( Δt) igt i h 2 2 h T. Lau, E. Gjonaj and T. Weiland, Time Integration Methods for Particle Beam Simulations with the Finite Integration Theory, FREQUENZ, Vol. 59 (2005), pp. 210 n cδt stable for: σ : = = 1 Δz
Implementing the LT splitting scheme n+ 1 e Δt Δt e = Gt igl( Δt) igt i h 2 2 h Numerical Method n 12 Numerical phase velocity and amplification vs. propagation angle 90 120 60 0.75 150 0.50 30 0.25 180 0 210 330 cδt σ : = = 1 Δz No longitudinal dispersion z 90 120 60 0.75 150 0.50 30 0.25 180 0 240 300 240 300 270 270 210 330 z Neutrally stable
Outline Introduction Numerical Method Parallelization Strategy Modal Termination of Beam Pipes PBCI Simulation Examples Conclusions 13
Parallelization Strategy A balanced domain partitioning approach total computational domain 14 W W N left right left = N N =, Nright = 2 2 N N left right intermediate subdomains processor #1 active subdomains binary tree leafs processor #2 processor #3 Equal loads assigned to each node: W = α i w Node Node i Grid Points
Parallelization Strategy Example: Tapered transition for PETRA III moving grid window Domain partitioning pattern for 7 processors 15 Total Min Max Dev. Grid points 64.481.201 9.119.943 9.292.230 < 1.0% Number of Grid Points 9.300.000 9.250.000 9.200.000 9.150.000 9.100.000 9.050.000 9.000.000 9.292.230 Distribution of grid points 9.246.000 9.246.000 9.200.000 9.211.428 9.165.600 1 2 3 4 5 6 7 Processor Number 9.119.943
Parallelization Strategy 16 Speedup 20 18 16 14 12 10 8 6 4 2 Parallel performance tests 10e5 Grid points 10e6 Grid points 50e6 Grid points 10e7 Grid points 20e7 Grid points Ideal 0 0 2 4 6 8 10 12 14 16 18 20 Number of Processors TEMF Cluster: 20 INTEL CPUs @ 3.4GHz, 8GB RAM, 1Gbit/s Ethernet Network
Parallelization Strategy 1,2 Parallel performance tests 1 17 Efficiency 0,8 0,6 0,4 0,2 10e5 Grid points 10e6 Grid points 50e6 Grid points 10e7 Grid points 20e7 Grid points Ideal 0 0 2 4 6 8 10 12 14 16 18 20 Number of processors TEMF Cluster: 20 INTEL CPUs @ 3.4GHz, 8GB RAM, 1Gbit/s Ethernet Network
Outline Introduction Numerical Method Parallelization Strategy Modal Termination of Beam Pipes PBCI Simulation Examples Conclusions 18
Modal Termination of Pipes y Indirect integration schemes C 2 C0 z = 0 C1 z 19 direct indirect 1 z+ s Wz() s = dzez(, z t ) Q = c 1 z+ s 1 TM s = dz Ez(, z t ) dy Gy (0, t ) Q = = c Q c C 0 2 1. Indirect integration of potential for 2D-structures (Weiland 1983, Napoly 1993) 2. Generalization for 3D-structures (A. Henke and W. Bruns, EPAC 06, July 2006, Edinburgh, UK) TM G = e E + cb + e E cb + e E TM TM TM TM ( ) ( ) x x y y y x z z C irrotational
Modal Termination of Pipes y z = 0 Modal approach C0 C1 z 20 direct 1 z+ s Wz() s = dzez(, z t ) Q = c = n modal n 1 i( ω / c) s z(, ) ω n( ω) i ω c k ω e x y d C e ( / zn, ( )) 1 z+ s 1 n = dz Ez(, z t ) ez(, x y) Wn() s Q = c Q = + = = z s i n ikn ( ω) z c dz Ez( x, y, z, t ( z s)/ c) dz dω Cn( ω) ez( x, y) e e ω + 0 0 n C 0 W () n s n-th (TM) mode contribution n spectral coefficient of n-th (TM) mode
Modal Termination of Pipes 1. Time domain integration in the inhomogeneous sections: 0 1 z+ s dz Ez (, z t ) Q = c 21 n n 2. Modal analysis at z = 0: E ( x, y,0, t) E (0, t), e ( x, y) z z z n 3. Compute spectral coefficients (FFT): E (0, t) C ( ω) 4. Compute wake potential contribution per mode (IFFT): C n ( ω) ( ω/ zn, ( ω) ) i c k W n () s 5. Compute wake potential transition in the outgoing pipe: 1 z+ s 1 n dz Ez(, z t ) ez(, x y) Wn() s Q = = c Q 0 n z n
Modal Termination of Pipes Using FD reconstruction in long intermediate pipes diagnostics cross injector section bunch 22 1mm step full Time Domain -20-10 0 10 20 E z / [kv /m] FD analysis reconstruction lowest 10 15 20 5 eigenmodes bunch
Outline Introduction Numerical Method Parallelization Strategy Modal Termination of Beam Pipes PBCI Simulation Examples Conclusions 23
Simulation Examples TESLA 9-cell cavity bunch 24-15 -10-5 0 5 10 15 E z / [kv /m] bunch length 5mm bunch charge 1nC cavity length 1.5m no. of grid points ~80e6 no. of processors 24 simulation time 3hrs
Simulation Examples TESLA 9-cell cavity 5 Longitudinal wake potential 25 Wz / [V / pc] 0-5 -10-15 -1,5 0 1,5 3 4,5 s / [cm] Bunch (a.u.) z = 20cm z = 40cm z = 60cm z = 80cm z = 100cm z = 120cm z = 140cm
PITZ diagnostics double cross Simulation Examples bunch 26-50 -25 0 25 50 E z / [kv /m] bunch size σ / Δz no. of grid points no. of processors simulation time 2.5mm 30 ~500e6 24 32hrs Vacuum Vessel Gap Beam Tube Tube Shielding Step
PITZ diagnostics double cross 1 Simulation Examples (Ackermann, Hampel, Schnepp) Wake field impact of the single components 27 W z / [V / pc] 0-1 -2-3 cross with shielding tube full model no shielding tube gap (1 mm) step (no taper) -15-10 -5 0 5 10 15 s [mm]
Simulation Examples 28 ILC-ESA collimator #8 Wz / [V / pc] 60 40 20 0-20 -40-60 -80 Convergence vs. grid step σ / Δz = 2.5 σ / Δz = 5 σ / Δz = 10 σ / Δz = 15 bunch size no. of grid points no. of processors simulation time -100-4 -3-2 -1 0 1 2 3 4 s / σ 300μm ~450e6 24 85hrs
Simulation Examples ILC-ESA collimator #8 Direct and transition wakes bunch 15.05 mm 17.65 mm 38.1 mm 132.54 mm 2.75 mm 38.05 mm 29 Wz / [V / pc] 30 Direct potential (z = 150mm) Transition potential Steady state potential 10-10 -30-50 -70-90 -4-3 -2-1 0 1 2 3 4 s / σ
Conclusions 1. PBCI: A fully 3D- code for wake field simulations a. using the moving window approach b. dispersionless in the bunch propagation direction c. massively parallelized d. using modal approach for indirect integration e. using modal approach for pipe termination 2. Work still in progress for a. including resistive wall wakes b. developing an appropriate boundary conformal discretization c. considering periodic structures of finite length 30
General split-operator schemes d dt e 1 T 0 Mε C = 1 h Mμ C 0 h e Numerical Method (homogeneous) FIT equations 32 Denote: e y =, h Solution after one time step: d y dt 0 M C H = M C 0 1 T ε, 1 μ ( H H ) + 1 HΔ 1+ 2 Δt Hiy y e iy e iy = = = A second order Strang scheme: n t n n y n H= H1+ H2 exact time evolution operator approximate time evolution operator Δt Δt H H 1 3 + 1 1 2 H2Δt 2 n = e e e i y + O( Δt )