Recent Innovative Numerical Methods

Recent Innovative Numerical Methods Russel Caflisch IPAM Mathematics Department, UCLA 1

Overview Multi Level Monte Carlo (MLMC) Hybrid Direct Simulation Monte Carlo (DSMC) Accelerated Molecular Dynamics (AMD) Multiscale Finite Element Method (MFEM) In-Painting for Scientific Computation Split Bregman Empirical Mode Decomposition (EMD) Stochastic Gradient Descent (SGD) 2

Multi-Level Monte Carlo for SDEs Stochastic differential equation (SDE) dv i = F i dt + D ij dw j, (11) where f is probability density of v and i, j are component indices W = W (t) is Brownian motion in velocity dw is white noise in velocity Objective is an average of f : 1 P(v)f (v, t) dv E[P(v(t))] (12) ρ Russel Caflisch MLMC for Plamas

Discretization of SDEs Euler-Maruyama discretization in time: v i,n+1 = v i,n + F i,n t + D ij,n W j,n, (13) W n = W n+1 W n (14) in which v i,n = v i (t n ) and F n = F(v n ) Choose N Brownian paths to get N values of P(v(T )) Average to approximate E[P(v(T ))] Computational cost vs. Error ε: Statistical error is O(N 1/2 ) t error is O( t), since W = O( t) and random Optimal choice is ε = N 1/2 = t Cost = N t 1 = ε 3 Russel Caflisch MLMC for Plamas

Like multi-grid MLMC Basics method. ˆv NL Introduce time step levels, t l = T 2 l, for l = 0,..., L Expectations with L = E[v 0]+ N l samples LX E[(v l v l 1 )] l=1 Convergent sum, L!1 when using Milstein method Statistical error converges with t Grid l λ Let P l = P(v tl ). Then 1. Giles in Monte Carlo and Quasi-Monte Carlo Method, Springer-Verlag, (2006) 7 E[P L ] = E[P 0 ] + L E [P l P l 1 ] (24) l=1 When computed using same Brownian path, the variance of (P l P l 1 ) is O(strong error) 2 Russel Caflisch MLMC for Plamas

MLMC Scaling Optimal number of samples used to compute each E[P l P l 1 ], constrained by RMSE < ε. The complexity now scales like 11 { ( O ε Cost = 2 (log ε) 2) for Euler-Maruyama O ( ε 2) for Milstein (25) Notes: MLMC-Euler-Maruyama scales better than standard MC MLMC-Milstein is even better Restricted to d = 1, 2 due to difficulty with Levy areas O ( ε 2) scaling is possible without Milstein, using antithetic sampling method 12 11 Giles, Operations Research, 56(3):607, 2008 12 Giles & Szpruch, arxiv:1202.6283, 2012 Russel Caflisch MLMC for Plamas

A Sample Plasma Problem 10 0 Direct MLMC Euler MLMC Milstein (lnε) 2 K ε 2 10-1 10-5 10-4 10-3 ε Rosin, Ricketson, et. al., submitted to JCP, 2013 Russel Caflisch MLMC for Plamas

Background Hybrid Negative New Numerics Hybrid DSMC for Plasmas Vlasov-Poisson-Landau (VPL) system for non-equilibrium plasma tf + v xf E vf = Q L(f, f ), x E = ρ(t, x) = f (t, x, v) dv, Landau-Fokker-Planck operator for Coulomb interactions is Q L(g, f )(v) = A ( uσ tr(u)(u 2 δ ij u iu j) ) g(w)f (v) dw 4 v i R 3 v j w j u = v u σ tr(u) u 3 A is proportional to Coulomb logarithm Magnetic field omitted here Q L(g, f ) is asymmetric, and describes the change in f due to collisions with g (used in negative particle section below) Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 8/ 46

Background Hybrid Negative New Numerics DSMC for Plasma Kinetics The PIC-DSMC method is widely used in plasma simulation Particle-In-Cell method (PIC) for collisionless plasma. Dawson 83, Birdsall-Langdon 85 Direct Simulation Monte Carlo (DSMC) for binary collisions. Takizuka-Abe 77, Nanbu 97, Bobylev-Nanbu 2000 In each time step of DSMC, perform collisions between N c randomly chosen pairs of particles For rarefied gas (charge neutral, short range), N c = O( tn). Each particle collides at the physically correct rate. Collisions are physical collisions For Coulomb gas (charged, long range), N c = N/2. Every particle collides once in every time step Collisions are aggregates depending on t Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 9/ 46

Background Hybrid Negative New Numerics Hybrid Scheme Combine fluid and particle simulation methods 1 : Treat as fluid Treat as particles Separate f into Maxwellian and non-maxwellian (particle) components: f = m + f p Treat m as fluid Simulate f p by Monte Carlo Interaction of m and f p : sample particles from m; collide with particles from f p Similar to δf methods, but fully nonlinear Limited to f p 0 1 Caflisch et. al, Multiscale Model. Simul. 2008 Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 11/ 46

Background Hybrid Negative New Numerics Motivation for Negative Particles Apply decomposition f (t, x, v) = M(t, x, v) + f d(t, x, v), with f d(t, x, v) alowed to be positive or negative, so that f d is minimized. 1.5 1 0.5 M f = M + f d 0 f d 0.5 3 2 1 0 1 2 3 v (f d) + and (f d) are represented by positive and negative deviational particles. Developed here for LFP. Similar work for RGD. 5 5 Baker & Hadjiconstantinou, Phys Fl (2005) Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 18/ 46

Background Hybrid Negative New Numerics Meaning of Negative Particles A negative particle w in f d cancels a (positive) particle w + in m or f d So a P N collision (v +, w ) cancels a corresponding P P collision (v +, w + ). The P P collision removes v +, w + and adds v +, w +: P-P: v +, w + v +, w + So the P N collision adds v +, removes w (i.e., adds w + ), and adds v, w (i.e., removes v +, w +) P-N: v +, w 2v +, v, w Derived from the Boltzmann equation. 7 Particle number can grow! New method controls this growth. 7 Baker & Hadjiconstantinou, Phys Fl (2005) Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 24/ 46

Background Hybrid Negative New Numerics Nonlinear Landau Damping in VPL system M (f d ) + (f d ) 0.4 0.3 0.2 0.1 0 HDP PIC DSMC Figure: The distribution in the x v 1 phase space at time t = 1.25 in the nonlinear Landau damping problem of the VPL system. Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 43/ 46

Background Hybrid Negative New Numerics Efficiency Test on VPL System ρ(t = 0, x) = 1 + α sin(x) error 10 1 10 2 PIC DSMC, α = 0.1 HDP, α = 0.1 PIC DSMC, α = 0.01 HDP, α = 0.01 PIC DSMC, α = 0.001 HDP, α = 0.001 10 3 10 4 10 3 10 4 10 5 cputime Figure: The efficiency test of the HDP method on the VPL system for different α in the initial density. Russel Caflisch Accelerated Simulation for RGD & Plasma Kinetics 45/ 46

Accelerated Molecular Dynamics Methods A very brief introduction Arthur F. Voter Theoretical Division Los Alamos National Laboratory Los Alamos, New Mexico USA Work supported by DOE/BES Los Alamos LDRD program DOE/ASCR, DOE/SciDAC Los Alamos

Example: Film or Crystal Growth Deposition event takes ~2 ps use molecular dynamics (can reach ns) Time to next deposition is ~1 s - diffusion events affect the film morphology - mechanisms can be surprisingly complex --> need another approach to treat these Los Alamos

Infrequent-Event System The system vibrates in 3-N dimensional basin many times before finding an escape path. If we could afford to run molecular dynamics long enough (perhaps millions of vibrations), the trajectory would find an appropriate way out of the state. It is interesting that the trajectory can do this without ever knowing about any of the other possible escape paths. Los Alamos

Accelerated Molecular Dynamics Methods Hyperdynamics Parallel Replica Dynamics Temperature Accelerated Dynamics Builds on transition state theory and importance sampling to hasten the escape from each state in a true dynamical way. The boosted time is calculated as the simulation proceeds. (AFV, J. Chem. Phys., 1997) Harnesses parallel power to boost the time scale. Very simple and very general; exact for any infrequent event system obeying exponential escape statistics. (AFV, Phys. Rev. B, 1998) Raise T to make events happen more quickly. Filter out events that should not have happened at correct T. More approximate, but more powerful. (M.R. Sorensen and AFV, J. Chem. Phys., 2000) Los Alamos

Wide range of systems can be studied Cu/Ag(100), 1 ML/25 s T=77K, Sprague et al, 2002. Interstitial defects in MgO, ps s, Uberuaga et al, 2004. Annealing nanotube slices, µs, Uberuaga et al, 2011. Driven Cu GB sliding, 500 µm/s Mishin et al, 2007. Hexadecane pyrolysis, µs, Kum et al, 2004. Cu void collapse to SFT, µs, Uberuaga et al, 2007. Ag nanowire stretch, µs - ms, Perez et al, TBP. Recent brief review: Perez et al, Ann. Rep. Comp. Chem. 5, 79 (2009). Los Alamos

Multiscale Finite Element Method PDEs with multiscale solutions. Goal: obtain the large scale solutions, without resolving small scales. Method: construct finite element base functions which capture the small scale information within each element. Small scale information correctly influences large scales global stiffness matrix. Base functions are constructed from the leading order homogeneous elliptic equation in each element. Difficulty: resonance can lead to larger errors Resonance is between the small and large scales Solution: Choose BCs for the base function to cancel resonance errors Convergence independent of the small scales Hou & Wu, JCP 134 (1997) 169-189 3

Multiscale Finite Element Method (MFEM) Steady flow in porous medium Pressure u from rapidly varying conductivity tensor a(x) ax ( ) u= f Velocity field q is related to the pressure u through Darcy s law: q = a u MFEM-L, MFEM-O refer to BC choices, MFEM-os refers to oversampling 4

In-Painting for Scientific Computation Image inpainting Extend image to region missing or corrupted where info is TV inpainting model: given image u 0 and region D, inpainted image u minimizes T. Chan and J. Shen (2002) D Ω = + V ( u u, D) u dx λ u u dx ( ) 2 0 0 Ω Ω\ D 5

TV Impainting Chan, Shen (2005) 6

Impainting of Texture Bertalmio, Vese, Sapiro, Osher (2003) 7

In-Painting for Plasma Computations Doesn t work well for continuation of PDE solution Jenko, Osher, Zhu 8

Split Bregman An optimization method for compressed sensing and related fields Norms are L 1 and L 2, respectively. Difficulty: L 1 term isn t smooth Relax the first term by min u Φ ( u) + Hu ( ) min ud d + Hu ( ) + λ d Φ( u) 1, 1 2 2 Φ ( u) = u H ( u) = µ Au f 2 2 9

Starting from Split Bregman min ud d + Hu ( ) + λ d Φ( u), 1 2 Improve iteration by feeding back the noise through term b: k+ 1 k+ 1 k 2 ( u, d ) = argmin ud, d + Hu ( ) + λ d Φ( u) b 1 2 b = b +Φ( u ) d k+ 1 k k+ 1 k+ 1 Split the first line into two pieces to get split Bregman u = argmin Hu ( ) + λ d Φ( u) b k+ 1 k k u k+ 1 k+ 1 k 2 = arg min d + λ Φ( ) 1 2 d d d u b b = b +Φ( u ) d k+ 1 k k+ 1 k+ 1 2 2 2 Easily solved: u problem is smooth, d problem is soft-thresholding Widely used for problems involving sparsity 10

Convergence results Split Bregman Results Reconstruction results 11

Empirical Mode Decomposition (EMD) Adaptive data analysis method Determine trend and instantaneous frequency of time series f(t) Find sparsest representation of f(t) within a dictionary of intrinsic mode functions (IMFs) Dictionary construction { ()cos θ(): θ'() 0, () cos θ() } D = a t t t a t smoother than t min M st.. f() t = a ()cos t θ (), t a cosθ D Smoothness measured through TV 3 norm M k = 1 3 (4) TV ( a) a () t dt = k k k k Huang, Proc Roy Soc, 1989 Hou & Shi, Adv Adaptive Data Anal, 2011 12

Example Empirical Mode Decomposition ( π ) ( π ) f( t) = 6t+ cos 8 t + 0.5cos 40 t IMFs captured almost exactly Frequencies captured very well 13

Stochastic Gradient Descent Optimization for functions of the form Widely used for machine learning and internet computations Each i represents piece of data used for training of a learning method Randomly choose data for ith step Perform gradient descent with step size η Many variants Qw ( ) = Qi ( w) n i= 1 w: = w η Q( w) i Series of batches with η constant within batch, decreasing between batches Wikipedia 14

Stochastic Gradient Descent Convergence of mini-batch method 15