Computing the Action of the Matrix Exponential Nick Higham School of Mathematics The University of Manchester higham@ma.man.ac.uk http://www.ma.man.ac.uk/~higham/ Joint work with Awad H. Al-Mohy 16th ILAS Conference, Pisa, June 2010
The f(a)b Problem Given matrix function f : C n n C n n, A C n n, b C n, compute f(a)b without first computing f(a). Most important cases f(x) = x 1, f(x) = e x. Application : y (t) = Ay(t), y(0) = b y(t) = e At b. MIMS Nick Higham Computing the Action of the Matrix Exponential 2 / 29
Second Order ODE d 2 y dt + Ay = 0, 2 y(0) = y 0, y (0) = y 0 has solution y(t) = cos( At)y 0 + ( A ) 1 sin( At)y 0. MIMS Nick Higham Computing the Action of the Matrix Exponential 3 / 29
Second Order ODE d 2 y dt + Ay = 0, 2 y(0) = y 0, y (0) = y 0 has solution y(t) = cos( At)y 0 + ( A ) 1 sin( At)y 0. But [ ] y = exp y ([ 0 ta t I n 0 ]) [ ] y 0. y 0 MIMS Nick Higham Computing the Action of the Matrix Exponential 3 / 29
Themes Simple, direct algorithm for computing e A B with arbitrary A using only matrix products. Many problems can be reduced to e A B. Backward error viewpoint avoids consideration of conditioning in algorithm design. MIMS Nick Higham Computing the Action of the Matrix Exponential 4 / 29
Exponential Integrators u (t) = Au(t) + g(t, u(t)), u(0) = u 0, t 0, Solution can be written u(t) = e ta u 0 + ( ) ϕ k ta t k u k, k=1 where u k = g (k 1) (t, u(t)) t=0 and ϕ l (z) = k=0 zk /(k + l)!. MIMS Nick Higham Computing the Action of the Matrix Exponential 5 / 29
Exponential Integrators u (t) = Au(t) + g(t, u(t)), u(0) = u 0, t 0, Solution can be written u(t) = e ta u 0 + ( ) ϕ k ta t k u k, k=1 where u k = g (k 1) (t, u(t)) t=0 and ϕ l (z) = k=0 zk /(k + l)!. u(t) û(t) = e ta u 0 + p ( ) ϕ k ta t k u k. k=1 Exponential time differencing (ETD) Euler (p = 1): y n+1 = e ha y n + hϕ 1 (ha)g(t n, y n ). MIMS Nick Higham Computing the Action of the Matrix Exponential 5 / 29
Saad s Trick (1992) exp ϕ 1 (z) = ez 1. z ([ ]) A b = 0 0 [ ] e A ϕ 1 (A)b 0 1 MIMS Nick Higham Computing the Action of the Matrix Exponential 6 / 29
Theorem (Al-Mohy & H, 2010) Let A C n n, U = [u 1, u 2,...,u p ] C n p, τ C, and define [ ] [ ] A U 0 B = C (n+p) (n+p) Ip 1, J = C p p. 0 J 0 0 Then for X = e τb we have X(1: n, n + j) = jk=1 τ k ϕ k (τa)u j k+1, j = 1: p.
Theorem (Al-Mohy & H, 2010) Let A C n n, U = [u 1, u 2,...,u p ] C n p, τ C, and define [ ] [ ] A U 0 B = C (n+p) (n+p) Ip 1, J = C p p. 0 J 0 0 Then for X = e τb we have X(1: n, n + j) = jk=1 τ k ϕ k (τa)u j k+1, j = 1: p. Completely removes the need to evaluate ϕ k functions!
Implementation of Exponential Integrators We compute û(t) = e ta u 0 + as, with U = [u p,...,u 1 ], û(t) = [ I n p ( ) ϕ k ta t k u k k=1 0 ] ( [ A ηu exp t 0 J ]) [ u 0 η 1 e p ]. Choose to avoid overscaling. η = 2 log 2 ( U 1) MIMS Nick Higham Computing the Action of the Matrix Exponential 8 / 29
Formulae for e A, A C n n Power series Limit Scaling and squaring I + A + A2 2! + A3 3! + lim (I + A/s)s (e s A/2s ) 2s Cauchy integral Jordan form Interpolation 1 n i 1 e z (zi A) 1 dz Zdiag(e J k )Z 1 f[λ 1,..., λ i ] (A λ j I) 2πi Γ i=1 j=1 Differential system Schur form Padé approximation Y (t) = AY(t), Y(0) = I Qe T Q p km (A)q km (A) 1 Krylov methods: Arnoldi fact. AQ k = Q k H k + h k+1,k q k+1 e T k with Hessenberg H: e A b Q k e H k Q k b. MIMS Nick Higham Computing the Action of the Matrix Exponential 9 / 29
Scaling and Squaring Method B A/2 i so B 1 r m (B) = [m/m] Padé approximant to e B X = r m (B) 2i e A MATLABexpm uses alg of H (2005). Improved algorithm: Al-Mohy & H (2009). Can we adapt this approach for e A B? MIMS Nick Higham Computing the Action of the Matrix Exponential 10 / 29
Computing e A B }{{} A, }{{} B, n 0 n. Exploit, for integer s, n n n n 0 e A B = (e s 1A ) s B = e s 1A e s 1A e s 1 A }{{} B. s times Choose s so T m (s 1 A) = m (s 1 A) j j=0 j! e s 1A. Then B i+1 = T m (s 1 A)B i, i = 0: s 1, B 0 = B yields B s e A B. MIMS Nick Higham Computing the Action of the Matrix Exponential 11 / 29
Computing e A B }{{} A, }{{} B, n 0 n. Exploit, for integer s, n n n n 0 e A B = (e s 1A ) s B = e s 1A e s 1A e s 1 A }{{} B. s times Choose s so T m (s 1 A) = m (s 1 A) j j=0 j! e s 1A. Then B i+1 = T m (s 1 A)B i, i = 0: s 1, B 0 = B yields B s e A B. How to choose s and m? MIMS Nick Higham Computing the Action of the Matrix Exponential 11 / 29
Backward Error Analysis Lemma (Al-Mohy & H, 2009) T m (s 1 A) s B = e A+ A B, where A = sh m+1 (s 1 A) and h m+1 (x) = log(e x T m (x)) = k=m+1 c k x k. Moreover, h m+1 (A) k=m+1 c k α p (A) k if m + 1 p(p 1), where α p (A) = max(d p, d p+1 ), d p = A p 1/p. MIMS Nick Higham Computing the Action of the Matrix Exponential 12 / 29
Why Use d p = A p 1/p? A k A pk 1/p ( A p 1/p) k. ρ(a) A p 1/p A. With A = [ ] 0.9 500 : 0 0.5 k 2 5 10 A k 1 2.0e2 2.2e2 1.2e2 A k 1 2.5e5 3.1e13 9.8e26 d2 k = ( A 2 1/2 ) k 1 2.0e2 5.7e5 3.2e11 d3 k = ( A 3 1/3 ) k 1 4.5e1 1.3e4 1.9e8 Cheaply estimate A k, for a few k (H & Tisseur, 2001). MIMS Nick Higham Computing the Action of the Matrix Exponential 13 / 29
Why Use d p = A p 1/p? cont. d p = A p 1/p provide information about the nonnormality of A. Their use helps avoid overscaling. What other uses do they have? MIMS Nick Higham Computing the Action of the Matrix Exponential 14 / 29
Choice of s and m α p (A) := max ( d p, d p+1 ) θ m := max { θ : k=m+1 c k θ k 1 ǫ } A A ǫ if m + 1 p(p 1) and s 1 α p (A) θ m. Computational cost for B s e A B is matrix products. C m (A) = m max ( α p (A)/θ m, 1 ) Cost decreases with m. Restrict 2 p p max, p(p 1) 1 m m max. Minimize cost over p, m MIMS Nick Higham Computing the Action of the Matrix Exponential 15 / 29
Size of Taylor Series Argument Constants θ m (via symbolic, high precision): m 10 20 30 40 55 single 1.0e0 3.6e0 6.3e0 9.1e0 1.3e1 double 1.4e-1 1.4e0 3.5e0 6.0e0 9.9e0 MIMS Nick Higham Computing the Action of the Matrix Exponential 16 / 29
Preprocessing Expand the Taylor series about µ C: e µ (A µi) k /k! k=0 Choose µ so A µi A. Alg is based on 1-norm, but minimizing A µi F does better empirically at minimizing d p (A µi). Recover e A from e µ[ T m (s 1 (A µi)) ] s, [ e µ/s T m (s 1 (A µi)) ] s. First expression prone to overflow, so prefer second. Balancing is an option. MIMS Nick Higham Computing the Action of the Matrix Exponential 17 / 29
Termination Criterion In evaluating T m (s 1 A)B i = m (s 1 A) j j=0 j! B i we accept T k (A)B i for the first k such that A k 1 B i (k 1)! + Ak B i k! ǫ T k (A)B i. MIMS Nick Higham Computing the Action of the Matrix Exponential 18 / 29
Algorithm for F = e ta B 1 µ = trace(a)/n 2 A = A µi 3 [m, s] = parameters(ta) 4 F = B, η = e tµ/s 5 for i = 1: s 6 c 1 = B 7 for j = 1: m 8 B = t AB/(sj), c 2 = B 9 F = F + B 10 if c 1 + c 2 tol F, quit, end 11 c 1 = c 2 12 end 13 F = ηf, B = F 14 end MIMS Nick Higham Computing the Action of the Matrix Exponential 19 / 29
George Forsythe Pitfalls (1970) MIMS Nick Higham Computing the Action of the Matrix Exponential 20 / 29
Conditioning of e A B κ exp (A, B) ea F B F e A B F (1 + κ exp (A)). A 2 κ exp (A) e A 2 A 2 e A 2 Relative forward error due to roundoff bounded by ue A 2 B 2 / e A B F. A normal implies κ exp (A) = A 2. Then instability if e A 2 e A 2. A Hermitian implies spectrum of A n 1 trace(a)i has λ max = λ min (normwise) stability! MIMS Nick Higham Computing the Action of the Matrix Exponential 21 / 29
e ta B for Several t Grid: t k = t 0 + kh, k = 0: q, where h = (t q t 0 )/q. Evaluate B k = e tka B, k = 0: q, directly. Form B 0 = e t0a B and then B k = e kha B 0, k = 1: q B 2 = e 2hA B 0 (1) Form B 0 = e t0a B and then B k = e ha B k 1, k = 1: q B 2 = e ha (e ha B 0 ) (2) Suffers from overscaling when h is small enough. MIMS Nick Higham Computing the Action of the Matrix Exponential 22 / 29
e ta B for Several t Grid: t k = t 0 + kh, k = 0: q, where h = (t q t 0 )/q. Evaluate B k = e t ka B, k = 0: q, directly. Form B 0 = e t 0A B and then B k = e kha B 0, k = 1: q B 2 = e 2hA B 0 (1) Form B 0 = e t 0A B and then B k = e ha B k 1, k = 1: q B 2 = e ha (e ha B 0 ) (2) Suffers from overscaling when h is small enough. x e x (1 + x) e x (1 + x/2) 2 9.9e-9 2.2e-16 6.7e-16 8.9e-9 0 6.7e-16 MIMS Nick Higham Computing the Action of the Matrix Exponential 22 / 29
e ta B for Several t Grid: t k = t 0 + kh, k = 0: q, where h = (t q t 0 )/q. Evaluate B k = e tka B, k = 0: q, directly. Form B 0 = e t0a B and then B k = e kha B 0, k = 1: q B 2 = e 2hA B 0 (1) Form B 0 = e t0a B and then B k = e ha B k 1, k = 1: q B 2 = e ha (e ha B 0 ) (2) Suffers from overscaling when h is small enough. Use (1) when no cost penalty (no scaling), else (2). Opportunity to save and re-use some matrix products. MIMS Nick Higham Computing the Action of the Matrix Exponential 22 / 29
The Sixth Dubious Way (1) Moler & Van Loan (1978, 2003) MIMS Nick Higham Computing the Action of the Matrix Exponential 23 / 29
The Sixth Dubious Way (2) Advantages of our method over the one-step ODE integrator: Fully exploits the linearity of the ODE. Backward error based; ODE integrator control local (forward) errors. Overscaling avoided. MIMS Nick Higham Computing the Action of the Matrix Exponential 24 / 29
Experiment 1 Trefethen, Weideman & Schmelzer (2006): A R 9801 9801, 2D Laplacian, -2500*gallery( poisson,99). Compute e αta b for 100 equally spaced t [0, 1]. tol = u d. α = 0.02 α = 1 speed cost diff speed cost diff Alg AH 1 1119 1 49544 expv 46.6 25575 4.5e-15 66.0 516429 6.2e-14 phipm 10.5 10744 5.5e-15 9.3 150081 6.3e-14 rational 107.8 700 9.1e-14 7.9 700 1.0e-12 MIMS Nick Higham Computing the Action of the Matrix Exponential 25 / 29
Experiment 2 A =-gallery( triw,20,4.1), b i = cos i, tol = u d. 10 5 10 0 e ta b 2 10 5 10 10 Alg AH phipm rational expv 0 20 40 60 80 100 t MIMS Nick Higham Computing the Action of the Matrix Exponential 26 / 29
Experiment 3 Harwell Boeing matrices: orani678, n = 2529, t = 100, b = [1, 1,...,1] T ; bcspwr10, n = 5300, t = 10, b = [1, 0,...,0, 1] T. 2D Laplacian matrix,poisson. tol = u s. Alg AH ode15s time cost error time cost error orani678 0.13 878 4e-8 136 7780+ 2e-6 bcspwr10 0.021 215 7e-7 2.92 1890+ 5e-5 poisson 3.76 29255 2e-6 2.48 402+ 8e-6 4 poisson 15 116849 9e-6 3.24 49+ 1e-1 MIMS Nick Higham Computing the Action of the Matrix Exponential 27 / 29
Comparison with Krylov Methods Alg AH Most time spent in matrix vector products. Direct method, cost predictable. No parameters to estimate. Storage: 2 vectors Evaluation of e At at multiple points on interval. Can handle mult col B. Cost tends to with A. Krylov Methods Krylov recurrence and e H can be significant. Iterative method; needs stopping test. Select Krylov subspace size. Storage: Krylov basis Need block Krylov method. Some A dependence. MIMS Nick Higham Computing the Action of the Matrix Exponential 28 / 29
Conclusions e A B = (e s 1A ) s B T m (s 1 A)...T m (s 1 A) B. }{{} s times Key ideas: s and m to achieve b err bound; terminate Taylor series prematurely; shifting and balancing; compute e t ka B on grid, avoiding overscaling. Alg AH is best method in our experiments. Very easy to implement. Exploits optimized matrix products. Attractive for exponential integrators using theorem on avoidance of ϕ functions. A. H. Al-Mohy and N. J. Higham, MIMS EPrint 2010.30, March 2010. Paper and MATLAB code. MIMS Nick Higham Computing the Action of the Matrix Exponential 29 / 29
References I A. H. Al-Mohy and N. J. Higham. A new scaling and squaring algorithm for the matrix exponential. SIAM J. Matrix Anal. Appl., 31(3):970 989, 2009. G. E. Forsythe. Pitfalls in computation, or why a math book isn t enough. Amer. Math. Monthly, 77(9):931 956, 1970. N. J. Higham. The Matrix Function Toolbox. http: //www.ma.man.ac.uk/~higham/mftoolbox. MIMS Nick Higham Computing the Action of the Matrix Exponential 26 / 29
References II N. J. Higham. The scaling and squaring method for the matrix exponential revisited. SIAM J. Matrix Anal. Appl., 26(4):1179 1193, 2005. N. J. Higham. Functions of Matrices: Theory and Computation. Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 2008. ISBN 978-0-898716-46-7. xx+425 pp. N. J. Higham and A. H. Al-Mohy. Computing matrix functions. Acta Numerica, 19:159 208, 2010. MIMS Nick Higham Computing the Action of the Matrix Exponential 27 / 29
References III N. J. Higham and F. Tisseur. A block algorithm for matrix 1-norm estimation, with an application to 1-norm pseudospectra. SIAM J. Matrix Anal. Appl., 21(4):1185 1201, 2000. C. B. Moler and C. F. Van Loan. Nineteen dubious ways to compute the exponential of a matrix. SIAM Rev., 20(4):801 836, 1978. C. B. Moler and C. F. Van Loan. Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later. SIAM Rev., 45(1):3 49, 2003. MIMS Nick Higham Computing the Action of the Matrix Exponential 28 / 29
References IV L. N. Trefethen, J. A. C. Weideman, and T. Schmelzer. Talbot quadratures and rational approximations. BIT, 46(3):653 670, 2006. MIMS Nick Higham Computing the Action of the Matrix Exponential 29 / 29