The Leja method revisited: backward error analysis for the matrix exponential

Transcription

1 The Leja method revisited: backward error analysis for the matrix exponential M. Caliari 1, P. Kandolf 2, A. Ostermann 2, and S. Rainer 2 1 Dipartimento di Informatica, Università di Verona, Italy 2 Institut für Mathematik, Universität Innsbruck, Austria arxiv: v1 [math.na] 29 Jun 2015 December 7, 2018 The Leja method is a polynomial interpolation procedure that can be used to compute matrix functions. In particular, computing the action of the matrix exponential on a given vector is a typical application. This quantity is required, e.g., in exponential integrators. The Leja method essentially depends on three parameters: the scaling parameter, the location of the interpolation points, and the degree of interpolation. We present here a backward error analysis that allows us to determine these three parameters as a function of the prescribed accuracy. Additional aspects that are required for an efficient and reliable implementation are discussed. Numerical examples that illustrate the performance of our Matlab code are included. Mathematics Subject Classification (2010): 65F60, 65D05, 65F30 Key words: Leja interpolation, backward error analysis, action of matrix exponential, exponential integrators, ϕ functions, Taylor series 1. Introduction In many fields of science the computation of the action of the matrix exponential is of great importance. As one example amongst others we highlight exponential integrators. These methods constitute a competitive tool for the numerical solution of stiff and highly oscillatory problems, see [Hochbruck et al. 2010]. Their efficient implementation heavily relies on the fast computation of the action of certain matrix functions among those the matrix exponential is the most prominent one. Recipient of a DOC Fellowship of the Austrian Academy of Science at the Department of Mathematics, University of Innsbruck, Austria 1

2 Given a square matrix A and a vector v the action of the matrix exponential is denoted by e A v. In general, the exponential of a sparse matrix A is a full matrix. Therefore, it is not appropriate to form e A and multiply by v for large scale matrices. The aim of this paper is to define a backward stable method to compute the action of the matrix exponential based on the Leja interpolation. The performed backward error analysis allows one to predict and reduce the cost of the algorithm resulting in a more robust and efficient method. For a given matrix A and vector v, one chooses a positive integer s so that the exponential e s 1A v can be well-approximated. Due to the functional equation of the exponential we can then exploit the relation resulting in an recursive approximation of e A v = v (s) by e A v = ( e s 1 A ) s v (1) v (i+1) = e s 1A v (i), v (0) = v. (2) There are various possibilities to compute the stages v (i). Usually, this computation is based on interpolation techniques. The best studied methods comprise Krylov subspace methods (see [Sidje 1998] and [Niesen et al. 2012]), truncated Taylor series expansion [Al-Mohy and Higham 2011], and interpolation at Leja points (see [Caliari et al. 2004; 2014]). In this paper we take a closer look on the Leja method (cf. Appendix A (27) and (28)) for the approximation v (i+1) L m,c (s 1 A) s v (i). Nevertheless, for the performed analysis the only requirement on the interpolation method is that zero is among the interpolation points. In this paper we present two different ways of performing a backward error analysis of the Leja method. This analysis indicates how the scaling parameter s, the degree of interpolation m and the interpolation interval [ c, c] can be selected in order to obtain an appropriately bounded backward error by still keeping the cost of the algorithm at a minimum. Furthermore, we discuss how the method benefits from a shift of the matrix to keep the cost low, and we show how a premature termination of the Leja method can help to reduce the cost in an actual computation. As a last step we illustrate the stability and behavior of the method by some numerical experiments. The paper is structured in the following way. In Section 2 we introduce the backward error analysis and draw some conclusions from it. In particular we show how this helps us to select the parameters s, m, and c. In Section 3 we discuss some additional aspects for a successful implementation based on the Leja method. Section 4 presents some numerical examples dealing with different features and benchmarks for the method. In Section 5 we give a discussion of the presented results. For a reader not familiar with the Leja method we included a brief description in Appendix A. 2. Backward error analysis In this section we concentrate on the backward error of the action of the matrix exponential when approximated by the Leja method. In the following we will denote the Leja 2

3 method by L m,c (cf. Appendix A, in particular (27) and (28)) where m is the degree of interpolation and the interpolation takes place on the (real) interval [ c, c]. The concept of a backward error analysis goes back to Wilkinson, see [Wilkinson 1961]. The idea is that one interprets the result of the interpolation L m,c (A)v as the exact solution of a perturbed problem e A+ A v. The perturbation A corresponds to the backward error and we aim to satisfy A tol A. As a consequence of this bound, one can describe how the scaling factor s (always a positive integer) has to be selected in order to bound the backward error of the interpolation by a given tolerance tol (for a certain degree of interpolation m). The here presented backward error analysis exploits a variation of the analysis given in [Al-Mohy and Higham 2011]. We define the set Ω m,c = {X C n n : ρ(e X L m,c (X) I) < 1}, where ρ denotes the spectral radius and L m,c is the Leja interpolation polynomial of degree m on the symmetric interval [ c, c], see (27) and (28). Note that Ω m,c is open and contains a neighborhood of 0 C n n for m 3. For X Ω m,c we define the function h m+1,c (X) = log(e X L m,c (X)). (3) Here log denotes the principal logarithm [Higham 2008, Thm. 1.31]. Furthermore, log(x) commutes with X. As a result we get L m,c (X) = e X+h m+1,c(x) for X Ω m,c. By introducing a scaling factor s such that s 1 A Ω m,c for A C n n we obtain L m,c (s 1 A) s = e A+sh m+1,c(s 1 A) =: e A+ A, (4) where A = sh m+1,c (s 1 A) represents the backward error resulting form the approximation of e A by the Leja method L m,c (s 1 A) s. On the set Ω m,c the function h m+1,c has a series expansion of the form h m+1,c (X) = a k,c X k. (5) In order to bound the backward error by a specified tolerance we need to ensure A A k=0 = h m+1,c(s 1 A) s 1 A tol (6) for a given tolerance tol and a chosen matrix norm. In contrast to [Al-Mohy and Higham 2011] we have the interpolation interval [ c, c] as an additional parameter to m and s for our analysis. In the following, we are going to introduce two different ways of bounding the backward error with the help of the series expansion in (5). The first one is closely related the analysis presented in [Al-Mohy and Higham 2011]. There, the idea is to study the convergence of the method for fixed m and tol. We aim at finding a circle at zero with the following property: if the norm of the matrix is smaller than the radius of the circle (6) holds. The second approach does the same, however, we replace the circles by ellipses. The later describe better the regime of convergence for interpolation on intervals. An estimate of the field of values for the matrix is used to scale the matrix into the ellipse. 3

4 2.1. Convergence in discs In this section we investigate bounds on the backward error represented by h m+1,c. The analysis is based on discs in the complex plane. Starting from (5) we can bound h m+1,c (X) by h m+1,c (X) = a k,c X k a k,c X k =: h m+1,c ( X ). (7) Inserting this into (6) we get k=0 k=0 A A = h m+1,c(s 1 A) s 1 A h m+1,c (s 1 A ) s 1. (8) A If we assume that for our interpolation method zero is among the interpolation points we get hm+1,c (θ) θ = a k,c θ k 1. k=1 This is a monotonically increasing function for θ 0. Note that, for c = 0, the Leja interpolation reduces to the Taylor series at zero. We thus have a 1,0 = 0 for m 1. The equation hm+1,c (θ) θ = tol (9) therefore has a unique positive root for c sufficiently small. In the following we call this root θ m,c. The number θ m,c can be interpreted in the following way: for the interpolation of degree m in [ c, c] the backward error fulfills A tol A, if the positive integer s fulfills s 1 A θ m,c. In other words, consider a circle of radius θ m,c. If the norm of a matrix is smaller than θ m,c, the interpolation of degree m with points in [ c, c] has an error less than or equal to tol. As mentioned earlier we are interested in the Leja method and therefore select as interpolation points the sequence of Leja points m 1 ξ m arg max ξ ξ j, m 1, ξ 0 given, ξ [ c,c] j=0 see (26). In order to compute h m+1,c in a stable manner we expand h m+1,c in terms of the Newton basis for Leja points in [ c, c] as h m+1,c (X) = k=m+1 k 1 h m+1,c [ξ 0,..., ξ k ] (X ξ j I), (10) j=0 4

5 where h m+1,c [ξ 0,..., ξ k ] denotes the kth divided difference. The above series starts with k = m + 1 as h m+1,c (ξ j ) = 0 for j = 0,..., m. If we rewrite this in the monomial basis we obtain (7) with the according coefficients a k,c. In Figure 1 we can see the behavior of θ m,c for the Leja interpolation with respect to c for m up to 100 and tol = 2 53, computed in high accuracy. For an actual computation one has to truncate the series (10) at some index M. We always used M = 3m c = θ θ m = 85 m = 80 m = 75 m = 70 m = 65 m = m = 25 m = 20 m = 15 m = 30 m = 35 m = 40 m = 45 m = 50 m = c Figure 1: Right endpoint c of the interpolation interval versus the radius of convergence θ = θ m,c with tolerance 2 53 for real Leja points in [ c, c]. Along each line the interpolation degree m is fixed. As we now have a way of bounding the backward error we discuss the choice of the number of iterations in (2). We propose to select the integer s depending on m and c in such a way that the cost of the algorithm becomes minimal. We have the limitation that m is bounded by 100 to avoid problems with over- and underflow. Nevertheless, we get several possibilities to select the free parameter c describing the interval. For a fixed m, c could be chosen in such a way that θ m,c is maximal. This corresponds to the interpolation interval with maximal radius of convergence. From Figure 1 we can see that the radius of convergence can be significantly larger than the right endpoint c of the interpolation interval. A second possibility is to set c = 0. The value θ m,0 corresponds to the truncated Taylor series expansion as described in [Al-Mohy and Higham 2011]. A third possibility is to select s such that the right endpoint c of the interpolation interval and the radius of convergence coincide. These are the points on the diagonal in Figure 1. 5

6 m half 6.43e e e e e e e+00 single 9.62e e e e e e e+00 double 1.74e e e e e e e+00 m half 1.00e e e e e e e+01 single 8.71e e e e e e e+01 double 5.48e e e e e e e+01 m half 1.84e e e e e e+01 single 1.76e e e e e e+01 double 1.46e e e e e e+01 Table 1: Samples of the (rounded) values θ m with tolerances half, single and double for the real Leja interpolation. A priory none of the above choices can be seen to be optimal for an arbitrary matrix A. Numerical experiments, however, show that, for a Jordan block with eigenvalue zero, the choice c = 0 is superior. On the other hand, if we multiply the Jordan block by 10 the first choice of c leads to better results. Last, for a normal matrix with eigenvalues equally distributed in an interval symmetric around the origin the third choice is preferable. We want to focus on normal matrices. Therefore, we choose c according to the third option. More precisely, we select θ m = min{c: θ m,c = c} (11) and consequently the interpolation interval in such a way that the radius of convergence coincides with the right endpoint of the interpolation interval. This means that θ m is the first intersection point of the graph (c, θ m,c ) with the diagonal, cf. Figure 1. Furthermore, if s 1 A θ m then s 1 A θ m,c for all 0 c θ m and we can safely interpolate with degree m for all intervals [ c, c] with 0 c θ m. For a fixed m the curve θ m,c does not vary much for 0 c θ m. In particular for m = 50 and tol = 2 53 we get (rounded) θ 50,0 = 8.55, max c {θ 50,c } = 8.77 and θ 50 = A further motivation for the choice of θ m is given by the early termination criterion we want to employ. This will be further specified in Section 3.2. We compute θ m by the combination of two root finding algorithms based on Newton s method. The inner equation (9) to compute θ m,c is solved by an exact Newton iteration up to 20 digits. The outer loop solving θ m,c = c is also solved by Newton s method but the necessary derivative is approximated by numerical differentiation. We compute the result up to 18 digits. The resulting values are truncated to 16 digits and used henceforth as the θ m values. In Table 1 we listed selected (rounded) values of θ m for varying m and certain tolerances. Now, for each m the optimal value of the integer s is given by s = A /θ m. (12) 6

7 The cost of the interpolation is dominated by the number of matrix-vector products computed during the Newton interpolation. Therefore, the cost is at most C m (A) := sm = m A /θ m, (13) resulting in the optimal m and corresponding s as { } A m = arg min m, s = 2 m m max θ m A θ m. (14) We recall that by the help of Gershgorin s disk theorem we can enclose the spectrum of A in a rectangle R with vertices (α, β), (α, η), (ν, β), (ν, η), see (25). Relating the circle to the rectangle R, it is clear that if R is close to a square it fits best into a circle. On the other hand, if the height-to-width ratio of R is smaller than 1 a circle is not the best possible shape. This is notably true for the limit case where R is a line segment. As we have the interpolation interval as an additional parameter we can improve the analysis for these examples. In the next section we will investigate one possibility to do this. Remark 2.1 (precomputed divided differences). As we now have a fixed discrete set of interpolation intervals given by θ m the associated divided differences can be precomputed, once and for all. Remark 2.2 (nonnormal matrices). As shown in [Al-Mohy and Higham 2011] the truncated Taylor series method is able to exploit the values d p = A p 1/p. For nonnormal matrices d p decreases. This fact can be beneficial for the implementation. Our method is not able to use this right away due to the fact that the series representation of h m+1,c starts at k = 1, see (7). As a result, the method might use more scaling steps for such problems, see Section 4 for some experiments. Nevertheless, the values d p can be of use for the implementation, see Section 3.3. Furthermore, if R is a square the interpolation interval could be contracted to a single point. This means the method is basically the truncated Taylor series method and d p could be used Convergence in ellipses The Leja method as defined in [Caliari et al. 2004] is based on ellipses and the convergence is studied in ellipses. Furthermore, in many of our applications an ellipse also represents the spectrum of the matrix better than a disc. In this section we investigate bounds on the backward error represented by h m+1,c based on ellipses in the complex plane rather than circles. Again our aim is to find a bound for h m+1,c. Due to the fact that zero is among the interpolations points for m 2 we can write h m+1,c as h m+1,c (X) = Xg m+1,c (X). (15) 7

8 For a fixed ε and by considering the Euclidean norm we can rewrite (6) as A 2 = h m+1,c(s 1 A) 2 A 2 s 1 g m+1,c (s 1 A) 2 A 2 1 g m+1,c (z)(zi s 1 A) 1 dz 2πi Γ L(Γ) 2πε g m+1,c Γ. 2 (16) Here Γ = K denotes the boundary of the domain K that contains Λ ε (s 1 A), the ε-pseudospectrum of s 1 A. The ε-pseudospectrum of a matrix X is given by Λ ε (X) = { z : (zi X) 1 2 ε 1}. The length of Γ is denoted by L(Γ) and Γ is the infinity norm on Γ. For given m, c, and K the last term in (16) can be computed in high precision. From now on we assume that K is an ellipse symmetric with respect to the origin and enclosing Λ ε (s 1 A). We do not only have one ellipse for every m, but rather a family of confocal ellipses with boundaries Γ γ,c (depending on the capacity γ and the focal interval [ c, c]) given by Γ γ,c = } {z C: z = γw + c2 4γw, w = 1. (17) An ellipse Γ γ,c will be considered a valid region of interpolation if L(Γ γ,c ) 2πε g m+1,c Γγ,c tol (18) is satisfied. Furthermore, for each c we select the ellipse with largest capacity γ =: γ m,c satisfying (18) as tolerance equality. This reduces the families of ellipses to a oneparameter family. For our backward error we can interpret the value γ m,c in the following way. If we prescribe an interval [ c, c] and select m+1 Leja points in this interval, we have A tol A under the assumption that s 1 is selected such that Λ ε (s 1 A) conv(γ γm,c,c). Before we describe the selection of confocal intervals we show how we can use the ellipse to compute s for a given matrix A C n n. We recall that by the help of Gershgorin s disk theorem we can enclose the spectrum of A in a rectangle R with vertices (α, β), (α, η), (ν, β), (ν, η), see (25). Furthermore, we assume that the rectangle is centred in zero ( α = ν and η = β), otherwise we shift the matrix accordingly. In order to keep the notation simple we consider a single ellipse K with boundary Γ, focal interval [ c, c] and capacity γ for which (18) is satisfied. Let ε denote the open disc of radius ε around the origin. Hence, we have the following chain of inclusions Λ ε (A) W(A) + ε R + ε. 8

9 (ν + ε, β + ε) = r ε R Γ S 1 r ε s 1 R s 1 β -c 0 c s 1 ν ε Figure 2: Illustration on the selection of the correct scaling factor s to ensure the enclosure of the estimated field of values s 1 (R + ε ) in the ellipse K in R 2 with boundary Γ. We have S = (ν + ε) 2 a 2 + (β + ε) 2 b 2 and s = S. The first inclusion connecting the pseudspectrum and the field of values can be found in [Trefethen and Embree 2005]. The above inclusions state that if R + ε K then A 2 tol A 2. Our aim is to determine the correct scaling factor s such that the inclusion s 1 (R + ε ) K is valid. We do this by computing the intersection of Γ with the straight line through zero and r ε = (ν + ε, β + ε), the upper right vertex of the rectangle extended by ε. The procedure is illustrated in Figure 2. If a = γ + c2 4γ, b = γ c2 4γ are the semi-axes of Γ then s = (ν + ε) 2 a 2 + (β + ε)2 b 2. (19) Due to our choice of r ε it holds that s 1 (R + ε ) K for s 1 r ε K. Finally we discuss the selection of the free parameter c. For our algorithm we select a discrete set of focal intervals for every m. More precisely, we use the known values θ j for j m from (11) and Table 1, respectively. For these values we have precomputed divided differences at hand. For the intervals [ θ j, θ j ] we compute ellipses fulfilling (18) until the semi-minor axis of the resulting ellipse is sufficiently small or we exceed a maximum length of the interpolation interval. In Figure 3 we can see several ellipses for m = 35 and tol = 2 53, where the dashed circle has radius θ 32 = 3.60 corresponding the the previous estimate; see Table 3 and Section 3.3 for further reasoning why to include this circle. We can see that for larger focal intervals the semi-minor axis decreases until we end up, in the limit, with an interval on the real axis. As a result, a matrix with spectrum on the real axis is better handled by ellipses than by circles. Therefore, it is possible to interpolate for certain matrices with fewer scaling steps than by the previous estimate. 9

10 z plane Figure 3: For ε = 1 50, m = 35 and tol = 2 53 the ellipses (17) satisfying (18) are shown for various focal intervals [ θ j, θ j ] for j = 35,..., 48. The value θ j is indicated on the ellipse. The dashed circle has radius θ 32 = 3.60, representing the previous estimate. It is the largest circle in lieu of the ellipse Γ γ,c that fulfills (18) for m = 35. Remark 2.3. If we reduce the size of the focal interval of our ellipses Γ m,c to a point, we end up with a circle. In fact, for a fixed m this circle is slightly smaller than the one obtained by the estimate θ m. The selection of the optimal degree of interpolation to minimize the cost is done in the following way. As discussed above and illustrated in Figure 3, for every m we get a family of ellipses with semi-axes a m,θj = γ m,θj + θ2 j 4γ m,θj and b m,θj = γ m,θj θ2 j 4γ m,θj, (20) where γ m,θj fulfils (18). We now use (19) to select the optimal ellipse for each m. In this family the optimal ellipse is identified as the one with the fewest number of scaling steps. For these optimal ellipses the number of scaling steps is given by (ν ) s m = min + ε 2 ( ) β + ε 2 j m + a m,θj b m,θj. Now we can minimize with a similar cost function as (13) over m and obtain our optimal 10

11 m half 1.94e e e e e+01 single 8.11e e e e e+01 double 1.16e e e e e+00 m half 1.46e e e e e+01 single 1.27e e e e e+01 double 9.57e e e e e+01 Table 2: Samples of the (rounded) values θ m with tolerances half, single and double for complex conjugate Leja interpolation. degree of interpolation m and scaling factor s as m = 2.3. Symmetrized complex Leja points arg min {ms m }, s = s m. (21) 2 m m max All the statements made in the previous discussion remain valid if we use complex conjugate Leja points. The advantage of such points is to better handle matrices having eigenvalues with dominant imaginary parts. This is characterized by a height-to-width ratio of more than 1 for the rectangle R. Examples include the (real) discretization matrices of transport equations or the discretization of the Schrödinger operator (a complex matrix) which has eigenvalues on the negative imaginary axis. Symmetrized or conjugate complex Leja points are defined on an interval on the imaginary axis, we namely use i [ c, c], as m 1 ξ m arg max ξ ξ j, ξ m+1 = ξ m for m 1 odd, and ξ 0 = 0, ξ i[ c,c] j=0 cf. (26) for the real case. We use conjugate complex pairs of points rather than standard Leja points in an interval along the imaginary axis as this allows real arithmetic for real arguments. This gives rise to polynomials of even degree. To facilitate conjugate complex Leja points in our backward error analysis we only need to change the actual computation of the values θ m,c, θ m and γ m,c. The theory stays the same. In Figure 4 we can see the characteristics of θ m,c for complex conjugate Leja points in i [ c, c]. Table 2 displays a selection of (rounded) θ m values for various tolerances. If we apply complex conjugate Leja points in the framework of Section 2.2 we get ellipses where the major axis is on the imaginary axis. Apart from that only the computed values change, compared to real Leja points Extension to ϕ functions The presented backward error analysis extends in a straight forward way to the ϕ functions that play an important role in exponential integrators, see [Hochbruck et al. 2010]. 11

12 24 22 c = θ m = 90 θ m = 80 m = 70 m = 60 m = 50 m = 40 m = 30 m = c Figure 4: Defining value c of the interpolation interval versus the radius of convergence θ = θ m,c with tolerance 2 53 for complex conjugate Leja points in i[ c, c]. Along each line the degree of interpolation m is fixed. We illustrate this here for the ϕ 1 function. For A C n n and w C n we observe that ([ ]) [ ] A w 0 ϕ 1 (A)w = [I, 0] exp, see [Sidje 1998]. For the choice A = [ ] A w 0 0 and v = the backward error is preserving the structure, i.e. [ ] A w A =. 0 0 Thus the above analysis applies. The shown procedure can be extended to general ϕ functions with the help of [Al-Mohy and Higham 2011, Thm. 2.1]. 3. Additional aspects of interpolation By using the previously described backward error analysis to compute the values m and s a working algorithm can be defined. Nevertheless, the performance of the algo- [ ]

13 rithm can be significantly improved by some preprocessing and by introducing an early termination criterion. In addition we need to consider that, for an actual computation, we do not compute in exact arithmetic. In some cases it is possible that, due to the hump phenomenon, the interpolation might suffer from roundoff errors. We address these issues and a remedy to the hump phenomenon in this section Spectral bounds and shift In the above backward error analysis, it was assumed that the rectangle R lies symmetrically about the origin. This requires a shift of A. On the other hand, it is clear that a well chosen shift µ satisfying A µi A is beneficial for the interpolation (lower degree or less scaling steps required). For the exponential function such a shift can easily be compensated by scaling. More precisely, if the shift µ is selected, we use [e µ/s L m,c (s 1 (A µi))] s for the iterative computation (2). This is also less likely to suffer from overflow, see [Al-Mohy and Higham 2011]. For our algorithm a straight forward shift is to centre the rectangle R in the origin, namely µ = α + ν 2 + i η + β. (22) 2 If A R n n then η = β and µ R. Therefore, a complex shift is only applied to complex matrices. It is easy to see that for a Hermitian or skew Hermitian A the proposed shift (22) coincides with the maximum norm minimizing shift presented in [Higham 2008, Thm. 4.21]. For a general matrix the shift somewhat symmetrizes the spectrum of the matrix with regard to its estimated field of values. The shift µ = trace(a)n 1 used in [Al-Mohy and Higham 2011] is a transformation that centers the spectrum of the matrix around the average eigenvalue. In many cases the two shifts are the same. Nevertheless, it is possible to find examples where one shift leads to better results than the other. The example matrix one-sided of Example 4 is one of these cases. For the trace shift a symmetrization of the rectangle R might be required, resulting in an potential increase of scaling steps for the estimate based on (16). For the method proposed here, we always use (22) as shift Early termination criterium The estimates based on (11) and (18) are worst case estimates and in particular do not take v into account. As a result, the choice of m is likely to be an overestimate and can be reduced in the actual computation. By limiting m in the computation of L m,c (s 1 A)v (i) in (2) we introduce a relative forward error that again should be bounded 13

14 by the tolerance tol. Our choice of e k = L k,c (s 1 A)v (i) L k 1,c (s 1 A)v (i) k 1 = exp[ξ 0,..., ξ k ] (s 1 A ξ j I)v (i) tol s L k,c(s 1 A)v (i) (23) j=0 as an a posteriori error estimate for the Leja method in the kth step experimentally turns out to be a good choice. In contrast to the method described in [Al-Mohy and Higham 2011] we divide the tolerance by the amount of scaling steps. This potentially increases the number of interpolation steps per iteration but in practice results in a more stable computation for normal matrices, see Section 4. Nevertheless, it sometimes leads to results of higher accuracy than prescribed. In practice it is advisable to take the sum of two or three successive approximation steps for the estimate as this captures the behavior better. On the other hand, it can also be beneficial to only make the error estimate every couple of iterations rather than in each step to save computational cost, see [Caliari et al. 2014]. A second approach for an a posteriori error estimate for the Leja method based an the computation of a residual can be found in [Kandolf et al. 2014]. This procedure can also be used here. Furthermore, it is possible to adapt the early termination criterion to complex conjugate Leja points. With the help of an early termination criterion computational cost can be saved drastically, see Section Handling the hump phenomenon In the following we want to describe a problem occurring in interpolation when inexact arithmetic is used. Note that the hump we are going to describe is not the same as that described in [Moler and Van Loan 2003] for nonnormal matrices. To describe the problem let us consider the interpolation of e ix with x R. We approximate this by a truncated Tayler series at zero. In order to allow an early termination we can not use Horner s scheme. For x large enough the coefficients of the series grow before they eventually decay, thus the name hump. For example, for x = 10, the largest coefficient is about and therefore we can not expect more than 13 digits of accuracy for the computation of e i10 when performed by the truncated Taylor series expansion. For the Leja interpolation the behavior in such an example is much better. Due to the distribution of the points over the whole interpolation interval the largest coefficient in the sequence is not as large as for the Taylor method. In fact for an interpolation interval of [ 23.88, 23.88], which corresponds to the largest θ m value used with double precision in the ellipse estimate, the relative error in the interpolation can be seen in Figure 5. The maximal error on the interval is about leading to a total loss of at most 2 digits of accuracy. If one would like to have the same error for Taylor interpolation, x must be smaller than 6. The behavior for an interpolation with a real argument is similar. However, as we consider the relative error the result is scaled by the largest value and therefore the hump has less influence on the computation. 14

15 Lm,c(ix) e ix Re x Figure 5: Error of the scalar interpolation L m,c (ix) e ix for x [ 23.88, 23.88], c = and m = 100. Nevertheless, there is a second problem that might occur during the computation. If A strongly overestimates ρ(a), the choice of the interpolation interval might be too large. This is especially the case for nonnormal matrices. In such a case the Leja method produces a hump as described above as some of the interpolation points are far away from the spectrum. This undesired behavior can be improved by obtaining a better estimate of the spectral radius and consequently reducing the interpolation interval. For this we use the values d p = A p 1/p as defined in Remark 2.2. As long as the sequence of {d p } decreases, we adjust the interpolation interval accordingly. For Taylor interpolation a similar idea is used in [Al-Mohy and Higham 2011]. For the estimate based on (11) the reduction of the interpolation interval is possible due to the behavior of the θ m,c curve shown in Figure 1. However, if we use the estimate based on (16) the reduction of the interpolation interval is not straight forward. If we fit our rectangle R into an ellipse with semi-axis given by (20), then in general, R will not fit into an ellipse with a smaller interpolation interval [ c, c] where c θ j. We overcome this problem by adding a circle to the ellipses. We use the largest circle defined by a m,θk = b m,θk = θ k for some k m that fulfills (18), see Figure 3 for an example. If the values d p now indicate a reduction of the interval we restrict the ellipse estimate to these circles and perform a reduction of the interpolation interval. The validity of this process can be checked in the same manner as for θ m. Remark 3.1. In the current version the algorithm does not allow to reduce the number s along with the decay of d p as in [Al-Mohy and Higham 2011]. Nevertheless, if a drastic decay is indicated it is possible to transform our method into Taylor interpolation by setting c = Numerical examples In order to illustrate the behavior of our method we provide a variety of numerical examples. We use matrices resulting from the spatial discretization of time dependent 15

16 partial differential equations already used in [Caliari et al. 2014]. Furthermore, we also utilize examples from [Al-Mohy and Higham 2011] and certain prototypical examples to illustrate some specific behavior of the method. All our experiments are performed with Matlab 2013a. Our main focus is the number of matrix-vector products performed by the method. We will mainly compare our method to the function expmv of [Al-Mohy and Higham 2011]. If we speak of performed matrix-vector products (mv) for any of the methods we only count the matrix-vector products in the main algorithms and no preprocessing tasks. Note, however, that the divided differences are precomputed. In the following we are going to compare different variations of our algorithm based on the different ways to compute the scaling factor s and degree of interpolation m. Alg.1: Uses m and s given by (14). Alg.2: Uses m and s given by (21). In both algorithms, the early termination criterion (23) is used, as well as the shift and the hump test discussed in the previous section, if not indicated otherwise. For Alg.2 the hump test procedure is only employed if the estimate of s is based on the circles among the ellipses. In Example 1 we take a look on the stability of the methods with and without early termination, Examples 2 and 3 focus on the selection of m and c for the different variations of our algorithm and Example 4 investigates the behavior for multiple scaling steps, i.e., s > 1. Example 1 (early termination). This example is taken from [Al-Mohy and Higham 2011, Exp. 2] to show the influence of the early termination criterion for a specific problem. We use the matrix A as given by gallery( lesp,10). This is a nonsymmetric, tridiagonal matrix with real, negative eigenvalues. We compute e ta v by Alg.1 and Alg.2, respectively for 50 equally spaced time steps t [0, 100]. We select the tolerance tol = 2 53 and v i = i. As A is a nonnormal matrix Alg.2 is restricted to circles to allow the hump reduction procedure. The results of the experiments can be seen in Figure 6 where the solid line corresponds to the condition number (24) multiplied by the tolerance. We can not expect the algorithms to perform much better than indicated by this line. As condition number we use κ exp (ta, v) := exp(ta) 2 v 2 exp(ta)v 2 + (vt I)K exp (ta) 2 vec(ta) 2 exp(ta)v 2, (24) as defined in [Al-Mohy and Higham 2011, Eq. (4.2)]. Here vec denotes the vectorization operator that converts its matrix argument to a vector by traversing the matrix columnwise. Furthermore, let L(A, A) denote the Fréchet derivative of exp at A in direction A given by e A+ A = e A + L(A, A) + o( A ). 16

17 10 12 κ exp (ta, v)tol Alg.1 Alg.2 Alg.1* Alg.2* rel. error in t Figure 6: Time step t versus relative error in the 2-norm for the computation of e ta v with tolerance tol = The indicates that no early termination was used. With the relation vec(l(a, A)) = K exp (A) vec( A) the Fréchet derivative is given in its Kronecker form as K exp (A). In addition we use the relation vec(l(a, A)v) = (v T I) vec(l(a, A)). For the computation we use the function expm cond form the Matrix Function Toolbox, see Higham [2008]. Overall we can see that the algorithms behave in a forward stable manner for this example. The early termination criterion shows no significant increase in the error. Both algorithms are well below κ exp (ta, v) for all values of t. For this rather small matrix we used the exact norm and not a norm estimate to allow for a better comparison. Example 2 (advection-diffusion equation). In order to show the difference between the two suggested processes for selecting the interpolation interval for our algorithms we use an example that allows us to easily vary the spectral properties of the discretization matrix. Let us consider the advection-diffusion equation t u = a u + b u on the domain Ω = [0, 1] 2 with homogeneous Dirichlet boundary conditions. This problem is discretized in space by finite differences with grid size x = (N + 1) 1, N 1. As a result of the discretization we get a sparse N 2 N 2 matrix A. We define the grid Péclet number Pe = b x 2a as the ratio of advection to diffusion, scaled by x. By increasing Pe the nonnormality of the discretization matrix can be controlled. In addition, Pe describes the height-towidth ratio of the rectangle R. The estimates for α and ν stay constant but η = β increase with Pe. 17

18 Alg.1 Alg.2 expmv m m rel. err c m m rel. err c m m rel. err Pe = e e e-15 Pe = e e e-15 Pe = e e e-15 Pe = e e e-15 Pe = e e e-15 Pe = e e e-15 Table 3: For varying grid Péclet number in Example 2 the selection of the degree of interpolation m, the actual degree due to the early termination m and the right endpoint c of the interpolation interval are shown. We compute exp(ta)v with a time step t = 5e-3, discretization parameter N = 20 and tolerance tol = The error is measured relative to the result of expm in the maximum norm. For the following computations the parameters are chosen as: N = 20, a = 1 and b = 2aPe x. As a result, for Pe = 0 we get that R is an interval on the real axis and for Pe = 1 a square. For Pe = 0 the matrix is equal to -(N+1)^2*gallery( poisson,n). As v we use the discretization of the initial value u 0 (x, y) = 256 x 2 (1 x) 2 y 2 (1 y) 2. Table 3 gives the results of an experiment where we varied the grid Péclet number. We show the results of the different selection procedures, namely the degree of interpolation m and the interpolation interval [ c, c]. The time step t = 5e-3 is chosen such that expmv is able to compute the result without scaling the matrix. The actual degree of interpolation m and the relative error with respect to the method expm in the maximum norm are shown. As the maximum norm of the matrix stays the same (for fixed N) the parameters of expmv and Alg.1 are always the same. For Pe = 1 the eigenvalues of the matrix ta are in a small circle around zero and therefore the Taylor approximation requires a lower degree of interpolation. On the other hand we can see that for a small height-to-width ration (small Pe) the estimate based on ellipses, i.e. Alg.2, produces a significantly smaller m with the same actual degree m of interpolation and comparable error. This means that less scaling is required for larger t, cf. Example 4. When the rectangle R is closer to a square the algorithm still produces reliable results but is slightly less efficient than Alg.1. Remark 4.1 (θ m selection). As mentioned in Section 2.1 it is possible to select the θ m values differently depending on c. By selecting ˆθ m = max c θ m,c the computation corresponding to Pe = 0 gives the following results: m = 51, m = 44 and c = This indicates a slower convergence with a similar error, as the eigenvalues of the shifted matrix are in [ 8.82, 8.82] but we interpolate in [ 4.31, 4.31]. Example 3 (hump and scaling steps). In order to illustrate the potential gain of testing for a hump in our algorithm we use the matrix A as -gallery( triw,20,4) and v i = cos i which corresponds to [Al-Mohy and Higham 2011, Experiment 6]. We, however, perform in this example only a single time step of size t = 0.5. The norm of the (shifted) matrix is A 1 = 76. In the following discussion we call the shifted matrix again A. 18

19 10 8 expmv Alg.1* Alg.1 rel. error in iterations Figure 7: Iteration number versus error for exp( 1 2A)v illustrating the behavior of the hump reduction procedure. Here Alg.1 indicates that no hump reduction was performed. The error is measured relative with respect to each of the scaling steps. No early termination criterion is used in the computation. In Figure 7 we illustrate the behavior of the hump reduction procedure described in Section 3. A first observation is that for this matrix no scaling is required to obtain a solution with acceptable tolerance for our choice of parameters. Nevertheless, our method selects s = 2. In fact, ρ(a) = 0 and the method expmv is performing a single step, whereas our methods introduce a scaling step. For this nonnormal matrix the rectangle R is a square and therefore we only show the results for Alg.1, as we can expect a better performance for this algorithm (cf. Example 2). We can see quite clearly that a hump of about 8 digits is formed when we use the initial guess of c = θ 92 = As expected we only get an error of about 10 8 in each step and consequently a final error of the same size. On the other hand, if we reduce the interval length, the hump is reduced as well. For this (shifted) matrix the values d p are (76, 52.3, 39.6, 31.6, 26.0) (see Section 3.3). These values suggest that the interpolation interval should be reduced. In fact, by reducing the interval to c = 0 we would recover the truncated Taylor series method. As we restrict p to p 5, we use the interpolation interval c = θ 45 = 6.67 in the reduced case. For this example it is clear that a reduction of the interpolation interval to a single point is beneficial, as all eigenvalues of the matrix are located at this point. Nevertheless, we can see that the algorithm reducing the hump is working. The hump is significantly reduced and the error is close to the error of the truncated Taylor series method. Due to the structure of the matrix we can see that Alg.1 requires, in total, more iterations than the truncated Taylor series method. More precisely, with the help of the early termination criterion we can see that about 60 iterations are necessary whereas expmv only requires about 20, cf. Example 4. If we only require 8 digits of accuracy for the above example we do not need to reduce the interpolation interval to achieve that accuracy. For this example it would still be 19

20 beneficial to reduce the interpolation interval nevertheless as the necessary degree of interpolation is reduced by this as well. This is related to the observations of Remark 4.1. Here the norm of the matrix is a high overestimate of the spectral radius leading to slow convergence. # A n t ν α β η κ orani e e e e e+00 2 bcspwr e e e e e+01 3 triw e e e e e+03 4 triu e e e e e+03 5 AD Pe= /4 8.0e e e e e+04 6 AD Pe= e e e e e+04 7 onesided e e e e e+01 8 S3D /2 0.0e e e e e+03 9 Trans1D e e e e e+03 (a) Summary of the spectral properties for the matrices. 1-norm Alg.1 Alg.2 expmv # t s mv rel.err s mv rel.err s mv rel.err e e e e e e e e e e e e / e e e e e e e e e / e e e e e e-08 (b) Result for each matrix and the used algorithms, respectively. Table 4: Results for Example 4. For a tolerance of tol = 2 24 we compute exp(ta)v in a single call of the respective algorithm. The value s indicates the number of scaling steps and mv denotes the matrix-vector products without preprocessing. The values α, ν, η, ν correspond to (25). Example 4 (behavior for multiple scaling steps). In this experiment we investigate the behavior of the methods for multiple scaling steps. We use the two matrices of Examples 2 and 3 from above and in addition the matrices orani676 and bcspwr10 which are obtained from the University of Florida sparse matrix collection [Davis and Hu 2011] as well as several other matrices. The matrix triu is an upper triangular matrix with entries uniformly distributed on [ 0.5, 0.5]. For the matrix onesided we have a upper triangular matrix with one eigenvalue at 10 and 40 eigenvalues uniformly distributed on [ 9.9, 10.1] with standard deviation of 0.1. The values in the strict upper triangular part are uniformly distributed on [ 0.5, 0.5]. Furthermore, we use S3D from [Caliari et al. 2014, Example 3], a finite difference discretisation of the three dimensional Schrödinger equation with harmonic potential in [0, 1] 3. The matrix 20

21 Alg.1 2-norm Alg.1 -norm # t s mv rel.err s mv rel.err e e e e e e e e e e-09 Table 5: For a tolerance of tol = 2 24 we compute exp(ta)v in a single call of the algorithm Alg.1 with the 2- and maximum norm, respectively. The value s indicates the number of scaling steps and mv denotes the matrix-vector products without preprocessing. The numbers # correspond to Table 4a. Trans1D is a periodic, symmetric finite difference discretisation of the transport equation in [0, 1]. For the matrices orani676, S3D and Trans1d complex conjugate Leja points are used during the computation. As initial vector v we use [1,..., 1] T for orani676, [1, 0,..., 0, 1] T for bcspwr10, v as specified in Example 2 for AD, the discretisation of 4096x 2 (1 x) 2 y 2 (1 y) 2 z 2 (1 z) 2 is used for S3D, the discretisation of exp( 100(x 0.5) 2 ) for Trans1D, and v i = cos i for all other examples. This corresponds to [Al-Mohy and Higham 2011, Exp. 7]. We summarize the properties of all the matrices used in this example in Table 4. The tolerance is chosen as 2 24 and the relative error is computed with respect to expmv running with highest accuracy. Furthermore we use κ 1 = AA A A 1 A 2 1 as an indicator for the nonnormality of the matrices. From now on we refer to the matrices by their number given in the first column of Table 4a. We can see that for the nonnormal matrices {1, 3, 4} the algorithm expmv is superior in terms of matrix-vector products, in comparison to both variants of our algorithm. This is largely due to the fact that for these matrices the strategy based on the values d p (see Remark 2.2) is very beneficial for the computation. However, this highly depends on the used norm, as can be seen in Tables 4a and 5, respectively. The 2- and the maximum norm are beneficial in these cases. Nevertheless, highly compatible results for our methods can only be achieved by computing more accurate bounds for the field of values or eigenvalues than those given by Gerschgorin s disk theorem. For the matrices {5, 6, 7, 8, 9} the results show a different picture. Here, the Leja methods are clearly beneficial in terms of matrix-vector products. Furthermore, we can see that we also produce a smaller error in comparison to the truncated Taylor series approach. This is due to the fact that we divide the tolerance by s in the early termination criterion cf. (23). In the case of the complex conjugate Leja points this leads to a higher accuracy than required. On the other hand for the AD2D problem we can see that if we increase t by a factor of 4 the errors of the Leja methods increase by a factor of 2 whereas for expmv it increases by 4. 21

22 For matrices {1, 8, 9} conjugate complex Leja points are used, see Section 2.3. For the two normal matrices {8, 9} the Leja method can save a lot of matrix-vector products in comparison to expmv. As here the rectangle R is a line, Alg.2 is again superior to Alg.1 as it leads to fewer scaling steps and less matrix-vector products. Matrix 2 is normal. However, only for the 2-norm we have that A = ρ(a). This is also the only case where the Leja method saves matrix-vector products in comparison to expmv for this matrix. The final error of the methods is always comparable. The more precautious approach we propose leads sometimes to an increase in accuracy. Nevertheless, in the cases where the Leja method is beneficial it still uses significantly less matrix-vector products than expmv. A comparison of Alg.1 and Alg.2 shows that none of the two approaches can be considered superior or the better overall choice. Due to the construction Alg.2 provides a scaling factor and degree of interpolation that are independent of the norm, even though the Gerschgorin discs are closely related to the 1- and maximum norm. Nevertheless, the reduction of the interpolation interval is connected to a norm. In fact, this is also the case where the two methods do provide similar estimates for s. Depending on the used norm Alg.1 can have some significant gain or loss in performance. In total this means that, depending on the norm, the computational advantage shifts from one method to the other. If we always select the method with the least (predicted) computational cost we always use the more efficient methods as we save matrix-vector products. This indicates that a combination of the two algorithms, where we always select the one with the least expected cost is beneficial. 5. Discussion The backward error analysis presented in this work provides a solid basis for the selection of the scaling parameter s as well as the degree of interpolation m for the Leja method. With this information at hand the algorithm becomes in a sense direct, as the maximal number of matrix-vector products is known after the initial preprocessing. The cost of the algorithm is determined by the rectangle R and the norm of the matrix, respectively. The convergence is monitored by the early termination criterion. The practical use of this approach is confirmed by the numerical experiments of Section 4. The algorithm can be adapted in a similar way as the expmv method to support dense output and provides essentially the same properties as [Al-Mohy and Higham 2011, Algorithm 5.2]. In particular this means that the new algorithm also has some benefits in contrast to Krylov subspace based methods. Note that, in many applications, one has to compute e ta V for a scalar t and a n n 0 matrix V instead. This problem, however, is not more general since the product ta can always be considered as a new matrix and the performed analysis extends to a matrix V instead of a vector v. This is especially interesting in comparison to Krylov subspace methods as the available implementations would need to be called repeatedly for each column of V. 22