Functions of a Matrix: Theory and Computation

Transcription

1 Functions of a Matrix: Theory and Computation Nick Higham School of Mathematics The University of Manchester higham@ma.man.ac.uk Landscapes in Mathematical Science, University of Bath, November

2 Outline 1 Definition of f(a) 2 Motivation and MATLAB 3 e A and its Frechét derivative 4 A 1/2 : Modified Newton Methods MIMS Nick Higham Functions of a Matrix 2 / 45

3 Function of a Matrix f : C n n C n n for an underlying scalar function f. These are not matrix functions: trace(a), det(a). The adjugate (or adjoint) matrix. Transfer function f(t) = B(tI A) 1 C. sin A = (sin a ij ). These are matrix functions: e A = I + A + A2 2! +. log(i + A) = A A2 2 + A3 3 A 1, A 1/2. +, ρ(a) < 1. MIMS Nick Higham Functions of a Matrix 3 / 45

4 Multiplicity of Definitions There have been proposed in the literature since 1880 eight distinct definitions of a matric function, by Weyr, Sylvester and Buchheim, Giorgi, Cartan, Fantappiè, Cipolla, Schwerdtfeger and Richter. R. F. Rinehart, The Equivalence of Definitions of a Matric Function, Amer. Math. Monthly (1955) MIMS Nick Higham Functions of a Matrix 4 / 45

5 Jordan Canonical Form Z 1 AZ = J = diag(j 1,...,J p ), J }{{} k = m k m k λ k 1 λ k λ k Definition f(a) = Zf(J)Z 1 = Zdiag(f(J k ))Z 1, f(λ k ) f f (m k 1) )(λ k ) (λ k )... (m k 1)! f(j k ) = f(λ k ) f (λ k ) f(λ k ) MIMS Nick Higham Functions of a Matrix 5 / 45

6 The Formula for f(j k ) Write J k = λ k I + E k C m k m k. For mk = 3 we have E k = , Ek 2 = , Ek 3 = Assume f has Taylor expansion f(t) = f(λ k ) + f (λ k )(t λ k ) + + f (j) (λ k )(t λ k ) j j! +. Then f(j k ) = f(λ k )I + f (λ k )E k + + f (m k 1) (λ k )E m k 1 k. (m k 1)! MIMS Nick Higham Functions of a Matrix 6 / 45

7 Interpolation Definition (Sylvester, 1883; Buchheim, 1886) Distinct e vals λ 1,...,λ s, n i = max size of Jordan blocks for λ i. Then f(a) = r(a), where r is unique Hermite interpolating poly of degree < s i=1 n i satisfying r (j) (λ i ) = f (j) (λ i ), j = 0: n i 1, i = 1: s. MIMS Nick Higham Functions of a Matrix 7 / 45

8 Interpolation Definition (Sylvester, 1883; Buchheim, 1886) Distinct e vals λ 1,...,λ s, n i = max size of Jordan blocks for λ i. Then f(a) = r(a), where r is unique Hermite interpolating poly of degree < s i=1 n i satisfying r (j) (λ i ) = f (j) (λ i ), j = 0: n i 1, i = 1: s. [ ] 2 2 Example. Let f(t) = t 1/2, A =, λ(a) = {1, 4}. 1 3 Taking +ve square roots, r(t) = f(1) t f(4) t = 1 (t + 2). 3 [ A 1/2 = r(a) = 1 3 (A + 2I) = 1 3 ]. MIMS Nick Higham Functions of a Matrix 7 / 45

9 Cauchy Integral Theorem Definition f(a) = 1 f(z)(zi A) 1 dz, 2πi Γ where f is analytic on and inside a closed contour Γ that encloses λ(a). MIMS Nick Higham Functions of a Matrix 8 / 45

10 Equivalence of Definitions Theorem The three definitions are equivalent, modulo analyticity assumption for Cauchy. Interpolation: for basic properties. JCF: for solving matrix equations (e.g., X 2 = A, e X = A). For evaluation (normal A). Cauchy: various uses. MIMS Nick Higham Functions of a Matrix 9 / 45

11 Equivalence of Definitions Theorem The three definitions are equivalent, modulo analyticity assumption for Cauchy. Interpolation: for basic properties. JCF: for solving matrix equations (e.g., X 2 = A, e X = A). For evaluation (normal A). Cauchy: various uses. For computation: Use the definitions (with care). Schur decomposition for general f. Methods specific to particular f. MIMS Nick Higham Functions of a Matrix 9 / 45

12 Outline 1 Definition of f(a) 2 Motivation and MATLAB 3 e A and its Frechét derivative 4 A 1/2 : Modified Newton Methods MIMS Nick Higham Functions of a Matrix 10 / 45

13 Toolbox of Matrix Functions Want to have techniques for evaluating interesting f at matrix args as well as scalar args. Example: has solution d 2 y dt 2 + Ay = 0, y(0) = y 0, y (0) = y 0 y(t) = cos( At)y 0 + ( A ) 1 sin( At)y 0, where A is any square root of A. MATLAB hasexpm,logm,sqrtm,funm. MIMS Nick Higham Functions of a Matrix 11 / 45

14 Classic MATLAB < M A T L A B > Version of 01/10/84 HELP is available <> help Type HELP followed by INTRO (To get started) NEWS (recent revisions) ABS ANS ATAN BASE CHAR CHOL CHOP CLEA COND CONJ COS DET DIAG DIAR DISP EDIT EIG ELSE END EPS EXEC EXIT EXP EYE FILE FLOP FLPS FOR FUN HESS HILB IF IMAG INV KRON LINE LOAD LOG LONG LU MACR MAGI NORM ONES ORTH PINV PLOT POLY PRIN PROD QR RAND RANK RCON RAT REAL RETU RREF ROOT ROUN SAVE SCHU SHOR SEMI SIN SIZE SQRT STOP SUM SVD TRIL TRIU USER WHAT WHIL WHO WHY < > ( ) =., ; \ / + - * : MIMS Nick Higham Functions of a Matrix 12 / 45

15 Classic MATLAB <> help fun FUN For matrix arguments X, the functions SIN, COS, ATAN, SQRT, LOG, EXP and X**p are computed using eigenvalues D and eigenvectors V. If <V,D> = EIG(X) then f(x) = V*f(D)/V. This method may give inaccurate results if V is badly conditioned. Some idea of the accuracy can be obtained by comparing X**1 with X. For vector arguments, the function is applied to each component. The availability of [FUN] in early versions of MATLAB quite possibly contributed to the system s technical and commercial success. Cleve Moler (2003) MIMS Nick Higham Functions of a Matrix 13 / 45

16 Outline 1 Definition of f(a) 2 Motivation and MATLAB 3 e A and its Frechét derivative 4 A 1/2 : Modified Newton Methods MIMS Nick Higham Functions of a Matrix 14 / 45

17 Matrix Exponential Large literature. Early exposition in Frazer, Duncan & Collar. Elementary Matrices and Some Applications to Dynamics and Differential Equations. CUP, Moler & Van Loan. Nineteen dubious ways to compute the exponential of a matrix, twenty-five years later, SIAM Rev., 45 (2003). Over 500 citations on ISI Citation Index. MIMS Nick Higham Functions of a Matrix 15 / 45

18 Scaling and Squaring Method B A/2 s so B 1 r m (B) = [m/m] Padé approximant to e B X = r m (B) 2s e A Used byexpm in MATLAB. Originates with Lawson (1967). Ward (1977): algorithm, with rounding error analysis and a posteriori error bound. Moler & Van Loan (1978): give backward error analysis covering truncation error in Padé approximants, allowing choice of s and m. MIMS Nick Higham Functions of a Matrix 16 / 45

19 Padé Approximations r m to e x r m (x) = p m (x)/q m (x) known explicitly: p m (x) = m j=0 (2m j)!m! x j (2m)!(m j)! j! and q m (x) = p m ( x). Error satisfies e x r m (x) = ( 1) m (m!) 2 (2m)!(2m + 1)! x2m+1 +O(x 2m+2 ). MIMS Nick Higham Functions of a Matrix 17 / 45

20 Analysis Let e A r m (A) = I + G = e H and assume G < 1. Then H = log(i + G) G j /j = log(1 G ). j=1 Hence r m (A) = e A e H = e A+H. Rewrite as r m (A/2 s ) 2s = e A+E, where E = 2 s H satisfies E 2 s log(1 G ). MIMS Nick Higham Functions of a Matrix 18 / 45

21 Result Theorem Let e 2 sa r m (2 s A) = I + G, where G < 1. Then the Padé approximant r m satisfies r m (2 s A) 2s = e A+E, where E A log(1 G ). 2 s A Need to bound G given m and 2 s A. Then E / A bounded in terms of m, s and A. Now select best (s, m) pair for given A. MIMS Nick Higham Functions of a Matrix 19 / 45

22 Bounding G ρ(x) := e x r m (x) 1 = i=2m+1 c i x i converges absolutely for x < min{ t : q m (t) = 0 } =: ν m. Hence, with θ := 2 s A < ν m, G = ρ(2 s A) i=2m+1 Thus E / A log(1 f(θ))/θ). c i θ i =: f(θ). ( ) If only A known, ( ) is optimal bound on G. Moler & Van Loan (1978) bound less sharp; Dieci & Papini (2000) bound a different error. MIMS Nick Higham Functions of a Matrix 20 / 45

23 Summary of Computations Solve bound = unit roundoff by summing 150 terms of series in 250 digit arithmetic. Work out cost of evaluating r m (B) and of the squaring phase. Minimize cost subject to retaining numerical stability in evaluation of r m (B). Precision m θ m IEEE single IEEE double IEEE quad MIMS Nick Higham Functions of a Matrix 21 / 45

24 Scaling and Squaring Algorithm for e A Algorithm 1 (H, 2005; MATLAB 7.2, Mathematica 5.1) 1 for m = [ ] 2 if A 1 θ m 3 X = r m (A). 4 quit 5 end 6 end 7 A A/2 s with s 0 minimal s.t. A/2 s 1 θ 13 = A 2 = A 2, A 4 = A 2 2, A 6 = A 2 A 4 U = A [ A 6 (b 13 A 6 + b 11 A 4 + b 9 A 2 ) + b 7 A 6 + b 5 A 4 + b 3 A 2 + b 1 I ] 10 V = A 6 (b 12 A 6 + b 10 A 4 + b 8 A 2 ) + b 6 A 6 + b 4 A 4 + b 2 A 2 + b 0 I 11 Solve ( U + V)r 13 = U + V for r X = r s 13 by repeated squaring. MIMS Nick Higham Functions of a Matrix 22 / 45

25 Comparison with Existing Algorithms Method m max 2 s A Alg Ward (1977) [θ 8 = 1.5] Old MATLABexpm [θ 6 = 0.54] Sidje (1998) A 1 > 1: Alg 1 requires 1 2 fewer mat mults than Ward, 2 3 fewer thanexpm. A 1 (2, 2.1): Alg 1 Ward expm Sidje mults MIMS Nick Higham Functions of a Matrix 23 / 45

26 Squaring Phase The bound A 2 fl(a 2 ) γ n A 2, γ n = nu 1 nu. shows the dangers in matrix squaring. Open question: are errors in squaring phase consistent with conditioning of the problem? Our choice of parameters uses 1 5 fewer matrix squarings than previous algorithms. Hence has potential accuracy advantages. MIMS Nick Higham Functions of a Matrix 24 / 45

27 Numerical Experiment 70 test matrices, dimension Evaluated 1-norm relative error X X 1 / X 1. Notation: expm: Alg 1 (MATLAB 7.2). old_expm: MATLAB 7.1. funm: MATLAB 7. padm: Sidje. e A+E e A 2 cond(a) = lim max. ǫ 0 E 2 ǫ A 2 ǫ e A 2 MIMS Nick Higham Functions of a Matrix 25 / 45

28 Different S&S Codes andfunm old_expm 10 6 padm funm 10 8 expm (Alg 1) cond*u MIMS Nick Higham Functions of a Matrix 26 / 45

29 Performance Profiles Dolan & Moré (2002) propose a new type of performance profile. For the given set of solvers and test problems, plot x-axis: α y-axis: probability that solver has error within factor α of smallest error over all solvers on the test set. MIMS Nick Higham Functions of a Matrix 27 / 45

30 Performance Profile expm (Alg 1) padm 0.5 p 0.4 funm old_expm α MIMS Nick Higham Functions of a Matrix 28 / 45

31 Frechét Derivative Fréchet derivative of f : C n n C n n at X C n n A linear mapping L : C n n C n n s.t. for all E C n n f(x + E) f(x) L(X, E) = o( E ). Example For f(x) = X 2 we have so L(X, E) = XE + EX. f(x + E) f(x) = XE + EX + E 2, MIMS Nick Higham Functions of a Matrix 29 / 45

32 Frechét Derivative of e A L(A, E) = 1 L(A) := max L(A, E) / E. 0 e A(1 s) Ee As ds. Condition no. of the exponential: κ exp (A) = L(A) A. e A L(A) L(A, I) = e A κ exp (A) A. Theorem If A C n n is normal then in the 2-norm, κ exp (A) = A 2. If A R n n is a nonnegative scalar multiple of a stochastic matrix then in the -norm, κ exp (A) = A. MIMS Nick Higham Functions of a Matrix 30 / 45

33 Quadrature Repeated trap rule: 1 f(t) dt 1 ( 1f 0 m ( f 1 + f f m 1 + 1f 2 m), f i := f(i/m) gives L(A, E) 1 1 m 2 ea E + m 1 i=1 ea(1 i/m) Ee Ai/m + ). 1 2 EeA Requires e A/m, e 2A/m,..., e (m 1)A/m, e A. Lemma Consider R 1 (A, E) = p i=1 w i e A(1 t i) Ee At i and denote its m-times repeated form by R m (A, E). If Q s = R 1 (2 s A, 2 s E), Q i 1 = e 2 ia Q i + Q i e 2 ia, i = s : 1 : 1, then Q 0 = R 2 s(a, E). MIMS Nick Higham Functions of a Matrix 31 / 45

34 Scaling and Squaring Recall L x 2(A, E) = AE + EA. Applying chain rule to e A = (e A/2 ) 2 gives ( L exp (A, E) = L x 2 e A/2, L exp (A/2, E/2) ) = e A/2 L exp (A/2, E/2) + L exp (A/2, E/2)e A/2. Recurrence for L 0 = L exp (A, E): L s = L exp (2 s A, 2 s E), L i 1 = e 2 ia L i + L i e 2 ia, i = s : 1 : 1. MIMS Nick Higham Functions of a Matrix 32 / 45

35 Quadrature Algorithm Algorithm (Kenney & Laub, 1998; Mathias, 1993) Approximates L = L exp (A, E) via repeated Simpson rule; order of magnitude estimate. 1 B = A/2 s with s 0 minimal s.t. A/2 s 1 1/2 2 X = e B 3 X = e B/2 4 Q s = 2 s (XE + 4 XE X + EX)/6 5 for i = s: 1: 1 6 if i < s, X = e 2 ia, end 7 Q i 1 = XQ i + Q i X 8 end 9 L = Q 0 MIMS Nick Higham Functions of a Matrix 33 / 45

36 Kronecker Formula L(A, E) = 1 0 e A(1 s) Ee As ds. MIMS Nick Higham Functions of a Matrix 34 / 45

37 Kronecker Formula L(A, E) = 1 A B = (a ij B) C n2 n 2. A B = A I n + I m B. e A B = e A e B. vec(axb) = (B T A)vec(X). 0 e A(1 s) Ee As ds. MIMS Nick Higham Functions of a Matrix 34 / 45

38 Kronecker Formula L(A, E) = 1 A B = (a ij B) C n2 n 2. A B = A I n + I m B. e A B = e A e B. vec(axb) = (B T A)vec(X). Theorem vec(l(a, E)) = K(A)vec(E), where 0 e A(1 s) Ee As ds. K(A) = 1 2 (eat e A )τ ( 1 2 [AT ( A)] ) C n2 n 2, τ(x) = tanh(x)/x and 1 2 AT ( A) < π/2 assumed. MIMS Nick Higham Functions of a Matrix 34 / 45

39 Approximating the Frechét Derivative [ A T ( A) θi ] vec(e) = [ A T I I A θi ] vec(e) = vec(ea AE θi) = vec(e(a θ/2 I) (A + θ/2 I)E). Obtain Padé approximant τ(x) = tanh(x)/x r m (x) = m (x/β i 1) 1 (x/α i 1) by truncating τ(x) = i=1 1 x 2 /(1 3) x 2 /(3 5) 1 + x 2 /((2k 1) (2k + 1)) 1 +. MIMS Nick Higham Functions of a Matrix 35 / 45

40 Algorithm (Fréchet derivative; Kenney & Laub, 1998) Evaluates L = L exp (A, E) using scaling and squaring and [8/8] Padé approximant to τ. 1 B = A/2 s with s 0 minimal s.t. A/2 s G 0 = 2 s E 3 for i = 1: 8 4 Solve (I + B/β i ) G i + G i (I B/β i ) = (I + B/α i )G i 1 + G i 1 (I B/α i ). 5 end 6 X = e B 7 L s = (G m X + XG m )/2 8 for i = s: 1: 1 9 if i < s, X = e 2 ia, end 10 L i 1 = XL i + L i X 11 end 12 L = L 0

41 Outline 1 Definition of f(a) 2 Motivation and MATLAB 3 e A and its Frechét derivative 4 A 1/2 : Modified Newton Methods MIMS Nick Higham Functions of a Matrix 37 / 45

42 Matrix Square Root X is a square root of A C n n X 2 = A. Number of square roots may be zero, finite or infinite. Definition For A with no eigenvalues on R = {x R : x 0} the principal square root A 1/2 is unique square root X with spectrum in open right half-plane. MIMS Nick Higham Functions of a Matrix 38 / 45

43 Newton s Method for Square Root Apply Newton to F(X) = X 2 A = 0: X 0 given, } Solve X k E k + E k X k = A Xk 2 k = 0, 1, 2,... X k+1 = X k + E k Modified Newton iteration: freeze Fréchet derivative at X 0 : } Solve X 0 E k + E k X 0 = A Xk 2 k = 0, 1, 2,..., X k+1 = X k + E k X 0 diagonal cheap to solve for E k. MIMS Nick Higham Functions of a Matrix 39 / 45

44 Visser Iteration Set X 0 = (2α) 1 I in modified Newton: Visser iteration (1937) X k+1 = X k + α(a X 2 k ), X 0 = (2α) 1 I. Stationary iteration. Richardson iteration. Linear convergence. Choice of α? MIMS Nick Higham Functions of a Matrix 40 / 45

45 Visser History X k+1 = X k + α(a X 2 k ), X 0 = (2α) 1 I. Used with α = 1/2 by Visser (1937) to show positive operator on Hilbert space has a positive square root. Likewise in functional analysis texts, e.g. Riesz & Sz.-Nagy (1956). Enables proof of existence of A 1/2 without using spectral theorem. Used computationally by Liebl (1965), Babuška, Práger & Vitásek (1966), Späth (1966), Duke (1969), Elsner (1970). Elsner proves cgce for A C n n with real, positive ei vals if 0 < α ρ(a) 1/2. MIMS Nick Higham Functions of a Matrix 41 / 45

46 Visser Transformations Let X k = θy k, β = θα and à = θ 2 A. Set β = 1/2: Y k+1 = Y k + β(ã Y 2 k ), Y 0 = 1 2β I. Y k+1 = Y k (à Y 2 k ), Y 0 = I. With à I C and Y k = I P k : Q k = P k /2: P k+1 = 1 2 (C + P2 k), P 0 = 0. Q k+1 = Q 2 k + C 4, Q 0 = 0. MIMS Nick Higham Functions of a Matrix 42 / 45

47 Visser Convergence X k+1 = X k + α(a Xk 2 ), X 0 = (2α) 1 I. Theorem (H, 2006) Let A C n n and α > 0. If Λ(I 4α 2 A) lies in the cardioid D = { 2z z 2 : z C, z < 1 } then A 1/2 exists and X k A 1/2 linearly MIMS Nick Higham Functions of a Matrix 43 / 45

48 Example A R spd with a ii = i 2, a ij = 0.1, i j. Aim for rel residual < nu in IEEE DP arithmetic. Pulay iteration D = diag(a): θ = 0.191, 9 iters. Visser iteration α = (hand optimized), 245 iters MIMS Nick Higham Functions of a Matrix 44 / 45

49 In Conclusion Many applications of f(a), e.g. control theory, computer graphics, theoretical physics. Need better understanding of conditioning of f(a). Can we exploit structure, e.g. A matrix automorphism group or Jordan or Lie algebra? Krylov methods needed for large, sparse A. How to use Cauchy integral computationally (H & Trefethen)? MIMS Nick Higham Functions of a Matrix 45 / 45

50 References I D. A. Bini, N. J. Higham, and B. Meini. Algorithms for the matrix pth root. Numer. Algorithms, 39(4): , C.-H. Guo and N. J. Higham. A Schur Newton method for the matrix pth root and its inverse. SIAM J. Matrix Anal. Appl., 28(3): , N. J. Higham. Functions of a Matrix: Theory and Computation. Book in preparation. MIMS Nick Higham Functions of a Matrix 44 / 45

51 References II N. J. Higham. The scaling and squaring method for the matrix exponential revisited. SIAM J. Matrix Anal. Appl., 26(4): , C. S. Kenney and A. J. Laub. A Schur Fréchet algorithm for computing the logarithm and exponential of a matrix. SIAM J. Matrix Anal. Appl., 19(3): , R. Mathias. Evaluating the Frechet derivative of the matrix exponential. Numer. Math., 63(2): , MIMS Nick Higham Functions of a Matrix 45 / 45