Computing Matrix Functions by Iteration: Convergence, Stability and the Role of Padé Approximants

Size: px

Start display at page:

Download "Computing Matrix Functions by Iteration: Convergence, Stability and the Role of Padé Approximants"

Evelyn Dawson
5 years ago
Views:

1 Computing Matrix Functions by Iteration: Convergence, Stability and the Role of Padé Approximants Nick Higham School of Mathematics The University of Manchester Approximation and Iterative Methods (AMI06) on the occasion of the retirement of Professor Claude Brezinski, Lille, June 22 23, 2006

2 What is Claude Photographing?

3 Overview Iterations X k+1 = g(x k ) for computing A, sign(a) ( p A, unitary polar factor,... ). Padé iterations for sign(a). Avoid Jordan form in convergence proofs. Reduce to: Does G k tend to zero? Can reduce stability to: Is H k bounded? Connections: matrix sign and matrix square root. Convergence of a linear iteration for A. MIMS Nick Higham Matrix functions 3 / 33

4 Overview Iterations X k+1 = g(x k ) for computing A, sign(a) ( p A, unitary polar factor,... ). Padé iterations for sign(a). Avoid Jordan form in convergence proofs. Reduce to: Does G k tend to zero? Can reduce stability to: Is H k bounded? Connections: matrix sign and matrix square root. Convergence of a linear iteration for A. Book in preparation: Functions of a Matrix: Theory and Computation MIMS Nick Higham Matrix functions 3 / 33

5 Matrix Sign Function Let A C n n have no pure imaginary eigenvalues and let A = ZJZ 1 be a Jordan canonical form with [ p q ] p J J = 1 0, Λ(J q 0 J 1 ) LHP, Λ(J 2 ) RHP. 2 [ Ip 0 sign(a) = Z 0 I q ] Z 1. MIMS Nick Higham Matrix functions 4 / 33

6 Matrix Sign Function Let A C n n have no pure imaginary eigenvalues and let A = ZJZ 1 be a Jordan canonical form with [ p q ] p J J = 1 0, Λ(J q 0 J 1 ) LHP, Λ(J 2 ) RHP. 2 [ Ip 0 sign(a) = Z 0 I q ] Z 1. sign(a) = A(A 2 ) 1/2. MIMS Nick Higham Matrix functions 4 / 33

7 Matrix Sign Function Let A C n n have no pure imaginary eigenvalues and let A = ZJZ 1 be a Jordan canonical form with [ p q ] p J J = 1 0, Λ(J q 0 J 1 ) LHP, Λ(J 2 ) RHP. 2 [ Ip 0 sign(a) = Z 0 I q ] Z 1. sign(a) = A(A 2 ) 1/2. sign(a) = 2 π A (t 2 I + A 2 ) 1 dt. 0 MIMS Nick Higham Matrix functions 4 / 33

8 Newton s Method for Sign Convergence X k+1 = 1 2 (X k + X 1 k ), X 0 = A. Let S := sign(a), G := (A S)(A + S) 1. Then X k = (I G 2k ) 1 (I + G 2k )S, Ei vals of G are (λ i sign(λ i ))/(λ i + sign(λ i )). Hence ρ(g) < 1 and G k 0. Easy to show X k+1 S 1 1 Xk X k S 2. 2 MIMS Nick Higham Matrix functions 5 / 33

9 Matrix Square Root X is a square root of A C n n X 2 = A. Number of square roots may be zero, finite or infinite. For A with no eigenvalues on R = {x R : x 0}, denote by A 1/2 the principal square root: unique square root with spectrum in open right half-plane. MIMS Nick Higham Matrix functions 6 / 33

10 Newton s Method for Square Root Newton s method: X 0 given, Solve X k E k + E k X k = A X 2 k X k+1 = X k + E k Assume AX 0 = X 0 A. Then can show } k = 0, 1, 2,... X k+1 = 1 2 (X k + X 1 k A). ( ) For nonsingular A, local quadratic cgce of full Newton to a primary square root. To which square root do the iterations converge? ( ) can converge when full Newton breaks down. Lack of symmetry in ( ). MIMS Nick Higham Matrix functions 7 / 33

11 Convergence (Jordan) Assume X 0 = p(a) for some poly p. Let Z 1 AZ = J be Jordan canonical form and set Z 1 X k Z = Y k. Then Y k+1 = 1 2 (Y k + Y 1 k J), Y 0 = J. Convergence of diagonal of Y k reduces to scalar case: Heron: y k+1 = 1 2 (y k + λyk ), y 0 = λ. Can show that off-diagonal converges. Problem: analysis does not generalize to X 0 A = AX 0! X 0 not necessarily a polynomial in A. MIMS Nick Higham Matrix functions 8 / 33

12 Convergence (via Sign) Theorem Let A C n n have no e vals on R. The Newton square root iterates X k with X 0 A = AX 0 are related to the Newton sign iterates S k+1 = 1 2 (S k + S 1 k ), S 0 = A 1/2 X 0 by X k A 1/2 S k. Hence, provided A 1/2 X 0 has no pure imag e vals, X k are defined and X k A 1/2 sign(s 0 ) quadratically. : X k A 1/2 if Λ(A 1/2 X 0 ) RHP, e.g., if X 0 = A. MIMS Nick Higham Matrix functions 9 / 33

13 More Sign Iterations Start with Newton: x k+1 = 1 2 (x k + x 1 k ). MIMS Nick Higham Matrix functions 10 / 33

14 More Sign Iterations Start with Newton: x k+1 = 1 2 (x k + x 1 k ). Invert: x k+1 = 2x k x 2 k + 1. MIMS Nick Higham Matrix functions 10 / 33

15 More Sign Iterations Start with Newton: x k+1 = 1 2 (x k + x 1 k ). Invert: Halley: x k+1 = 2x k x 2 k + 1. x k+1 = x k(3 + xk 2) xk 2 MIMS Nick Higham Matrix functions 10 / 33

16 More Sign Iterations Start with Newton: x k+1 = 1 2 (x k + x 1 k ). Invert: Halley: Newton Schulz: x k+1 = 2x k x 2 k + 1. x k+1 = x k(3 + xk 2) xk 2 x k+1 = x k 2 (3 x 2 k ). MIMS Nick Higham Matrix functions 10 / 33

17 The Padé Family: Derivation For non-pure imaginary z C, ξ := 1 z 2, sign(z) = z z 2 = z 1 (1 z2 ) = z 1 ξ, [l/m] Padé approximant r lm (ξ) = p lm (ξ)/q lm (ξ) to h(ξ) = (1 ξ) 1/2 known explicitly. Kenney & Laub (1991): set up iterations x k+1 = f lm (x k ) := x k p lm (1 x 2 k ) q lm (1 x 2 k ), x 0 = a. MIMS Nick Higham Matrix functions 11 / 33

18 The Padé Family: Examples l m = 0 m = 1 m = 2 0 x 1 2 x 2 (3 x2 ) x 8 (15 10x2 + 3x 4 ) 2x 1 + x 2 8x 3 + 6x 2 x 4 x(3 + x 2 ) 1 + 3x 2 4x(1 + x 2 ) 1 + 6x 2 + x 4 x ( x 2 x 4 ) x(5 + 10x 2 + x 4 ) x x 2 + 5x 4 MIMS Nick Higham Matrix functions 12 / 33

19 The Padé Family: Convergence Theorem (Kenney & Laub) Let A C n n have no pure imag e vals. Let l + m > 0. (a) For l m 1, if I A 2 < 1 then X k sign(a) as k and I X 2 k < I A2 (l+m+1)k. (b) For l = m 1 and l = m, (S X k )(S + X k ) 1 = [ (S A)(S + A) 1] (l+m+1) k and hence X k sign(a) as k. MIMS Nick Higham Matrix functions 13 / 33

20 The Padé Family: Examples l m = 0 m = 1 m = 2 0 x 2x 8x 1 + x x 2 x x 2 (3 x2 ) x 8 (15 10x2 + 3x 4 ) x(3 + x 2 ) 1 + 3x 2 4x(1 + x 2 ) 1 + 6x 2 + x 4 x ( x 2 x 4 ) x(5 + 10x 2 + x 4 ) x x 2 + 5x 4 MIMS Nick Higham Matrix functions 14 / 33

21 Principal Padé Iterations l m = 0 m = 1 m = 2 0 x 2x 1 + x 2 8x 3 + 6x 2 x 4 1 x 2 (3 x2 ) x(3 + x 2 ) 1 + 3x 2 4x(1 + x 2 ) 1 + 6x 2 + x 4 2 x 8 (15 10x2 + 3x 4 ) x ( x 2 x 4 ) x(5 + 10x 2 + x 4 ) x x 2 + 5x 4 MIMS Nick Higham Matrix functions 15 / 33

22 Principal Padé Iteration Properties Define g r (x) taking from the main diagonal and superdiagonal of Padé table in zig-zag fashion. Theorem (Kenney & Laub) (a) g r (x) = (1 + x)r (1 x) r (1 + x) r + (1 x) r. (b) g r (x) = tanh(r arctanh(x)). (c) g r (g s (x)) = g rs (x) (semigroup property). (d) g r (x) = 2 r r 2 2 i=0 sin 2( ) (2i+1)π 2r x + cos 2 ( (2i+1)π 2r ) x 2. MIMS Nick Higham Matrix functions 16 / 33

23 g r (x) = tanh(r arctanh(x)) r = 2 r = 4 r = 8 r =

24 Class of Square Root Iterations Theorem (H, Mackey, Mackey & Tisseur, 2005) Suppose the iteration X k+1 = X k h(x 2 k ), X 0 = A converges to sign(a) with order m. If Λ(A) R = and Y k+1 = Y k h(z k Y k ), Y 0 = A, Z k+1 = h(z k Y k )Z k, Z 0 = I, then Y k A 1/2 and Z k A 1/2 as k with order m. MIMS Nick Higham Matrix functions 18 / 33

25 Class of Square Root Iterations Theorem (H, Mackey, Mackey & Tisseur, 2005) Suppose the iteration X k+1 = X k h(x 2 k ), X 0 = A converges to sign(a) with order m. If Λ(A) R = and Y k+1 = Y k h(z k Y k ), Y 0 = A, Z k+1 = h(z k Y k )Z k, Z 0 = I, then Y k A 1/2 and Z k A 1/2 as k with order m. Proof makes use of sign ([ ]) [ 0 A = I 0 0 A 1/2 A 1/2 0 ]. MIMS Nick Higham Matrix functions 18 / 33

26 Examples Newton Sign: X k+1 = X k 1(I + (X 2 2 k ) 1 ) X k h(xk 2), X 0 = A. Y k+1 = 1Y 2 k(i + (Z k Y k ) 1 ) = 1(Y 2 k + Z 1 k ), Y 0 = A. Denman Beavers sq root iteration: Y k+1 = 1 ( ) Yk + Z 1 k, 2 Y0 = A, Z k+1 = 1 ( ) Zk + Y 1 k, 2 Z0 = I. MIMS Nick Higham Matrix functions 19 / 33

27 Examples Newton Sign: X k+1 = X k 1(I + (X 2 2 k ) 1 ) X k h(xk 2), X 0 = A. Y k+1 = 1Y 2 k(i + (Z k Y k ) 1 ) = 1(Y 2 k + Z 1 k ), Y 0 = A. Denman Beavers sq root iteration: Y k+1 = 1 ( ) Yk + Z 1 k, 2 Y0 = A, Z k+1 = 1 ( ) Zk + Y 1 k, 2 Z0 = I. Newton Schulz sq root iteration: Y k+1 = 1 2 Y k (3I Z k Y k ), Y 0 = A, Z k+1 = 1 2 (3I Z ky k )Z k, Z 0 = I. MIMS Nick Higham Matrix functions 19 / 33

28 History of Newton Sqrt Instability X k+1 = 1(X 2 k + X 1 k A). Instability of Newton noted by Laasonen (1958): Newton s method if carried out indefinitely, is not stable whenever the ratio of the largest to the smallest eigenvalue of A exceeds the value 9. Described informally by Blackwell (1985) in Mathematical People: Profiles and Interviews. Analyzed by H (1986) for diagonalizable A by deriving error amplification factors. MIMS Nick Higham Matrix functions 21 / 33

29 Stability Definition The iteration X k+1 = g(x k ) is stable in a nbhd of a fixed point X if Fréchet derivative dg X has bounded powers. MIMS Nick Higham Matrix functions 22 / 33

30 Stability Definition The iteration X k+1 = g(x k ) is stable in a nbhd of a fixed point X if Fréchet derivative dg X has bounded powers. For scalars & superlinear g, dg X = g (x) = 0. MIMS Nick Higham Matrix functions 22 / 33

31 Stability Definition The iteration X k+1 = g(x k ) is stable in a nbhd of a fixed point X if Fréchet derivative dg X has bounded powers. For scalars & superlinear g, dg X = g (x) = 0. Let X 0 = X + E 0, E k := X k X. Then X k+1 = g(x k ) = g(x + E k ) = g(x) + dg X (E k ) + o( E k ). So, since g(x) = X, E k+1 = dg X (E k ) + o( E k ). If dgx i (E) c, then recurring leads to E k c E 0 + kc o( E 0 ). MIMS Nick Higham Matrix functions 22 / 33

32 Stability of Newton Square Root g(x) = 1 2 (X + X 1 A). dg X (E) = 1 2 (E X 1 EX 1 A). Relevant fixed point: X = A 1/2. dg A 1/2(E) = 1 2 (E A 1/2 EA 1/2 ). Ei vals of dg A 1/2 are For stability we need 1 (1 λ 1/2 i λ 1/2 j ), i, j = 1: n. 2 1 max i,j 2 1 λ 1/2 For hpd A, need κ 2 (A) < 9. i λ 1/2 j < 1. MIMS Nick Higham Matrix functions 23 / 33

33 Advantages Uses only Fréchet derivative of g. No additional assumptions on A. Perturbation analysis is all in the definition. General, unifying approach. Facilitates analysis of families of iterations. MIMS Nick Higham Matrix functions 24 / 33

34 Stability of Sign Iterations (H) Theorem Let X k+1 = g(x k ) be any superlinearly convergent iteration for S = sign(x 0 ). Then dg S (E) = L S (E) = 1 2 (E SES), where L S is the Fréchet derivative of the matrix sign function at S. Hence dg S is idempotent (dg S dg S = dg S ) and the iteration is stable. All sign iterations are automatically stable. MIMS Nick Higham Matrix functions 25 / 33

35 Theorem (H, Mackey, Mackey & Tisseur, 2005) Consider the iteration function G(Y, Z) = [ Yh(ZY) h(zy)z where X k+1 = X k h(xk 2 ) is any superlinearly cgt iteration for sign(x 0 ). Any pair P = (B, B 1 ) is a fixed point for G, and the Fréchet derivative of G at P is dg P (E, F) = 1 2 [ ], E BFB F B 1 EB 1 dg P is idempotent and hence the iteration is stable. ]. In particular: Denman Beavers iteration is stable. MIMS Nick Higham Matrix functions 26 / 33

36 Binomial Iteration for (I C) 1/2 Suppose ρ(c) < 1. Define (I C) 1/2 =: I P and square to obtain P 2 2P = C. Suggests: I P k reproduces (I C) 1/2 = P k+1 = 1 2 (C + P2 k ), P 0 = 0. j=0 up to terms of order C k. ( 1 ) 2 ( C) j I j α j C j, (α j > 0) j=1 MIMS Nick Higham Matrix functions 27 / 33

37 Binomial Iteration Convergence (1) Theorem P k+1 = 1 2 (C + P2 k), P 0 = 0. Let C R n n satisfy C 0 and ρ(c) < 1 and write (I C) 1/2 = I P. Then P k P with 0 P k P k+1 P, k 0. MIMS Nick Higham Matrix functions 28 / 33

38 Binomial Iteration Convergence (1) Theorem P k+1 = 1 2 (C + P2 k), P 0 = 0. Let C R n n satisfy C 0 and ρ(c) < 1 and write (I C) 1/2 = I P. Then P k P with 0 P k P k+1 P, k 0. Let X k = P k /2. Then X k+1 = X 2 k + C/4. MIMS Nick Higham Matrix functions 28 / 33

39 Binomial Iteration Convergence (2) Theorem (H) Let the e vals λ of C C n n lie in the cardioid { 2z z 2 : z C, z < 1 }. Then (I C) 1/2 =: I P exists and P k P linearly MIMS Nick Higham Matrix functions 29 / 33

40 Paradox (I C) 1/2 = ( 1) 2 j=0 j ( C) j has radius of convergence 1. P k+1 = 1 2 (C + P2 k ), P 0 = 0: P k reproduces first k + 1 terms of the power series. P k converges in the cardioid unit circle. MIMS Nick Higham Matrix functions 30 / 33

41 Conclusions Stability equivalent to power boundedness of Fréchet derivative. Better understanding of convergence analysis (prefer matrix powers to Jordan form.) Matrix sign function is fundamental and connections with sqrt can be exploited. Padé iterations for sign have many interesting properties. Analysis of linearly convergent iterations can be tricky. MIMS Nick Higham Matrix functions 31 / 33

42 What is Claude Photographing?

The matrix sign function

The matrix sign function 1 Re x > 0, sign(x) = 1 Re x < 0, undefined Re x = 0. Suppose the Jordan form of A is reblocked as A = [ V 1 V 2 ] [ J 1 J 2 ] [ V 1 V 2 ] 1, where J 1 contains all eigenvalues