max λ Λ(A) p(λ) b Ax 0 2

Transcription

1 ON THE CONVERGENCE OF THE ARNOLDI PROCESS FOR EIGENVALUE PROBLEMS M BELLALIJ, Y SAAD, AND H SADOK Abstract This paper takes another look at the convergence of the Arnoldi procedure for solving nonsymmetric eigenvalue problems Three different viewpoints are considered The first uses a bound on the distance from the eigenvector to the Krylov subspace from the smallest singular value of matrix consisting of the Krylov basis A second approach relies on the Schur factorization Finally, a third viewpoint, uses expansions in the eigenvector basis for the diagonalizable case Introduction Projection techniques on Krylov subspaces are currently among the most important tools used to compute eigenvalues and eigenvectors of large sparse non-hermitian matrices The convergence of these methods has been analyzed in detail in the Hermitian case, but the analysis becomes much more difficult in the non-hermitian or non-normal case and so there are few results available in the literature This is in contrast with the convergence analysis of Krylov methods for solving linear systems which received far more attention In this paper we will examine several approaches to the problem Each of these approaches utilizes a different parameter, or set of parameters, which is (are) singled out as the main value (s) on which the analysis depends Often this parameter is difficult to estimate This approach is similar to standard analyses where there is a core expression used as a measure against which an error bound is developed For example, for linear systems, there has been analyses which exploit the polynomial representation of a vector in the Krylov subspace Thus, when solving a linear system Ax = b, in the case when A is diagonalizable, with a matrix X of eigenvectors, the standard bound for GMRES [3] b Ax m 2 κ 2 (X) min p P m max λ Λ(A) p(λ) b Ax 0 2 uses the min-max quantity on the right as a parameter which is then estimated in certain ways Here and throughout the paper v 2 denotes the 2-norm of a vector v and for a matrix A, A 2 denotes its 2-norm and κ 2 (A) denotes its 2-norm condition number of A We will also denote by I the identity matrix and by e k its kth column and assume exact arithmetic In general, the above residual bound for GMRES is not satisfactory because it involves the condition number of X, the matrix that diagonalizes A, which is not known and which can be very large An alternative, exploited in [4] (see also Ipsen [7]), uses as a primary indicator the (,) entry of the inverse of KmK T m where K m is the matrix whose columns are the vectors of the canonical basis of Krylov subspace Other ways to analyze convergence of Krylov methods have been explored For GMRES, one such type of analysis assumes that the field of values does not contain the origin With this, quite a few bounds can be found; we refer for example to [5, 3, 4, 2] Underlying the difficulty is the fact that the norm p(a) 2 is not always Université de Valenciennes, Le Mont Houy, 5933 Valenciennes Cedex, France MohammedBellalij@univ-valenciennesfr University of Minnesota, Dept of Computer Science and Engineering saad@csumnedu Work supported by NSF under grant ACI , and by the Minnesota Supercomputing Institute LMPA, Université du Littoral, 50 rue F Buisson BP699, F Calais Cedex, France sadok@lmpauniv-littoralfr

2 easy to estimate in the non-normal case Yet, most bounds will rely in an implicit way, in an estimate of the minimum of p(a)v 2 over some normalized polynomials In the sequel we will consider three distinct approaches for analyzing the convergence of the Arnoldi process It is important to recall at the outset that, unlike the situation for Hermitian matrices, there are no easy optimality results to be exploited Most of our analysis will be based on estimating the distance from an exact eigenvector from the Krylov subspace 2 Background Consider the eigenvalue problem: find u belonging to C N and λ belonging to C such that A u = λu, (2) where the matrix A is of order N For a given vector v C N, the Krylov subspace K m (A, v) is defined by K m (A, v) = span{v, A v,, A m v} (22) 2 The Arnoldi process The Arnoldi method computes approximate eigenpairs λ (m), ũ (m) by enforcing the standard Petrov-Galerkin condition and ũ (m) K m (A, v), (23) (A ũ (m) λ (m) ũ (m), A i v) = 0 for i = 0,, m (24) The standard way of extracting the approximate eigenpairs from the above conditions is to resort to the Arnoldi algorithm which generates an orthonormal basis v,, v m of K m in which the conditions (24) are expressed Algorithm 2 Arnoldi Input: Initial vector v, and m Set v = v v 2 For j =,, m do Compute w := Av j { hi,j := (w, v For i =,, j, do i ) w := w h i,j v i h j+,j = w 2 ; v j+ = w/h j+,j End The outputs of the algorithm are an othonormal basis V m = [v, v 2,, v m ] and a Hessenberg matrix H m whose entries are the scalars h ij generated by the procedure In addition, the following relations are satisfied: AV m = V m H m + h,m v e m T 2 V T m AV m = H m The approximate eigenvalue problem can now be written as where y (m) = R m z (m) Which is equivalent to V H m AV m y (m) = λ (m) y (m), (25) V H m (A λ (m) I)V m y (m) = 0 Approximate eigenvalues are eigenvalues of H m and are obtained by solving the preceding eigenvalue problem, the associated approximate eigenvectors are ũ (m) = V m y (m) Typically, a few of the outermost eigenvalues will converge first 2

3 22 Convergence Convergence of projection methods for eigenvalue problems, such as the Arnoldi algorithm, are difficult in the non-normal case The simplest analysis, though somewhat incomplete, uses the distance of a given eigenvector from the Krylov subspace It is based on the viewpoint described below, see [] Let P m be the orthogonal projector onto K m Then, the approximate problem amounts to solving or in operator form P m (Ax λx) = 0, x K m, P m AP m x = λx Define, A m P m AP m Then the following theorem is easy to prove (see, []) Theorem 2 Let γ m = P m A(I P m ) 2 Then the residual norms of the pairs λ, P m u and λ, u for the linear operator A m satisfy, respectively (A m λi)p m u 2 γ m (I P m )u 2 (A m λi)u 2 λ 2 + γ 2 m (I P m )u 2 Note that the second bound of the theorem gives an unusual result in that it states how accurate the exact eigenpair is with respect to the approximate problem This is stated in terms of the distance of the exact eigenvector u from the Krylov subspace The remaining issue is how to estimate (I P m )u 2 3 Projection-based analysis In the following we analyze the distance d(w, X) min x X w x 2 in general terms where X is an arbitrary subspace of some dimension m We begin by showing a number of simple results First observe that given any basis V of the subspace X, x can be written as V y, where y C m, so that w x 2 2 = w V y 2 2 = w H w 2w H V y + y H V H V y The above expression is in fact of the form ( ) H ( ) w x 2 w 2 = H w w H V y V H w V H V } {{ } C ( ) y (3) Note that minimizing w x 2 over X is equivalent to minimizing w + x 2 over the same subspace, so the signs of the y s in the above expression can be changed when seeking the minimum distance In the end, min x X w x 2 2 = min w V y C y 2 m 2 = min y C m ( ) H C y ( ) y (32) where C was defined in (3) It is interesting to note the above minimization can be converted into a trivial generalized eigenvalue problem: ( ) H ( ) min C = min z H z H Cz Cz = min y C m y y z C, e H z= z C z H e e H z (33) 3

4 Therefore, the smallest squared distance achieved between the vector w and vectors of the subspace X, is the smallest eigenvalue of the generalized eigenvalue problem Cz = µ(e e H )z This problem has only one finite eigenvalue as can be seen from converting it with the help of the Cholesky factorization C = LL H : LL H z = µ(e e H )z L H z = µ(l e e H L H )L H z µ u = (L e e H L H )u where we have set u = L H z The only nonzero eigenvalue of the rank-one matrix L e e H L H is e H L H L e So µ min = e H L H L e = e H C e Therefore, we have proved the following result: Lemma 3 Let X be an arbitrary subspace with a basis V = [v,, v m ] and let w / X Let P be the orthogonal projector onto X Then, we have where (I P)w 2 2 = e H C e ( C = [w, V ] H w [w, V ] = H w w H V V H w V H V Proof The result was proved above An alternative proof which will help establish some relations is as follows Given an arbitrary vector w C N, observe that (I P)w 2 2 = w H (I P)(I P)w = w H (I P)w = w H w w H Pw with P = V (V H V ) V H From this it follows that (I P)w 2 2 = w H w w H Pw = w H w w H V (V H V ) V H w (34) The right-hand side of (34) is simply the Schur complement of the (,) entry of C, which as is well-known is the inverse of the (,) entry of C Let σ min [w, V ] and σ max [w, V ] be the smallest and largest singular values of [w, V ] Then a consequence of (32) and (33) is that σ min [w, V ] (I P)w 2 σ max [w, V ] (35) This is because e H C e is a Rayleigh quotient of C, and so λ max (C) λ min(c ) e H C e λ max (C ) λ min (C) and the result follows by inverting the above inequalities Clearly, the right part of the bound (35) is too pessimistic We expect (I P)w 2 to be closer to the smallest singular value A sharper result can be obtained by exploiting an appropriate singular vector in (33) Lemma 32 Let σ min [w, V ] be the smallest singular value of [w, V ] and w min the associated right singular vector and assume that e H w min 0 Then, σ min [w, V ] (I P)w 2 σ min[w, V ] e H w min 4 ) (36)

5 Proof To prove the right inequality, we use (32) and (33), and select as particular vector z the right singular vector w min This results in, min w x X x 2 2 wh min Cw min w H min e e H w min = wh min [w, V ][w, V ]H w min w H min e e H w min = σ2 min [w, V ] e H w min 2 The left inequality was established above It also follows from (33) and the observation that e H z < z 2 Consider now using this result for the situation of interest, ie, when X is a Krylov subspace K m and w is an eigenvector u i of A The left side of (36) indicates that we cannot have a good approximation if [u i, V ] is well conditioned Linear dependence of the set [u i, V ] can take place in two ways As expected, the first is when u i is close to the subspace K m The second is when the basis V is ill-conditioned However, in this situation we would also need e H w min to be not too small Note that the above bound has one additional degree of freedom, which is the selection of the basis 4 Analysis in terms of Schur vectors In this section we will exploit a Schur decomposition of A of the form A = QRQ H, where Q is unitary, R is upper triangular with its (,) entry being the eigenvalue to which convergence is being analyzed We thus write R in the form ( ) λ s R = H (4) 0 R It will be assumed that λ is a simple eigenvalue, so the eigenvalues λ 2,, λ N of R are all distinct from λ Since the powers of A are at the basis of Krylov methods, we examine the sequence of the powers of the matrix R Lemma 4 For any k > 0 we have ( ) R k λ k = s H k 0 R k with s H k = s H (λ I R ) (λ k I R) k (42) Proof The proof is by induction and is straightforward If we apply this result to arbitrary polynomials the following corollary will be obtained Corollary 42 For any polynomial p, we have: p(r) = ( p(λ ) s H q(r ) 0 p(r ) ) with q(λ) = p(λ ) p(λ) λ λ (43) It is interesting to note in passing that the above result can be applied recursively with the matrix R replaced by R Now consider the problem of estimating the distance of the first eigenvector, which is q, the first column of Q, from the Krylov subspace We need to minimize q K m y 2 over all vectors y C m, where K m is the Krylov basis K m = [v, Av,, A m v] of K m However, since each basis vector is of the form A j v = QR j Q H v, we can work in the Schur basis and minimize instead e [z, Rz,, R m z]y 2 e p m (R)z 2 5

6 where z = Q H v over polynomials p m of degree m Lemma 43 Let R be given by (4), and z = Q H v where v is of norm unity and let Define z = ( ) ( ; t = η z ɛ m = Then, assuming (t, z) = t H z 0 we have (λ I R ) H s min p(r ) z 2 p P m p(λ )= ɛ m min e p(r)z 2 p P m cos θ(t, z) (44) Proof We define t to be the vector with bottom n components of t, ie, we have, t H = s H (λ I R ) ) and seek first an approximation to ηe by writing ( ) ( p(λ ))η t ηe p(r)z = H [p(λ )I p(r )] z p(r ) z, where the previous corollary was exploited We now select the polynomial p m such that p m (λ ) = and p m (R ) z 2 is minimum Then for this polynomial, ( ) t ηe p m (R)z = H z + t H p m (R ) z p m (R ) z From this we get (η + t H z)e p m (R)z = ) ( t H p m (R ) z p m (R ) z Using the notation defined above for ɛ m, dividing both sides by η + t H z, under the assumption that η + t H z 0, results in e p ) m (R) (η + t H z) z = ( t H p m (R ) z (η + t H z) p m (R ) z Calling p the polynomial p m (λ)/[η + t H z] and taking 2-norms of both sides, gives the following upper bound, with the help of the Cauchy-Schwarz inequality, + t 2 2 e p(r)z 2 ɛ η + t H m (45) z The numerator of the fraction in the right-hand side of (45) is the norm of the vector t, and the denominator is the absolute value of the inner product (t, z) Since we assumed that z 2 = then the factor in front of the term ɛ m in (45) is the inverse 6

7 of the cosine of the angle between t and z The minimum on the left-hand side of (44) does not exceed the right-hand side of (45), so the result is proved The lemma can be translated in terms of known quantities related to the original matrix Theorem 44 Let w be the left eigenvector of A associated with λ and assume that cos θ(w, v) 0 Let P be the orthogonal projector onto the right eigenspace associated with λ, ie, : P = q q H, and let B be the linear operator B = (I P )A(I P ) Define Then, we have ɛ m = min p(b )(I P )v 2 (46) p P m p(λ )= (I P m )q 2 = min y C m q K m y 2 ɛ m cos θ(w, v) (47) Proof As was stated above, the theorem is nothing but a translation of the lemma into the Q basis First, we have already seen that (I P m )q 2 = min q K m y 2 = min q p(a)v 2 = min e p(r)z 2 y p P m p P m which only re-expresses the quantity q p(a)v 2 in the basis Q, so q p(a)v 2 = e p(r)z 2 Second, it can be seen that the vector w = Qt, where t is defined in the lemma is a left eigenvector associated with λ Indeed we have ( ) w H (λ I A) = t H Q H Q(λ I R)Q H = [ s H (λ I R ) 0 s H ] Q H = 0 0 λ I R So, (w, v) = (Qt, Qz) = (t, z) Finally, the scalar ɛ m in this theorem is identical with the one in the lemma It is just expressed with a different basis Indeed, in the Q basis the vector (I P )v is z In the same basis, the operator (I P )A(I P ) is represented by the matrix R The above inequality gives an analysis in terms of the variable ɛ m It differs from other bounds by not using eigenbases or spectral expansions, [, 5] Whether or not eigenvectors are used, there are unknown quantities On the one hand the eigenbasis expansion can lead to large coefficients (Ill-conditioned bases for example) The Schur factorization on the other hand does not have such difficulties as it should retain the non-normality effects in the quantitie ɛ m A number of papers give a detailed analysis of the minimum of p(a)v 2 or p(a) 2 under a normalization assumption of the form p(0) =, see for example the papers [, 2] In [2] for example, it is shown that there is a universal constant ĉ 3375 such that if W (A) is the field of values of A then for any polynomial p(a) 2 ĉ max p(z) z W (A) These bounds can easily be adapted to our situation to have an idea of the scalar ɛ m Consider, for example, the situation where λ is the dominant eigenvalue and the other eigenvalues are located in a disk of center c and radius r not containing 7

8 λ For illustration we will take instead of the optimal polynomial, a simple power polynomial, namely Then, p k (λ) = (λ c)k (λ c) k p k (λ ) = ; p k (R ) = ( ) k R ci λ c Notice that the spectral radius of R ci λ c is ( ) R ci r ρ = λ c λ c < so (R ci) k /(λ c) k tends to zero We can write ˆɛ m p m (R ) z 2 = (R ci) m z 2 λ c m = m ( r λ c ) m where m is a sequence which converges to a certain constant z 2 One of the difficulties of all methods based on an analysis of this sort, is the well-known fact that the sequence m can become very large before settling to its limit This is characteristic of highly non-normal matrices 5 Analysis in terms of eigenvectors A common technique for estimating (I P m )u i 2 assumes that A is diagonalizable and expands v, the first vector of the Krylov sequence, in the eigen-basis If A is diagonizable then for some matrix of eigenvectors U, we have A = U Λ U, where Λ = diag(λ,, λ N ) We examine the convergence of a given eigenvalue which is indexed by, ie, we consider u, the -st column column of U The initial vector v is expanded in the eigen-basis as v = N j= α ju j It is assumed throughout that α 0 and u j 2 = for all j In [], it was shown that in the general non-hermitian case, under the above assumptions, we have where ξ = j α j α (I P m )u 2 ξ ɛ (m) (5) and ɛ (m) = min n p Pm p(λ )= max p(λ j) (52) j This result, which requires a few easy manipulations to prove, does not exploit the orthogonality of the eigenbasis in the normal case In this particular case, the same result holds but ξ can be sharpened to n α j 2 ξ,normal = which represents the tangent of the angle between u and v Note that the distance (I P m )u 2 is nothing but the sine of the angle (u, K m ) Once an estimate for ɛ (m) is available, Theorem 2 can be invoked to give an idea on the residual of the exact eigenpair with respect to the approximate (projected) problem See also [5, th 30] for an alternative approach which exploits the spectral decomposition 8 α,

9 5 The approximation theory viewpoint What is left to do is to estimate ɛ (m) For the sake of notational convenience, we will consider estimating ɛ () instead of ɛ (m) The underlying problem is one of approximation theory For any continuous function f defined on a compact set Ω, denote the uniform norm: f = max z Ω f(z) (53) The set Ω will later be taken to be the spectrum of A excluding λ Estimating ɛ () amounts to finding an upper bound for the distance, in the sense of the inf-norm just defined, between the function f(z) = and polynomials of degree m of the form p(z) = (z λ )q(z), or, equivalently: ɛ () = min q P m (z λ )q(z) We recall that a subspace S of continuous functions on Ω, generated by k functions φ,, φ k satisfies the Haar condition if each function in S has at most k distinct roots This means that any linear combination of the φ i s vanishes iff it has k distinct roots in Ω Let f be a continuous function and let p be the best uniform approximation of f over Ω The difference f p reaches its maximum modulus at a number of extremal points The characteristic property [0] of the best approximation states the following Theorem 5 Let f be a continous function and S a k-dimensional subspace of the space of continuous functions on Ω, which satisfies the Haar condition Then p S is the best uniform approximation of f over Ω, iff there exist r extremal points z i, i =,, r in Ω, and positive numbers µ,, µ r, with k + r 2k + such that r µ i [f(z i ) p (z i )]φ(z i ) = 0 φ S (54) i= One important point here is that the number of extremal points is only known to be between k + and 2k + in the general complex case That r must be k + is a consequence of the Haar condition and can be readily verified When Ω is real, then r = k + The fact that r is only known to be 2k + in the complex case, comes from Caratheodory s characterization of convex hulls which expresses a point in co(ω), the convex hull of Ω, as a convex combination of k + points of Ω in real spaces and 2k + points of Ω in complex spaces We will now translate the above result for our situation Let Ω = Λ(A)\{λ } and S = span{φ j (z)} j=,,m where φ j (z) = (z λ )z j Then, the dimension of S is m and therefore the theorem states that there are r eigenvalues from the set Ω, with m + r 2m + such that r µ k [ (λ k+ λ )q (λ k+ )]φ j (λ k+ ) = 0 j =,, m k= Although we do not know how many extremal points there are we can still express the best polynomial by selecting any set of m extremal points Assume without loss of generality that these points are labeled from 2 to m + Let p (z) = (z λ )q (z) We can write p (λ k ) at each of the extremal points λ k as p (λ k ) = ρe iθ k 9

10 where θ k is a real number and ρ is real and positive Then it is easily seen that p (z) = m+2 e iθ k l k (z), (55) e iθ k lk (λ ) m+2 where each l k (z) is the Lagrange polynomial: l k (z) = m+2 j k z λ j λ k λ j (56) Indeed, p (z), which is of degree m, takes the values ρe iθ k at the m + points λ 2,, λ m+2 Therefore it is uniquely determined by the Lagrange formula p (z) = m+2 ρe iθ k l k (z) In addition p (λ ) = and this determines ρ as the inverse of m+2 e iθ k l k (λ ), yielding the relation (55) This establishes the following, Theorem 52 There are r eigenvalues in Ω = Λ(A)\{λ }, where m + r 2m +, at which the optimal polynomial p (z) = (z λ )q (z) reaches its maximum value In addition, given any subset of m + among these r eigenvalues, which can be labeled λ 2, λ 3,, λ m+2, the polynomial can be represented by (55) In particular, ɛ () = m+2 e iθ m+2 k j k (57) λ λ j λ k λ j Proof The result was proved above Note that ɛ () is equal to ρ which is the inverse of the denominator in (55) In [] it was shown that when r = m +, then the sign e iθ k in the denominator of (57) becomes equal to the conjugate of the sign of l k (λ ), which is the product term in the denominator of (57) In this case, (57) simplifies to ɛ () = m+2 (58) λ λj λ k λ j The result in [] was stated incorrectly for the general complex case, because the lemma on which it was based is only valid for real functions Therefore, the result holds true only for the situation when the spectrum is real or when it is known that r = m + (eg, when N = m + ) A proof of this result is given in Appendix 52 Optimization viewpoint An explicit formula for (I P m )u 2 is obtained immediatly from Lemma 3 0 m+2 j k

11 Corollary 53 Let L be the rectangular matrix of C N (), with columnvectors α u, v, A v,, A m v, then (I P m ) α u 2 = e H (LH L ) e (59) Proof The result follows by applying Lemma 3 with w = α u, and V = [v, Av,, A m v] We now consider a factorization of the matrix L If we set α to be the vector of C m such that v = U α, then we obtain A j v = U Λ j α for j = 0,, m Then, we have L = [α u, v, A v,, A m v] = U [α e, α, Λ α,, Λ m α] = U D α W, with D α = α α α N 2 and W = 0 λ i λ m i 0 λ N λ m N λ λ m 0 λ 2 λ m (50) As a result, L H L = W H D H α (U H U) D α W (5) With theses formulas, it is now possible to express the residual norms of the Arnoldi process in terms of eigenvalues, eigenvectors, and the expression of v in the eigenbasis Theorem 54 (I P m ) α u 2 = e H (W H DH α (U H U)D α W ) e In the case of normal matrices (A H A = A A H ), we have U H U = I, so the preceding formula simplifies to (I P m ) α u 2 = e H (W H DH α D α W ) e (52) It is interesting to consider first the particular situation when m = If m =, and A is normal, a little direct calculation yields, α 2 N α i 2 (I P ) α u 2 i=2 = N α i 2 i=

12 Now, let β the vector whose components are defined by β i = α i 2 N j= α j 2 We then obtain, as a consequence of the above equality, (I P ) u = N β i = β Note that β = cos 2 θ(u, v), where θ(u, v) is the angle between v and u, and so the above relation can be expressed as (I P ) u = sin θ(u, v), which is another expression for the well-known relation (I P m )u = sin θ(u, K m ), for the case m = Observe that the bound given in (5) does not exploit the orthogonality of the eigenvectors (for the normal case) Indeed, for the situation when m =, it yields i=2 (I P ) u N α i α (53) i=2 If we exploit this orthogonality in the proof of the result, we would obtain a slightly N sharper bound in which the numerator in (53) is replaced by i=2 α i 2 For the general case (m ), we will derive the optimal bound for (I P m ) α u 2 α 2 = f m (β), by solving the following optimization problem where min β 0,,β n 0 NP β i= i= f m (β), f m (β) = e H (W H D β W ) e We now state the main result which gives an upper bound for the general situation Theorem 55 If m < N and (I P k ) u 0 for k {,, m}, then if the matrix A is normal then (I P m ) u α α η(m) (54) 2 if the matrix A is non normal but diagonalizable then (I P m ) u N α j j= α η (m), (55) where η (m) is defined as follows 2

13 if all eigenvalues are real or if m = N, there exists m eigenvalues, labeled λ 2,, λ such that η (m) = + j k λ j λ λ j λ k if at least one eigenvalue is non real and if m < N, there exists m eigenvalues, labeled λ 2,, λ, and m real numbers, θ 2,, θ such that η (m) = e ıθ k j k (λ j λ ) (λ j λ k ) The proof is given in Appendix 2 We now make a few remarks In order to compare the bound given in (5) and the bound given in Theorem 55, we need to unravel a relationship between ɛ (m) and η (m) From the expression of ɛ (m) we deduce that η (m) = ɛ(m) + ɛ (m) Hence if the matrix is normal, the inequality (54) becomes (I P m ) u α α ɛ (m) + ɛ (m) (56) A second remark is that if m = N and λ k = (k )/(N ), k =,, N (Uniform distribution) then (56) reduces to while (5) becomes (I P N ) u N i= α i 2 α 2 N, N i=2 (I P N ) u α i α 2 N, If the components of v in the eigen-decomposition are of equal size, ie, α i = w, i {,, N} then we obtain for normal matrices N (I P N ) u 2 N, while (5) becomes (I P N ) u N 2 N In general, for normal matrices the bound (56) is a slight refinement of (5) This comes from the fact that the bound (5) restricts the polynomials with the constraint p(λ ) = to obtain a simple result There is no such restriction with the above result 3

14 Example We now examine an example to illustrate Theorem 55 in the complex case We consider the following diagonal matrix, A = ı 0, and let v = (α, α 2, α 3, α 4 ) T We have u j = e j for j =,, 4 Let us set λ = and consider the second step m = 2 Using (52), we have where (I P 2 ) α e 2 = α 2 f 2 (β, β 2, β 3, β 4 ) f 2 (β, β 2, β 3, β 4 ) = β β 2 + 2β β 3 + β 2 β 3 + 4β β 4 + β 2 β 4 + 2β 3 β 4 β (β 2 β 3 + β 2 β 4 + 2β 3 β 4 ) Solving the optimization problem min f 2 (β), β 0,β 2 0,β 3 0,β 4 0 β +β 2+β 3+β 4= we obtain: f 2 (β ) = = ( + 5) 2 and 5 β =, β2 = 5 5, β3 = 5 5, β4 = The optimal bounds are (I P ) α e α 2 and 3 The real numbers θ 2, θ 3, θ 4 are such that e ı θ2 (I P 2 ) α e α + 5 = 2 + ı 5, e ı θ3 = + 2 ı 5, and e ı θ4 = 2 ı 5 It is important to remark that the vector β solution of the minimization problem is unique and all its components are non zero If all the eigenvalues are real and m = 2, we can show that there exists a solution vector with only three non zero components, as we will see in the proof of Theorem 55 6 Conclusion The analysis of convergence of the Arnoldi process for computing eigenvalues and eigenvectors is difficult and results in the non-normal case are bound to be limited in scope This is essentially because the behavior of polynomials of A or simply of the succesive powers A k, can by itself be difficult to predict All three forms of analysis covered in this paper (based on projectors, or the Schur form of A, or the diagonalization of A), confront this limitation What is important is that some insight can be gained from each of these forms of analysis The Schur form tells us that we can reduce the problem to that of estimaing min p(b )ˆv where B is a reduced Schur form of A, and ˆv is the orthogonal projection of the initial vector v onto the orthogonal of the eigenvector u The analysis based on eigenvectors works essentially in the eigenbasis (assuming there is one) and converts the problem 4

15 of estimating the error into a min-max problem Here, there are three limitations The first comes from the nature of Theorem 2 which expresses the residual for the projected problem of the exact eigenpair, not the other way around as desired The second one comes from the fact that the bounds utilize the eigen-coefficients α i of the initial vector in the eigenbasis These coefficients can be very large in case the basis is ill conditioned The last problem comes from the fact that the min-max quantities required by the bounds (ɛ (m), η (m) ) can themselves very difficult to estimate REFERENCES [] B Beckermann Image numérique, GMRES et polynomes de Faber CRAS (Proceedings of the French Academy of Sciences), 2006 In French [2] M Crouzeix Numerical range, holomorphic calculus, and applications, 2005 Manuscript [3] M Eiermann Fields of values and iterative methods Linear Algebra and its Applications, 80:67 97, 993 [4] M Eiermann and O G Ernst Geometric aspects of the theory of krylov subspace methods Acta Numerica, 0:25 32, 200 [5] M Eiermann, O G Ernst, and O Schneider Analysis of acceleration strategies for restarted minimal residual methods J Comput Appl Math, 23: , 2000 [6] R Fletcher Practical methods of optimisation Wiley, Chichester, 2nd edition, 987 [7] I C F Ipsen Expressions and bounds for the gmres residuals BIT, 40: , 2000 [8] P Lancaster and M Tismenetsky The Theory of Matrices Academic Press, New York, second edition, 985 [9] G Meurant Computer solution of large linear systems North-Holland, Amsterdam, 999 Vol 28, Studies in Mathematics and its Applications [0] T J Rivlin The Chebyshev Polynomials: from Approximation Theory to Algebra and Number Theory J Wiley and Sons, New York, 990 [] Y Saad Numerical Methods for Large Eigenvalue Problems Halstead Press, New York, 992 [2] Y Saad Further analysis of minimal residual iterations Numerical Linear Algebra with Applications, 7:67 93, 2000 [3] Y Saad Iterative Methods for Sparse Linear Systems, 2nd edition SIAM, Philadelpha, PA, 2003 [4] H Sadok Analysis of the convergence of the minimal and the orthogonal residual methods Numer Algorithms, 40: 5, 2005 [5] G W Stewart Matrix Algorithms II: Eigensystems SIAM, Philadelphia, 200 [6] F Zhang Matrix Theory Springer Verlag, New York, 999 5

16 Appendix We denote by P m the set of polynomials p of degree m such that p(λ ) = 0 We seek the best uniform approximation of the function f(z) = by polynomials of degree m in P m Note that P m is of dimension m Let the set of r extremal points be λ 2,, λ r+ (See theorem 5) According to Theorem 52 given any subset of m + among these r extremal points, which we label λ 2, λ 3,, λ m+2, the best polynomial can be represented by (55) in which e iθ k = sign( p (λ k )) Not much can be said from this result in the general situation However, when r = m +, then we can determine max p (z) In this situation the necessary and sufficient conditions of Theorem 5 express the extremal points as follows Let us set ξ j µ j [f(z j ) p (z j )] for j =, m +, and select any basis φ,, φ m of the polynomial subspace P m Then, the condition (54) translates to k= ξ k φ j (λ k ) = 0 for j =,, m (6) The above equations constitute an underdetermined system of linear equations with the unknowns ξ k In fact, since the ξ k s are all nonzero, we can fix any one component, and the rest will then be determined uniquely This is best done in a more convenient basis of polynomials given by: ω j (z) = (z λ )ˆl j (z), j = 2,, m +, (62) where ˆl j is the Lagrange polynomial of degree m, ˆlj (z) = k j z λ k λ j λ k, j = 2,, m + (63) With this we can prove the following lemma Lemma 6 The underdetermined linear system of m equations and m + unknowns ξ k, k = 2,, m + 2 m+2 admits the nontrivial solution ω j (λ k )ξ k = 0, j = 2, 3,, m + (64) ξ k = m+2 j k λ λ j λ k λ j, k = 2,, m + 2 (65) Proof The proof requires a straightforward algebraic verification; see [] for details We can prove the main result of this appendix Theorem 62 Let p be the (unique) polynomial of degree m satisfying the constraint p(λ ) = 0, and which is the best uniform approximation to the function f(z) = on a compact set Ω consisting of at least m + points Assume that there are m + extremal points labeled λ 2,, λ m+2 and let ξ k, k = 2,, m + 2 be any solution of the linear system (64) Write each ξ k in the form ξ k = k e iθ k where k 6

17 is real and positive and θ k is real Then, p can be expressed as p (z) = m+2 m+2 e iθ k l k (z) l k (λ ) (66) where l k is the Lagrange polynomial of degree m l k (z) = m+2 j k z λ j λ k λ j As a consequence, ɛ () =,k j λ k λ λ k λ j (67) Proof Equation (54) states that m+2 p (z) = ρ e iθ k l k (z) with ρ = m+2 eiθ k lk (λ ) (68) We now apply Theorem 5 which states that at the extremal points (now known to be unique) there are m + positive coefficients µ j such that ξ k µ k [ p (λ k )] = ρµ k e iθ k (69) satisfy the system (6) As was already mentioned, the solution to (6) is uniquely determined if we set any one of its components Set ξ m+2 = l m+2 (λ ) Then, according to Lemma 6, we must have ξ k = l k (λ ), for k = 2,, m + 2 Since ρ and µ k are positive, (69) shows that e iθ k = sign(l k (λ )) e iθ k = l k(λ ) l k (λ ) ρ = m+2 l k(λ ) The result (67) is an expanded version of the above expression for ρ 7

18 Appendix 2: Proof of Theorem 55 We will consider the 2 cases A normal and A non-normal separately In this appendix, the notation for the 2-norm is changed to Case : A is normal To prove the first part of this Theorem, we need two lemmas Lemma 63 Let ω, ω 2,, ω m be m distinct complex numbers and V the m m Vandermonde matrix whose entries V i,j are given by : v (i, j) = ω i j, i, j =,, m and let y = (, ρ, ρ 2,, ρ m ) T Then, the dual Vandermonde system V T admits the unique solution x = (x, x 2,, x m ) T, where x i = ( ) ρ ωj ω i ω j j m j i x = y The proof is obvious Now, let us recall that β is the vector whose components are defined by β i = α i 2 N j= α j 2 and define Z (β) to be the matrix Z (β) = W H D β W C,, where D β and W have been defined in (50) If the matrix function Z (β) is nonsingular, we define the functions f m by f m (β) = e H Z (β) e, with β S = { β = (β,, β N ) [0, ] N / } N β i = i= Then we have the following result Lemma 64 If the matrix function Z (β) is nonsingular, then the following properties hold : f m is homogeneous of degree, ie, f(rβ) = r f(β) for r > 0 2 f m is a convex function defined on the closed convex set S 3 f m is differentiable at β S and we have f m β i (β) = e H i W t 2, where t = (t, t 2,, t ) T is such that Z (β) t = e Moreover, we have N i= β f m (β) i = f m (β) β i Proof Let r be some positive real Since D r β = r D β, then f m (rβ) = e H (r (W H D β W )) e = r f m (β) 8

19 2 It is easy to verify that the set S is convex By taking x = e, G = Z (rβ) and G 2 = Z (( r)β ) where β, β S and 0 < r < in the inequality x H ( rg + ( r) G 2 ) x x H r (G ) x + x H ( r) (G 2 ) x where G and G 2 are positive definite hermitian matrices [6, p 74], we obtain f m (rβ + ( r)β ) r f m (β) + ( r) f m (β ) Hence f m is convex 3 Clearly, f m is differentiable at β S By using the derivative of the inverse of the matrix function, we have Z (β) = Z β (β) Z (β) Z i β (β) i It follows that where E i = e i e T i Z (β) = Z β (β) W H E i W Z (β) i R N,N Therefore, f m β i (β) = e T i W t 2 The last equality follows from a simple algebraic manipulation, noting that N i= β i f m (β) β i = N t H W(β H i e i e H i )W t = t H WD H β W t = f(β) i= Proof of Theorem 55 From (52) we deduce that (I P m ) α u 2 = α 2 e H (W H D βw ) e Since β i 0 and N i= β i =, in order to get an optimal bound of (I P m ) α u 2 α 2 = f m (β), we must solve the following optimization problem min β 0,,β N 0 NP β i= i= f m (β) (60) We introduce the Lagrangian function for this problem : ( ) N N L m (β,, µ) = f m (β) β i µ i β i, where R ; µ = (µ,, µ N ) R N According to the Karush-Kuhn-Tucker (KKT) conditions, if f has a local minimizer β in S, then there exist Lagrangian multipliers, µ = (µ,, µ N ), such that (β,, µ ), satisfy the following conditions: 9 i= i=

20 (i) L m ( β,, µ ) = 0, for i =,, N, β i (ii) ( N βi ) = 0 and β i 0, for i =,, N i= (iii) µ i β i = 0 for i =,, N, (iv) µ i 0 for i =,, N We note that in this case the KKT conditions are also sufficient since the problem is convex Condition (i) and Lemma 64 give where L m β i (β,, µ ) = e H i W t 2 + µ i = 0, (6) t = Z (β ) e From (iii), it follows that for i =,, N, either µ i = 0 or βi = 0 It can be shown that β 0 Indeed, notice that the best approximation to u from K m is P m u = K(K T K) (K T u ) and its first component β = u T K(K T K) (K T u ) is clearly zero iff K T u = 0 So if α 0 then β 0 We also notice that the vector β = (β,, βn ) has exactly m + s non zeros components which will be labeled β,, βm+s, with s It follows that β = (β,, βm+s, 0,, 0) and µ i = 0 for i =,, m + s Substituting in (6) it follows that e H i W t 2 =, for i =,, m + s, (62) where t = (t,, t m) T = (W H D β W ) e Hence, we have e H i W t = e ıθi, for i =,, m + s, (63) where ı = and θ i R We then obtain the following linear system t + t 2 + λ t λ m t = e ıθ t 2 + λ 2 t λ m 2 t = e ıθ2 t 2 + λ m+s t λ m m+s t = e ıθm+s (64) From Lemma 64, we can easily show that t = fm (β ) = = e ıθ λ λ m e ıθ2 λ 2 λ m 2 e ıθ λ λ m λ 2 λ m (65) 2 λ 3 λ m 3 λ λ m 20

21 Now, since W H D β W t = e, we deduce that e ıθ β = m+s, and i= λ j i e ıθi β i = 0, for j =,, m (66) Since β is a positive real number, we obviously have: β = and eıθ = Hence, expanding the numerator of (65) with respect to its first column, gives where l k (ω) = = j k e ıθ k j k (λ j λ ) (λ j λ k ) = e ıθ k l k (λ ), (67) (λ j ω) (λ j λ k ) It is obvious that θ 2,, θ are such that eıθ k l k (λ ) is real The vector (e ıθ2 β2,, e ıθm+s βm+s) T satisfies (66), which is equivalent to the following linear system e ıθ2 β e ıθm+s β m+s =, λ 2 e ıθ2 β λ m+s e ıθm+s β m+s = λ, λ m 2 e ıθ2 β2 + + λ m m+s e ıθm+s βm+s = λm (68) To obtain β l and e ıθ l for l = 2,, m + s, we have to solve (68) Let us derive the Lagrange multipliers µ l We have µ l = 0, for l =,, m + s and the other components are deduced from (6): µ l = t 2 + λ l t λ m l t 2 for l = m + s +,, N In view of (64), and using Lemma 63, we obtain 2 µ l = e ıθ (λ k j λ l ) (λ j λ k ) j k 0 for l = m + s +,, N Hence the eigenvalues λ 2,, λ and the scalars θ 2,, θ must be chosen such that e ıθ k l k (λ l ), for l = m + s +,, N We have = p m (λ ), where p m is the complex valued polynomial of degree m satisfying the following conditions : p m (ω) = eıθ k l k (ω), p m (λ ) = 2, p m (λ j ) = e ıθj for j = 2,, m + s, p m (λ j ) for j = m + s +,, N 2 (C)

22 To obtain the vector solution of the optimization problem we need to solve the linear system (68) This can readily be solved by Cramer s rules if for example s = If s > and all the eigenvalues are real the system (68) will have infinitely many solutions But if one of the eigenvalues λ l is complex, this system will have a unique solution We will now investigate the possible values that s can take There are 3 distinct possibilities ) Case s = This means that the system (68) has a solution with exactly m + non zeros elements Using Lemma 63 for solving (68), we obtain e ıθ l β l = l l(λ ) for l = 2,, m + We can deduce β l and e ıθ l, by noticing that β l 0 So we get for l = 2,, m + β l = l l(λ ) and e ıθ l = l l(λ ) l l (λ ) (69) Moreover β = and β l = 0, for l = m + 2,, N Now, since N l= β l =, then = + l=2 j l λ j λ λ j λ l (620) Since is the minimum of the function f m (β), we deduce that λ 2,, λ are such that l=2 j l λ j λ λ j λ l = min λ i,,λ i l=2 j l λij λ λ ij λ il Since s =, the minimum is attained only for the set {λ 2,, λ } which will be called the extremal set 2) Case when s > and all the eigenvalues are real In this case e ıθ k = ± Let us set σ j = ζ j β j, where ζ j = ± the system (68) can be written as σ σ = σ m+2 σ m+s, λ 2 σ λ σ = λ λ m+2σ m+2 λ m+s σ m+s, λ2 m σ λ m σ = λm λm m+2 σ m+2 λ m m+sσ m+s (62) Using lemma 63, we obtain for l = 2,, m + ζ l β l = σ l = l l(λ ) σ m+2 l l (λ m+2 ) σ m+s l l (λ m+s ) 22

23 On the other hand, the conditions (C) can be written in that case p m (ω) = ζ k l k (ω), p m (λ ) 2, p m (λ j ) = 2 for j Θ = {j {2,, m + s}, ζ j = } p m (λ j ) = 0 for j Θ + = {j {2,, m + s}, ζ j = } p m (λ j ) [0, 2] for j = m + s +,, N (C) First, we dispose of the case m = 2 If m = 2, conditions (C) imply that s = or p m 2 We assume now that m 3 In this case the polynomial p m is not constant and we have p m (0) > 2 The sets Θ and Θ contain at least one element otherwise p m 0 or p m 2, which is impossible since p m (λ ) > 2 Moreover, since {2,, m + s} = Θ + Θ, the polynomials p m and 2 p m are of degree m and will have at most m zero Therefore, the number of elements of Θ + and Θ is less or equal than m Hence s m The polynomial p m is of degree m 2 and will have m 2 zeros If we assume that λ 2 < λ 3 < < λ, there exists an extremal set of eigenvalues {λ 2,, λ } with alternating sign, ie, ζ j = ( ) j ζ 2 So in the real case (all the eigenvalues are real), there exists a solution β of the system (62), with only m + non zeros components: β l = l l(λ ) for l = 2,, m + Invoking the fact that N β k = we obtain = + l=2 And the extremal {λ 2,, λ } is such that l l (λ ) l=2 j l λ j λ λ j λ l = min λ i,,λ i l=2 j l λij λ λij λ il This extremal set is not the unique set verifying the preceding property 3) Case when s > and one of the eigenvalues is complex Here the situation is more complicated If we consider the case m = 2 The conditions (C) now give p 2 (ω) = e ıθ2 l 2 (ω) e ıθ3 l 3 (ω), p 2 (λ ) = 2, p 2 (λ j ) = e ıθj for j = 2,, 2 + s, p 2 (λ j ) for j = s + 3,, N (C2) We can have s = 2 (see the example) And it is not easy to explicit in general the real numbers θ 2,, θ m+s We can only conclude that if there exists a solution of the optimization problem (60) with only m + non zeros components then the optimal solution is given by (620) Otherwise the optimal solution satisfies (67) 23

24 Case 2: A is non normal but diagonalizable Let be the diagonal matrix defined by α 0 0 = 0 α2 N 0 α j 0 0 αn j= By Theorem 54, we may write (I P m ) α u 2 = Using the Courant-Ficher theorem, we obtain e H (W H D H α U H U D α W ) e (I P m ) α u 2 U D α 2 e H (W H 2 W ) e Moreover for all vector x = (x,, x N ) T, we have U D α x = N N α j j= j= α j αj x ju j Since u i =, we have by the triangle inequality U D α x N N α j α j αj x j = N α j j= j= j= N j= α j x j N The Cauchy-Schwarz inequality implies that U D α α j, since We thus have (I P m ) α u 2 N U D α x α j x j= j= ( N ) 2 j= α j f m (ρ,, ρ N ), where ρ α j j = N k= α k Since ρ j 0 and N k= ρ k =, we conclude that ( N ) 2 (I P m ) α u 2 j= α j min ρ 0,,ρ N 0 f m (ρ,, ρ N ) = ρ ++ρ N = This completes the proof by denoting η (k) = ( N ) 2 j= α j Let us consider the following examples for illustrating the conditions (C) given in the preceding proof 24

25 Example 2 If λ k = k, k =,, N If N is even ( N = 2n > 4) If m = 2, then s = and the extremal set is unique and is {2, N}, the polynomial verifying the conditions (C) is p 2 (w) = 2 2 w 2 and we N 2 have p 2 () = 2n n, β = n 2n, β 2 = 2, β 3 = 2(2n ), therefore (I P 2 ) α u α n 2n If m = 3, then s = and the extremal set is {2, n +, N}, and we have (w 2)(w N) p 3 (w) = 2 2 (n )(n + N), and (I P 3 ) α u α (n ) 2(n + ) 2 If N is odd (N = 2n + ) If m = 2, then s = and the extremal set is unique and is {2, N}, the polynomial verifying the conditions (C) is p 2 (w) = 2 2 w 2 and we N 2 have (I P 2 ) α u α 2n 4n If m = 3, then s = 2 The sets {2, n +, 2n + } and {2, n + 2, 2n + } are the two extremal sets with alternating sign The polynomial p 3 is (w 2)(w N) given by p 3 (w) = 2 2, and we have (n )(n + N) (I P 3 ) α u α 2n + 2 2(n ) Notice that p 3 (n + ) = p 3 (n + 2) = 0 And the solution of the system 68, is β = 2n + 2 β 2 = 2(n ) n 2 2n 2 + n β 4 β3 = n + β 4 β5 n = 2(2n 2 + n ) + 2 β 4 [ ] β4 0, n + n + 2n 2 + n n + 2(2n 2 + n ) We remark that neither β2 nor β5 can be zero Example 3 Here we reconsider Example ( N = 4, λ =, λ 2 = 2, λ 3 = 2 + ı, λ 4 = 3) For m = 2, we have s = 2 and the polynomial p 2 verifying conditions (C2) is p 2 (w) = + 8 i 3 i w, and we have p 2 () =