Numerische Mathematik

Transcription

1 Numer. Math. (009) 11:67 93 DOI /s Numerische Mathematik The rate of convergence of GMRES on a tridiagonal Toeplitz linear system Ren-Cang Li Wei Zhang Received: 1 November 007 / Revised: 6 August 008 / Published online: 19 December 008 Springer-Verlag 008 Abstract The Generalized Minimal Residual method (GMRES) is often used to solve a nonsymmetric linear system Ax = b. But its convergence analysis is a rather difficult task in general. A commonly used approach is to diagonalize A = X X 1 and then separate the study of GMRES convergence behavior into optimizing the condition number of X and a polynomial minimization problem over A s spectrum. This artificial separation could greatly overestimate GMRES residuals and likely yields error bounds that are too far from the actual ones. On the other hand, considering the effects of both A s spectrum and the conditioning of X at the same time poses a difficult challenge, perhaps impossible to deal with in general but only possible for certain particular linear systems. This paper will do so for a (nonsymmetric) tridiagonal Toeplitz system. Sharp error bounds on and sometimes exact expressions for residuals are obtained. These expressions and/or bounds are in terms of the three parameters that define A and Chebyshev polynomials of the first kind. Mathematics Subject Classification (000) 65F10 1 Introduction The Generalized Minimal Residual method (GMRES) is often used to solve a nonsymmetric linear system Ax = b. The basic idea is to seek approximate solutions, R.-C. Li (B) Department of Mathematics, University of Texas at Arlington, P.O. Box 19408, Arlington, TX 76019, USA rcli@uta.edu W. Zhang Department of Mathematics, University of Kentucky, Lexington, KY 40506, USA wzhang@ms.uky.edu 13

2 68 R.-C. Li, W. Zhang optimal in certain sense, from the so-called Krylov subspaces. Specifically, the kth approximation x k is sought so that the kth residual r k = b Ax k satisfies [1] (without loss of generality, we take initially x 0 = 0 and thus r 0 = b). r k = min y K k b Ay, where the kth Krylov subspace K k K k (A, b) of A on b is defined as K k K k (A, b) def = span{b, Ab,...,A k 1 b}, (1.1) and generic norm is the usual l norm of a vector or the spectral norm of a matrix. This paper is concerned with the convergence analysis of GMRES on linear system Ax = b whose coefficient matrix A is a (nonsymmetric) tridiagonal Toeplitz coefficient matrix: λ µ A = ν µ, ν λ where λ, µ, ν are assumed nonzero and possibly complex. When µ = ν, A is normal, including symmetric and symmetric positive definite as subcases, and convergence analysis of GMRES (or the conjugate gradient method) on the corresponding linear systems has been well studied (see [14] and the references therein). This paper will focus on the nonsymmetric case: µ = ν. Throughout this paper, exact arithmetic is assumed, A is N-by-N, and k is GMRES iteration index. Since in exact arithmetic GMRES computes the exact solution in at most N steps, r N = 0. For this reason, we restrict k < N at all times. Our first main contribution in this paper is the following error bound (Theorem.1) r k k k + 1 ζ j Tj (τ) r 0 j=0 1/ where T j (t) is the jth Chebyshev polynomial of the first kind, and µν ξ =, τ = λ ν µν, ζ = min{ ξ, ξ 1 }., (1.) We will also prove that this upper bound is nearly achieved by b = e 1 (the first column of the identity matrix) when ξ 1orbyb = e N (the last column of the identity matrix) when ξ 1. By nearly achieved, we mean it is within a factor about at most (k + 1) 3/ of the exact residual ratios. 13

3 Convergence of GMRES on tridiagonal Toeplitz linear system 69 Our second main contribution is about the worst asymptotic speed of r k among all possible r 0. It is proven that (Theorem.) [ ] lim inf r k 1/k [ ] sup = lim r 0 r 0 sup r k 1/k { } sup = min (ζρ) 1, 1, (1.3) r 0 r 0 { τ where ρ = max + τ 1, τ } τ 1. Equivalently, the limits in (1.3) can be interpreted as follows: for any given ɛ>0, there exists a positive integer K such that [ ] r k 1/k { sup min (ζρ) 1, 1} <ɛ for all N > k K. (1.4) r 0 r 0 It is worth mentioning that sup r0 among all possible r 0 in (1.3) can be replaced by sup r0 {e 1,e N }. A related work that also studied asymptotic speed of convergence but for the conjugate gradient method (CG) and special right-hand sides and λ = and µ = ν = 1is[], where that N/k remains constant is required as k, which is a much stronger restriction than N > k needed in (1.3) and (1.4). A by-product of (1.3) is that the worst asymptotic speed can be separated into the factor ζ 1 1 contributed by A s departure from normality and the factor ρ 1 contributed by A s eigenvalue distribution. Take, for example, λ = 0.5, µ = 0.3, and ν = 0.7 which was used in [4, p. 56] to explain the effect of non-normality on GMRES convergence. We have (ζρ) 1 = , whereas in [4, p. 56] it is implied r k / r 0 (0.913) k for N = 50, which is rather good, considering that N = 50 is rather small. Tridiagonal Toeplitz linear systems naturally arise from a convection-diffusion model problem [19]. When µ or ν is zero, A becomes a Jordan block, and GMRES residual bounds by Ipsen [1] and Liesen and Strakoš [18] suggests possibly slow convergence unless λ is much bigger in magnitude relative to the nonzero number µ or ν. This is more or less expected because otherwise A is highly non-normal, a case GMRES usually does poorly. Liesen and Strakoš also presented a detailed analysis aiming at showing that GMRES for tiny µ behaves much like GMRES after setting µ to 0. Their general residual expression and bound, however, involve the pseudo-inverse and the smallest singular value, respectively, of a matrix that do not seem to lead to simple bounds like ours without further, likely complicated, analysis. Ernst [9], in our notation, obtained the following inequality: if A s field of values does not contain the origin, then r k ( ξ k + ξ k) ρ k, (1.5) r 0 1 ρ k { where ρ = max τ + τ 1, τ τ } [ 1 π 1 and τ = cos N+1] τ. Our bound (1.) is comparable to Ernst s bound for large N. This can be seen by noting that for N large enough, τ τ and ρ ρ, and that T j (τ) 1 ρ j when ρ>1and 13

4 70 R.-C. Li, W. Zhang ζ k ξ k + ξ k ζ k. Ernst s bound also leads to lim sup sup [ ] r k 1/k { } sup min (ζρ) 1, 1. (1.6) r 0 r 0 In differentiating our contributions here from Ernst s, we use a different technique to arrive at (1.) and (1.3). While our approach is not as elegant as Ernst s which was based on A s field of values (see also [5]), it allows us to establish both lower and upper bounds on relative residuals for special right-hand sides to conclude that our bound is nearly achieved. Also (1.3) is an equality while only an inequality (1.6) can be deduced from Ernst s bound and approach. We also obtain residual bounds and exact expressions especially for right-hand sides b = e 1 and b = e N (Theorem.3). They suggest, besides the sharpness of (1.), an interesting GMRES convergence behavior. For b = e 1, that ξ > 1 speeds up GMRES convergence, and in fact r k is roughly proportional to ξ k. So the bigger the ξ is, the faster the convergence will be. Note as ξ gets bigger, A gets further away from a normal matrix. Thus, loosely speaking, the non-normality contributes to the convergence rate in the positive way. Nonetheless this does not contradict our usual perception that high non-normality is bad for GMRES if the worst behavior of GMRES among all b is considered. This mystery can be best explained by looking at the extreme case: ξ =, i.e., ν = 0, for which b = e 1 is an eigenvector (and convergence occurs in just one step). In general for ν = 0, as ξ gets bigger and bigger, roughly speaking b = e 1 comes closer and closer to A s invariant subspaces of lower dimensions and consequently speedier convergence is witnessed. Similar comments apply to the case when b = e N. The rest of this paper is organized as follows. We state our main results in Sect.. Tedious proofs that rely on residual reformulation involving rectangular Vandermonde matrices and complicated analysis will be presented separately in Sect. 3. Exact residual norm formulas for two special right-hand sides b = e 1 and e N are established in Sect. 4. Finally in Sect. 5 we present our concluding remarks. Notation Throughout this paper, K n m is the set of all n m matrices with entries in K, where K is C (the set of complex numbers) or R (the set of real numbers), K n = K n 1, and K = K 1. I n (or simply I if its dimension is clear from the context) is the n n identity matrix, and e j is its jth column. The superscript takes conjugate transpose while T takes transpose only. σ min (X) denotes the smallest singular value of X. We shall also adopt MATLAB-like convention to access the entries of vectors and matrices. The set of integers from i to j inclusive is i : j. For a vector u and a matrix X, u ( j) is u s jth entry, X (i, j) is X s (i, j)th entry, diag(u) is the diagonal matrix with (diag(u)) ( j, j) = u ( j) ; X s submatrices X (k:l,i: j), X (k:l,:), and X (:,i: j) consists of intersections of row k to row l and column i to column j,rowk to row l and all columns, and all rows and column i to column j, respectively. Finally p (1 p )is the l p norm of a vector or the l p operator norm of a matrix, defined as 13

5 Convergence of GMRES on tridiagonal Toeplitz linear system 71 u p = j u ( j) p 1/p, X p = max u p =1 Xu p. α be the largest integer that is no bigger than α, and α the smallest integer that is no less than α. Main results.1 Setting the stage An N N tridiagonal Toeplitz A takes this form λ µ A = ν µ CN N. (.1) ν λ Throughout the rest of this paper, ν, λ, and µ are reserved as the defining parameters of A, and set µν ξ =, τ = λ { ν µν, ζ = min ξ, 1 }, (.) ξ { τ ρ = max + τ } 1, τ τ 1. (.3) Matrix A is diagonalizable when µ = 0 and ν = 0. In fact [, pp ] (see also [9,18]), A = X X 1, X = SZ, = diag(λ 1,...,λ N ), (.4) λ j = λ µν t j, t j = cos θ j, θ j = jπ N + 1, (.5) Z (:, j) = N + 1 (sin jθ 1,...,sin jθ N ) T, (.6) S = diag(1,ξ 1,...,ξ N+1 ). (.7) It can be verified that Z T Z = I N ;SoA is normal if ξ =1, i.e., µ = ν > 0. Set By (.5), we have ω = µν. λ j = ω(t j τ), 1 j N. (.8) 13

6 7 R.-C. Li, W. Zhang Any branch of µν, once picked and fixed, is a valid choice in this paper. Note ρ 1 always because (τ + τ 1)(τ τ 1) = 1. In particular if λ R, µ<0 and ν>0, then ρ = τ + τ + 1. A common starting point in existing quantitative analysis for GMRES [10, Page 54] on Ax = b with diagonalizable A = X X 1 and = diag(λ 1,...,λ N ) is r k / r 0 κ(x) min max φ k (0)=1 i φ k (λ i ). (.9) as it seems that there is no easy way to do otherwise. It simplifies the analysis by separating the study of GMRES convergence behavior into optimizing the condition number of the eigenvector matrix X and a polynomial minimization problem over A s spectrum, but it could potentially overestimate GMRES residuals. This is partly because, as observed by Liesen and Strakoš [18], possible cancelations of huge components in X and/or X 1 were artificially ignored for the sake of the convergence analysis. For tridiagonal Toeplitz matrix A we are interested in here, however, rich structure allows us to do differently, as we shall do later. Let V,N be the (k + 1) N rectangular Vandermonde matrix V,N def = λ 1 λ λ N λ k 1 λ k λ k N (.10) having nodes {λ j } N j=1, and Y = Xdiag(X 1 b). (.11) Using (b, Ab,...,A k b) = YV,N T [5, Lemma.1], we have for GMRES r k = min u (1) =1 (b, Ab,...,Ak b)u = min u (1) =1 YVT,N u. (.1) Recall Chebyshev polynomials of the first kind: T m (t) = cos(m arccos t) for real t and t 1, (.13) = 1 ( ) m t + t 1 ( m 1 + t t 1), (.14) and define the mth Translated Chebyshev Polynomial in z of degree m as 13 T m (z; ω,τ) def = T m (z/ω + τ) (.15) = a mm z m + a m 1 m z m 1 + +a 1m z + a 0m, (.16)

7 Convergence of GMRES on tridiagonal Toeplitz linear system 73 where a jm a jm (ω, τ) are functions of ω and τ, and upper triangular R m C m m, a matrix-valued function in ω and τ, too, as R m R m (ω, τ) def = a 00 a 01 a 0 a 0 m 1 a 11 a 1 a 1 m 1 a a m 1, (.17).....ȧm 1 m 1 i.e., the jth column consists of the coefficients of T j 1 (z; ω,τ). Set T N def = T 0 (t 1 ) T 0 (t ) T 0 (t N ) T 1 (t 1 ) T 1 (t ) T 1 (t N )... T N 1 (t 1 ) T N 1 (t ) T N 1 (t N ) (.18) and V N = V N,N for short. Then V T N R N = T T N. (.19) Equation (.19) yields VN T = T T N R 1 N. Extracting the first k + 1 columns from both sides of VN T = T T N R 1 N yields V T,N = T T,N R 1, (.0) where T,N = (T N ) (1:,:). Now notice Y = X diag(x 1 b) and X = SZ with Z in (.6) being real and orthogonal to get YV,N T S 1 b)(t T N ) (:,1:) R 1 = SM (:,1:) R 1 (.1) = SM (:,1:) S 1 R 1 (.) where S = S (1:,1:),the(k + 1)th leading principle submatrix of S, It follows from (.1) and (.) that σ min (SM (:,1:) S 1 ) r k min u(1) =1 S R 1 u M = Z diag(z T S 1 b) T T N. (.3) SM (:,1:) S 1. (.4) The second inequality in (.4) is our foundation to bound GMRES residuals for a general right-hand side, through explicitly expressing min u(1) =1 S R 1 u in terms of 13

8 74 R.-C. Li, W. Zhang Chebyshev polynomials and the parameters τ and ξ and bounding SM (:,1:) S 1 in terms of b.. An equivalence principle We shall now describe an equivalence principle between the linear system with A and b and the one with A T and a permuted b. The principle will simplify our analysis by allowing us to focus on the case ξ 1 only. Let = (e N,...,e, e 1 ) R N N be the permutation matrix. Notice T A = A T and thus Ax = b is equivalent to A T T x = ( T A )( T x) = T b. (.5) Note K k (A T, T b) = K k ( T A, T b) = T K k (A, b), and r k = min y K k (A,b) b Ay = min T (b A T y) T y T K k (A,b) = min T b A T w. (.6) w K k (A T, T b) The last expression is the GMRES residual for the equivalent system (.5). Therefore Any result on GMRES for Ax = b leads to one for A T y = T b after performing the following substitutions µ ν, ν µ, ξ ξ 1, b T b. (.7).3 General right-hand sides Our first main result is given in Theorem.1 whose proof, along with the proofs of other results in the section, involve complicated computations and will be postponed to Sect. 3. Theorem.1 For Ax = b, where A is tridiagonal Toeplitz as in (.1) with nonzero (real or complex) parameters ν, λ, and µ. Then the kth GMRES residual r k satisfies for 1 k < N 13 r k [ ] 1 1/ k + 1 r 0 + (τ, ζ ), (.8)

9 Convergence of GMRES on tridiagonal Toeplitz linear system r k / r 0, upper bounds ( τ =0.8, ξ =0.7) 10 0 r k / r 0, upper bounds ( τ =1, ξ =0.7) r / r for µ=0.4375, ν= k 0 r / r for µ= , ν= k r / r for µ=0.35, ν= k 0 r / r for µ= 0.35, ν= k r k / r 0, upper bounds ( τ =1., ξ =0.7) 10 0 r k / r 0, upper bounds ( τ =1.4, ξ =0.7) r k / r 0 for µ=0.9167, ν= r k / r 0 for µ=0.5, ν=0.510 r k / r 0 for µ= , ν= r k / r 0 for µ= 0.5, ν= Fig. 1 GMRES residuals for random b uniformly in [ 1, 1], and their upper bounds (dashed lines) by (.8). All indicate that our upper bounds are tight, except for the last few steps. Upper bounds for the case µ>0 in the top two plots are visually indistinguishable from the horizonal line 10 0, suggesting slow convergence. Only ξ < 1 is shown because of (.7) where (t, s) def = k s j Tj (t), (.9) j=0 and j means the first term is halved. Figure 1 plots residual histories for several examples of GMRES with each of b s entries being uniformly random in [ 1, 1]. In each of the plots in Fig. 1,aswellasin 13

10 76 R.-C. Li, W. Zhang Figs. and 3, wefix τ and ξ,takeλ = 1 always, and then take µ = ξ 1, µ =± µ, and ν = ν = τ τξ. Thus µ, ν R, and in fact ν > 0 always. When µ>0, ξ = µ/ν < 0 and τ = 1/( µν) > 0, but when µ<0, both ξ = ι µ/ν and τ = ι/( µν ) are imaginary, where ι = 1 is the imaginary unit. Figure 1 indicates that GMRES converges much faster for µ<0 than for µ>0 in each of the plots. There is a simple explanation for this: the eigenvalues of A are further away from the origin for a pure imaginary τ than for a real τ for any fixed τ. Our next main result given in Theorem. tells the worst asymptotic speed for r k. Theorem. Under the conditions of Theorem.1, [ ] lim inf r k 1/k [ ] sup = lim r 0 r 0 sup r k 1/k sup = min{(ζρ) 1, 1}. (.30) r 0 r 0 Moreover, sup r0 among all possible r 0 can be replaced by sup r0 {e 1,e N }..4 Special right-hand sides We now consider three special right-hand sides: b = e 1 or e N or b (1) e 1 + b (N) e N.In particular they show that the upper bound in Theorem.1 is within a factor about at most (k + 1) 3/ of the true residual for b = e 1 or e N, depending on whether ξ 1 or ξ 1. Because of the equivalence principle (.7), only b = e 1 will be considered. Inequalities in Theorems.3 and.5 upon replacing ξ by ξ 1 lead to related results for the case b = e N. Theorem.3 In Theorem.1,ifb= e 1, then the kth GMRES residual r k satisfies for 1 k < N 1 1 j=0 ξ j 1 [ (τ, ξ) 1 4] 1/ r k 1 [ (1 + ξ ) (τ, ξ) 4] 1 1/. (.31) In particular, for ξ 1 [ 1 (τ, ξ) 4] 1 1/ [ r k (τ, ξ) 4] 1 1/. (.3) 13

11 Convergence of GMRES on tridiagonal Toeplitz linear system r k / r 0, lower and upper bounds ( τ =0.8, ξ =0.7) 0 10 r k / r 0, lower and upper bounds ( τ =0.8, ξ =1.) r / r for µ=0.4375, ν= k 0 r / r for µ= , ν= k r / r for µ=0.75, ν= k 0 r / r for µ= 0.75, ν= k r k / r 0, lower and upper bounds ( τ =1, ξ =0.7) 10 0 r k / r 0, lower and upper bounds ( τ =1, ξ =1.) r k / r 0 for µ=0.35, ν= r k / r 0 for µ=0.6, ν= r k / r 0 for µ= 0.35, ν= r / r, lower and upper bounds ( τ =1., ξ =0.7) k 0 r k / r 0 for µ= 0.6, ν= r / r, lower and upper bounds ( τ =1., ξ =1.) k r / r for µ=0.9167, ν= k 0 r / r for µ= , ν= k r / r for µ=0.5, ν=0.347 k 0 r / r for µ= 0.5, ν=0.347 k Fig. GMRES residuals for b = e 1, sandwiched by their lower and upper bounds by (.31). All lower and upper bounds are very good for ξ 1 as expected, but only upper bounds are good when ξ > 1 13

12 78 R.-C. Li, W. Zhang 10 0 r k / r 0, lower and upper bounds ( τ =0.8, ξ =0.7) 10 0 r k / r 0, lower and upper bounds ( τ =1, ξ =0.7) r / r for µ=0.4375, ν= k 0 r / r for µ= , ν= k r / r for µ=0.35, ν= k 0 r / r for µ= 0.35, ν= k r k / r 0, lower and upper bounds ( τ =1., ξ =0.7) 10 0 r k / r 0, lower and upper bounds ( τ =1.4, ξ =0.7) r / r for µ=0.9167, ν= k 0 r / r for µ= , ν= k r / r for µ=0.5, ν=0.510 k 0 r / r for µ= 0.5, ν=0.510 k Fig. 3 GMRES residuals for b = e 1 + e N, sandwiched by their lower and upper bounds by (.33) and (.34). Strictly speaking, (.33) is only proved for k N/, but it seems to be very good even for k > N/ as well. We also ran GMRES for b = e 1 e N and obtained residual history that is very much the same. Only ξ < 1 is shown because of (.7) The upper bound and the lower bound in (.3) differ by a factor roughly (k + 1), and thus they are rather sharp; so are the bounds in (.31) for ξ 1. Comparing them to (.8), we conclude that the upper bound by (.8) is fairly sharp for worst possible b. But the bounds in (.31) differ by a factor O( ξ ) for ξ > 1, and thus at least one of them (upper or lower bound) is bad. Our numerical examples indicate that the upper bounds are rather good regardless of the magnitude of ξ for b = e 1 (see Fig. ). Given Theorem.3 and its implied result for b = e N by (.7), it would not be unreasonable to expect that the upper bound in Theorem.1 would be sharp for very large or tiny ξ within a factor possibly about at most (k + 1) 3/ for right-hand side b with b (i) = 0for i N 1 and b (1) = b (N) > 0. The following theorem indeed confirms this but only for k N/. Our numerical examples even support that 13

13 Convergence of GMRES on tridiagonal Toeplitz linear system 79 the lower bounds by (.33) would be good for k > N/ (see Fig. 3), too, but we do not have a way to mathematically justify it yet. Theorem.4 In Theorem.1, ifb (i) = 0 for i N 1, then the kth GMRES residual r k satisfies for 1 k N/ 1 and for 1 k < N where r k r 0 min i {1,N} b (i) χ r 0 [ (τ, ζ ) 1 4] 1/, (.33) r k [ ] 1 1/ 3 r 0 + (τ, ζ ), (.34) 1 <χ= 1 j=0 k + 1 ζ j Figures and 3 plot residual histories for several examples of GMRES with b = e 1 and b = e 1 + e N, respectively. Finally we have the following theorem about the asymptotic speeds of r k for b = e 1. Theorem.5 Assume the conditions of Theorem.1 hold. Let b = e 1. Then and min{( ξ ρ) 1,( ξ ρ) 1, 1} lim inf lim sup. inf r k 1/k sup r k 1/k min{( ξ ρ) 1, 1}, (.35) lim inf r k 1/k = lim sup r k 1/k = min{( ξ ρ) 1, 1} for ξ 1. (.36) Remark 1 As we commented before, our numerical examples indicate that the upper bounds in Theorem.3 is rather accurate regardless of the magnitude of ξ for b = e 1 (see Fig. ). This leads us to conjecture that the following equations lim inf r k 1/k = lim sup r k 1/k = min{( ξ ρ) 1, 1} for b = e 1, (.37) lim inf r k 1/k = lim sup r k 1/k = min{( ξ 1 ρ) 1, 1} for b = e N, (.38) would hold. As before if one of them is proven, the other one follows because of (.7). 13

14 80 R.-C. Li, W. Zhang 3 Proofs In what follows, we will prove theorems in the previous section in the order of their appearance, except Theorem. whose proof requires Theorem.5 will be proved last. In (.4), there are two quantities to deal with min S R 1 u (1) =1 u and SM (:,1:) S 1. (3.1) We shall now do so. In its present general form, the next lemma was proven in [13,16]. It was also implied by the proof of [1, Theorem.1]. See also [17]. Lemma 3.1 If W has full column rank, then [ ] 1/ min Wu = e1 T (W W ) 1 e 1. (3.) u (1) =1 In particular if W is nonsingular, min u(1) =1 Wu = W e 1 1. By this lemma, we have (note a 00 = 1) [ ] 1 1/ min S R 1 u (1) =1 u = S R e 1 1 = + (τ, ξ). (3.3) This gives the first quantity in (3.1). We now turn to the second one there. It can be seen that SM (:,1:) S 1 = (SMS 1 ) (:,1:) since S is diagonal. To compute SMS 1, we shall investigate M in (.3) first. M = = = N Z diag(zs 1 b (l) e l ) T T N l=1 N b (l) ξ l 1 Z diag(ze l ) T T N l=1 N b (l) ξ l 1 Z diag(z (:,l) ) T T N. (3.4) l=1 In Lemma 3. and in the proof of Lemma 3.3, without causing notational conflict, we will temporarily use k as a running index, as opposed to the rest of the paper where k is reserved for GMRES step index. The following lemma is probably well-known, but an exact source is hard to find. For a proof, see [15, Lemma 4.1]. Lemma 3. For θ j = j N+1π and integer l, N N, if l = m(n + 1) for some integer m, cos lθ k = 0, if l is odd, (3.5) k=1 1, if l is even, but l = m(n + 1) for any integer m. 13

15 Convergence of GMRES on tridiagonal Toeplitz linear system 81 Lemma 3.3 Let M l def = Z diag(z (:,l) )T T N for 1 l N. Then the entries of M l are zeros, except at those positions (i, j), graphically forming four straight lines: (a) i + j = l + 1, (b) i j = l 1, (c) j i = l + 1, (d) i + j = (N + 1) l + 1. (3.6) (M l ) (i, j) = 1/ for (a) and (b), except at their intersection (l, 1) for which (M l ) (l,1) = 1. (M l ) (i, j) = 1/ for (c) and (d). Notice no valid entries for (c) if l N 1 and no valid entries for (d) if l. Proof For 1 i, j N, (N + 1) (M l ) (i, j) = 4 Since all = 4 = = N sin kθ i sin lθ k cos( j 1)θ k k=1 N sin iθ k sin lθ k cos( j 1)θ k k=1 N [cos(i l)θ k cos(i + l)θ k ] cos( j 1)θ k k=1 N [cos(i + j l 1)θ k + cos(i j l + 1)θ k k=1 cos(i + j + l 1)θ k cos(i j + l + 1)θ k ]. i 1 = i + j l 1, i = i j l + 1, i 3 = i + j + l 1, i 4 = i j + l + 1 are either even or odd at the same time, Lemma 3. implies (M l ) (i, j) = 0 unless one of them takes the form m(n +1) for some integer m. We now investigate all possible situations as such, keeping in mind that 1 i, j, l N. 1. i 1 = i + j l 1 = m(n + 1). This happens if and only if m = 0, and thus i + j = l + 1. Then i = j +, i 3 = l, i 4 = j + l +. They are all even. i 3 and i 4 do not take the form m(n + 1) for some integers m. This is obvious for i 3, while i 4 = m(n + 1) implies m = 0 and j = l + 1, and thus i = 0 which cannot happen. However if i = m(n + 1), then m = 0 and j = 1, and thus i = l. 13

16 8 R.-C. Li, W. Zhang So Lemma 3. implies (M l ) (i, j) = 1/ fori + j = l + 1 and i = l, while (M l ) (l,1) = 1.. i = i j l + 1 = m(n + 1). This happens if and only if m = 0, and thus i j = l 1. Then i 1 = j, i 3 = j + l, i 4 = l. They are all even. i 3 and i 4 do not take the form m(n + 1) for some integers m. This is obvious for i 4, while i 3 = m(n + 1) implies m = 1 and j = N + l, and thus i = N + 1 which cannot happen. However if i 1 = m(n + 1), then m = 0 and thus j = 1 and i = l which has already been considered in Item 1. So Lemma 3. implies (M l ) (i, j) = 1/ fori j = l 1 and i = l, while (M l ) (l,1) = i 3 = i + j + l 1 = m(n + 1). This happens if and only if m = 1, and thus i + j = (N + 1) l + 1. Then i 1 = (N + 1) l, i =(N + 1) j l +, i 4 =(N + 1) j +. They are all even. i 1 and i do not take the form m(n + 1) for some integers m. This is obvious for i 1, while i = m(n + 1) implies m = 0 and j = N + l, and thus i = N + 1 which cannot happen. However if i 4 = m(n + 1), then m = 1 and thus j = 1 and i = (N + 1) l which is bigger than N + and not possible. So Lemma 3. implies (M l ) (i, j) = 1/ fori + j = (N + 1) l i 4 = i j + l + 1 = m(n + 1). This happens if and only if m = 0, and thus j i = l + 1. Then i 1 = j l, i = l, i 3 = j. They are all even, and do not take the form m(n + 1) for some integers m.this is obvious for i. i 1 = m(n + 1) implies m = 0 and j = l + 1, and thus i = 0 which cannot happen. i 3 = m(n + 1) implies m = 0 and thus j = 1 and i = l which cannot happen either. So Lemma 3. implies (M l ) (i, j) = 1/ for j i = l + 1. This completes the proof. Now we know M l. We still need to find out SMS 1. Let us examine it for N = 5 in order to get some idea about what it may look like. SMS 1 for N = 5is b 1 (1) ξ b () 1 ξ b (1) + 1 ξ 4 b (3) 1 ξ 4 b () + 1 ξ 6 b (4) 1 ξ 6 b (3) + 1 ξ 8 b (5) b 1 () b (1) + 1 b (3) ξ 1 b (4) ξ 4 1 ξ b (1) + 1 ξ 6 b (5) 1 ξ 4 b () b 1 (3) b () + 1 ξ b 1 (4) b (1) + 1 b (5) ξ ξ b (1) 1 ξ 6 b (5). b 1 (4) b (3) + 1 b (5) ξ 1 b 1 () b (1) 1 b (5) ξ 4 1 b (4) ξ 4 b 1 (5) b 1 (4) b (3) 1 b (5) ξ 1 b () 1 ξ b 1 (4) b (1) 1 b (3) ξ 13

17 Convergence of GMRES on tridiagonal Toeplitz linear system 83 We observe that for N = 5, the entries of SMS 1 are polynomials in ξ with at most two terms. This turns out to be true for all N. Lemma 3.4 The following statements hold. 1. The first column of SM S 1 is b. Entries in every other columns taking one of the three forms: (b (n1 )ξ m 1 + b (n )ξ m )/ with n 1 = n,b (n1 )ξ m 1/, and 0, where 1 n 1, n N and m i 0 are non-negative integer.. In each given column of SM S 1, any particular entry of b appears at most twice. As the consequence, we have SM (:,1:) S 1 k + 1 b if ξ 1. Proof Notice M = N l=1 b (l) ξ l 1 M l and consider M s (i, j)th entry which comes from the contributions from all M l. But not all of M l contribute as most of them are zero at the position. Precisely, with the help of Lemma 3.3, those M l that contribute nontrivially to the (i, j)th position are the following ones subject to satisfying the given inequalities. (a) if 1 i + j 1 N or equivalently i + j N + 1, M i+ j 1 gives a 1/. (b) if 1 i j + 1 N or equivalently i j, M i j+1 gives a 1/. (c) (d) if 1 j i 1 N or equivalently j i +, M j i 1 gives a 1/. if 1 (N +1) (i + j)+1 N or equivalently i + j N +3, M (N+1) (i+ j)+1 gives a 1/. These inequalities, effectively 4 of them, divided entries of M into nine possible regions as detailed in Fig. 4. We shall examine each region one by one. Recall and let (SMS 1 ) (i, j) = ξ i+1 M (i, j) ξ j 1 = ξ j i M (i, j), γ a = 1 b (i+ j 1)ξ j, γ b = 1 b (i j+1), γ c = 1 b ( j i 1)ξ ( j i 1), γ d = 1 b ((N+1) (i+ j)+1)ξ (N+1) (i+ j). Each entry in the nine possible regions in the rightmost plot of Fig. 4 is as follows. 1. (a) and (b): (SMS 1 ) (i, j) = γ a + γ b.. (a) and (c): (SMS 1 ) (i, j) = γ a + γ c. 3. (b) and (d): (SMS 1 ) (i, j) = γ b + γ d. 4. (c) and (d): (SMS 1 ) (i, j) = γ c + γ d. 5. (a) and i j = 1: (SMS 1 ) (i, j) = γ a. 6. (b) and i + j = N + : (SMS 1 ) (i, j) = γ b. 7. (c) and i + j = N + : (SMS 1 ) (i, j) = γ c. 13

18 84 R.-C. Li, W. Zhang Fig. 4 Computation of M (i, j). Left: Regions of entries as divided by inequalities in (a) and(d); Middle: Regions of entries as divided by inequalities in (b) and(c); Right: Regions of entries as divided by all inequalities in (a), (b), (c), and (d) 8. (d) and i j = 1: (SMS 1 ) (i, j) = γ d. 9. i j = 1 and i + j = N + : (SMS 1 ) (i, j) = 0. In this case, i = (N + 1)/ and j = (N + 3)/. So there is only one such entry when N is odd, and none when N is even. With this profile on the entries of SMS 1, we have Item 1 of the lemma immediately. Item is the consequence of M = N l=1 b (l) ξ l 1 M l and Lemma 3.3 which implies that there are at most two nonzero entries in each column of M l. As the consequence of Item 1 and Item, each column of SMS 1 can be expressed as the sum of two vectors w and v such that w, v b / when ξ 1, and thus (SMS 1 ) (:, j) b for all 1 j N. Therefore SM (:,1:) S 1 (SMS 1 ) (:, j) k + 1 b, j=1 as expected. Proof of Theorem.1 We shall only prove r k b k + 1 [ 1 + (τ, ξ)] 1/ for ξ 1 (3.7) since for the other case when ξ > 1, we have, by (.7) and (3.7) assuming proven true, r k T b k + 1 [ 1 + (τ, ξ 1 )] 1/. The conclusion of the theorem now follows from the observation that ζ = min{ ξ, ξ 1 }. Assume ξ 1. Inequality (3.7) is the consequence of (.4), (3.3), and Lemma

19 Convergence of GMRES on tridiagonal Toeplitz linear system 85 Remark The leftmost inequality in (.4) gives a lower bound on r k in terms of σ min (SM (:,1:) S 1 ) which, however, is hard to bound from below because it can be as small as zero, unless we know more about b such as b = e 1 as in Theorem.3. Proof of Theorem.3 If b = e 1, then M = M 1 is upper triangular. More specifically and, by (.1), 1 0 1/. 1/ 0.. M = M 1 = 1/ 1/ / ( YV,N T = S MR 1 ) ( S MS = 1 D 1 DS R 1 ), 0 0 where M = M (1:,1:) and D = diag(, 1,1,...,1). Therefore,N u σ min (S MS 1 D 1 ) min u (1) =1 Y V T min u(1) =1 DS R 1 Recall r k = min u(1) =1 Y V T,N u.let It can be seen that u (3.8) S MS 1 D 1. (3.9) P = (e 1, e 3,...,e, e 4,...) R () (). P T (S 1 MS D 1 )P = 1 ( E1, E) where E 1 C, E C 1 ξ E i = ξ, 1, and E 1 i = Hence E i E i 1 E i = 1 + ξ. Therefore 1 ξ ξ (m 1) ξ. 1 S MS 1 D 1 = 1 max{ E 1, E } 1 (1 + ξ ). 13

20 86 R.-C. Li, W. Zhang Similarly use Ei 1 E 1 i 1 E 1 to get i E j=0 ξ j, E 1 1 j=0 ξ j. Therefore σ min (S MS 1 D 1 ) = 1 min{σ min(e 1 ), σ min (E )} = 1 min{ E 1 1 1, E ξ j. j=0 } Finally, by Lemma 3.1, wehave [ min DS R 1 u (1) =1 u = D S R e 1 1 = (τ, ξ) 4] 1 1/. This, together with (3.9), lead to (.31). Proof of Theorem.4 Now b = b (1) e 1 + b (N) e N. Notice the form of M 1 in (3.8), and that M N is M 1 after its rows reordered from the last to the first. For the case M = b (1) M 1 + ξ N 1 b (N) M N, and also Lemma 3.4 implies that only positive powers of ξ appear in the entries of SMS 1. Therefore when ξ 1, SM (:,1:) S 1 SMS 1 b (1) M 1 + b (N) M N b (1) 3/ + b (N) 3/ 3 b, (3.10) where M l takes entrywise absolute value, and we have used M N = M 1 M 1 1 M 1 = 3/. Inequality (.34) for ξ 1 is the consequence of (.4), (3.3), and (3.10). Inequality (.34) for ξ 1 follows from itself for ξ 1 applied to the permuted system (.5). 13

21 Convergence of GMRES on tridiagonal Toeplitz linear system 87 To prove (.33), we use the lines of arguments in the proof for Theorem.3 and notice that for 1 k N/ 1 YV T,N = W 1 N () 0 W It can be seen from the proof for Theorem.3 that min W 1u b (1) u (1) =1 min W u b (N) u (1) =1 Finally use 1 j=0 1 j=0 ξ j ξ j 1 [ 1 [ (τ, ξ) 1 4] 1/, (τ, ξ 1 ) 1 4] 1/. { } min u (1) =1 YVT,N u max min W 1u, min W u u (1) =1 u (1) =1 to complete the proof. Proof of Theorem.5 We note that lim sup sup r k 1/k 1 because r k is nonincreasing. Suppose b = e 1. Consider first ρ>1. Then T j (τ) 1 ρ j, and thus (τ, ξ) k ( ξ ρ) j = 1 4 ( ξ ρ)() 1 ( ξ ρ). (3.11) 1 j=0 If ξ ρ >1, then (3.11) and Theorem.3 imply lim sup sup r k 1/k lim [ ] 1 1/k (1 + ξ ) [ (τ, ξ) 1 4 ] 1/(k) = ( ξ ρ) 1, (3.1) 13

22 88 R.-C. Li, W. Zhang lim inf inf r k 1/k lim = 1 1/k 1 j=0 ξ j ] 1/(k) [ (τ, ξ) 1 4 ( ξ ρ) 1, for ξ 1, ( ξ ρ) 1, for ξ > 1. 1/k (3.13) They together give (.35) for the case ξ ρ >1. If ξ ρ 1, then must ξ < 1 and min{( ξ ρ) 1, 1} =1, min{( ξ ρ) 1,( ξ ρ) 1, 1} =1, and lim inf inf r k 1/k 1 by (3.13) because (τ, ξ) 4 1 is approximately bounded by (k + 1)/4by(3.11). So (.35) holds for the case ξ ρ 1, too. Now consider ρ = 1. Then τ + τ 1 = e ιθ for some 0 θ π, where ι = 1 is the imaginary unit. Thus τ [ 1, 1] and in fact τ = (τ + τ 1) + (τ τ 1) = cos θ, T j (τ) = cos jθ. Therefore (τ, ξ) k j=1 ξ j (cos jθ). Instead of (3.1) and (3.13), we have where lim sup sup r k 1/k η, lim inf inf r k 1/k ξ 1 η 1, (3.14) η 1 = lim inf η = lim sup 1 k 4 + ξ j (cos jθ) j=1 1 k 4 + ξ j (cos jθ) j=1 1/(k) 1/(k),. Now if ξ 1, then k j=1 ξ j (cos jθ) k and thus 13 η 1 = η = 1 if ξ 1. (3.15)

23 Convergence of GMRES on tridiagonal Toeplitz linear system 89 If, however, ξ > 1, we claim that α ξ (k 1) 1 k 4 + ξ j (cos jθ) (k + 1) ξ k (3.16) j=1 for some constant α>0, independent of θ and k. The second inequality in (3.16) is quite obvious. To see the first inequality, we notice k j=k 1 ξ j (cos jθ) ξ (k 1) [cos kθ + cos (k 1)θ] = ξ (k 1) [ 1 + cos kθ + = ξ (k 1) [1 + cos(k 1)θ cos θ]. ] 1 + cos (k 1)θ It suffices to show that there is positive constant α, independent of k and θ, such that 1 + cos(k 1)θ cos θ α. Assume to the contrary that inf [1 + cos(k 1)θ cos θ] = 0, (3.17) k,θ which means there are sequences {k i } and θ i such that 1 + cos(k i 1)θ i cos θ i 0 as i. Since cos( ) 1, this implies as i either cos(k i 1)θ i 1 and cos θ i 1 or cos(k i 1)θ i 1 and cos θ i 1. But none of which is possible. Therefore (3.17) cannot hold. This proves (3.16) which yields η 1 = η = ξ 1 if ξ > 1. (3.18) (.35) forρ = 1 is the consequence of (3.14), (3.15), and (3.18). Finally if ξ 1, then both the leftmost side and rightmost side of (.35) are equal to min{( ξ ρ) 1, 1}. This proves (.36). Proof of Theorem. Note that lim sup sup [ ] r k 1/k sup 1. r 0 r 0 Because of the equivalence principle (.7), it suffices to consider only ξ 1. Then ζ = ξ. First we prove lim sup sup [ ] r k 1/k sup min{( ξ ρ) 1, 1}. (3.19) r 0 r 0 13

24 90 R.-C. Li, W. Zhang If ρ = 1, then min{( ξ ρ) 1, 1} =1 because ξ 1 1; no proof is needed. If ρ>1, then T j (τ) 1 ρ j, and thus (3.11). Now if ξ ρ >1, then (3.11) and Theorem.1 imply lim sup sup [ ] r k 1/k sup ( ξ ρ) 1 r 0 r 0 which also holds if ξ ρ 1 because then ( ξ ρ) 1 1. Next we notice lim inf [ 1/k r k r 0 lim inf r0 =e 1] lim sup [ inf r k sup r 0 r 0 sup [ sup r 0 r k r 0 ] 1/k ] 1/k, (3.0) where the first limit has been proven to exist and equal to min{( ξ ρ) 1, 1} in Theorem.5. The proof is now completed by combining (3.19), (3.0), and (.36). 4 Exact residual norms for b = e 1 and e N In this section we present two theorems in which exact formulas for r k for b = e 1 and b = e N are established. Let S and S have their assignments as in Sect. 3. Theorem 4.6 In Theorem.1,ifb= e 1, then the kth GMRES residual r k satisfies for 1 k < N where y C N/ is defined as r k = S y (1:) 1, (4.1) j j y ( j 1) = T i (τ), y ( j) = i=1 i=1 T i 1 (τ) for j = 1,,..., N/, and T j (τ) is the complex conjugate of T j (τ). ( Proof We still have (3.8), and YV,N T = S MR 1 ), where M = M 0 (1:,1:) as in the proof of Theorem.3.LetD = diag(, 1, 1,...,1). Noticing S MR 1 = S MD 1 DR 1 is nonsingular, we have by Lemma min u (1) =1 YVT,N u = w 1,

25 Convergence of GMRES on tridiagonal Toeplitz linear system 91 where w = S ( MD 1 ) T D T R e 1, or equivalently We shall now solve it for w.let It can be verified that ( MD 1 ) T S w = D T R e 1. P = (e 1, e 3,...,e, e 4,...) R () (). P T ( MD 1 )P = 1 ( G1, G) where G 1 R, G R, and 1 1 G i = , G 1 1 i = Solve (P T MD 1 P ) T P T S w = PT D T R e 1 P T z for w to get ( ) w = S G T P 1 G T P T z, where z = ( 1 T 0(τ), T 1 (τ), T (τ),...,t k (τ)). Finally notice w = S y (1:) to complete the proof. Remark 3 For b = e N = T e 1, apply Theorem 4.6 to the permuted system (.5) to get the kth GMRES residual r k satisfies for 1 k < N r k = S y (1:) 1, (4.) where y C N/ is the same as the one in Theorem Concluding remarks There are a few GMRES error bounds with simplicity comparable to the well-known bound for the conjugate gradient method [3,10,0,4]. In [6, Section 6], Eiermann and Ernst proved r k [ k/ 1 γ(a)γ(a )] 1, (5.1) r 0 13

26 9 R.-C. Li, W. Zhang where γ(a) = inf{ z Az : z = 1} is the distance from the origin to A s field of values. When A s Hermitian part, H = (A + A )/, is positive definite, it yields a bound by Elman [8](seealso[7]) [ ( ) ] r k 1 k/ 1 r 0 H 1. (5.) A As observed in [1], this bound of Elman can be easily extended to cover the case when only γ(a) >0 r k r 0 (sin θ) k, θ = arccos γ(a) A. (5.3) Recently Beckermann, Goreinov, and Tyrtyshnikov [1] improved (5.3)to r k r 0 ( + / 3)( + δ)δ k, δ = sin θ 4 θ/π. (5.4) All three bounds (5.1), (5.3), and (5.4) yield meaningful estimates only when γ(a) > 0, i.e., A s field of values does not contain the origin. However in general, there is not much concrete quantitative results for the convergence rate of GMRES, based on limited information on A and/or b. In part, it is a very difficult problem, and such a result most likely does not exist, thanks to the negative result of Greenbaum, Pták, and Strakoš [11] which says that Any Nonincreasing Convergence Curve is Possible for GMRES. A commonly used approach, as a step towards getting a feel of how fast GMRES may be, is through assuming that A is diagonalizable to arrive at (.9): r k / r 0 κ(x) min max φ k (0)=1 i φ k (λ i ), (5.5) and then putting aside the effect of κ(x) to study only the effect in the factor of the associated minimization problem. This approach does not always yield satisfactory results, especially when κ(x) 1 which occurs when A is highly non-normal. Getting a fairly accurate quantitative estimate for the convergence rate of GMRES for a highly non-normal case is likely to be very difficult. Trefethen [3] established residual bounds based on pseudospectra, which sometimes is more realistic than (5.5) but is very expensive to compute. In [4], Driscoll, Toh, and Trefethen provided an nice explanation on this matter. Our analysis here on tridiagonal Toeplitz A represents one of few diagonalizable cases where one can analyze r k directly to arrive at simple quantitative results such as (5.1) (5.4). Previous other results except those in Ernst [9], while helpful in explaining and understanding various convergence behaviors, are more qualitative than quantitative. Two conjectures are made in Remark 1. There is also an unsolved question whether (.33) or a slightly modified version of it holds for N/ < k < N (see Fig. 3). 13

27 Convergence of GMRES on tridiagonal Toeplitz linear system 93 Acknowledgments The authors were supported in part by the National Science Foundation under Grant No. DMS and DMS The authors wish to thank an anonymous referee for his constructive suggestions that improve the presentation considerably. References 1. Beckermann, B., Goreinov, S.A., Tyrtyshnikov, E.E.: Some remarks on the Elman estimate for GMRES. SIAM J. Matrix Anal. Appl. 7, (006). Beckermann, B., Kuijlaars, A.B.J.: Superlinear CG convergence for special right-hand sides. Electron. Trans. Numer. Anal. 14, 1 19 (00) 3. Demmel, J.: Applied Numerical Linear Algebra. SIAM, Philadelphia (1997) 4. Driscoll, T.A., Toh, K.-C., Trefethen, L.N.: From potential theory to matrix iterations in six steps. SIAM Rev. 40, (1998) 5. Eiermann, M.: Fields of values and iterative methods. Linear Algebra Appl. 180, (1993) 6. Eiermann, M., Ernst, O.G.: Geometric aspects in the theory of Krylov subspace methods. Acta Numer. 10, (001) 7. Eisenstat, S.C., Elman, H.C., Schultz, M.H.: Variational iterative methods for nonsymmetric systems of linear equations. SIAM J. Numer. Anal. 0, (1983) 8. Elman, H.C.: Iterative Methods for Large, Sparse Nonsymmetric Systems of Linear Equations. Ph.D. thesis, Department of Computer Science, Yale University (198) 9. Ernst, O.G.: Residual-minimizing Krylov subspace methods for stabilized discretizations of convection-diffusion equations. SIAM J. Matrix Anal. Appl. 1, (000) 10. Greenbaum, A.: Iterative Methods for Solving Linear Systems. SIAM, Philadelphia (1997) 11. Greenbaum, A., Pták, V., Strakoš, Z.: Any nonincreasing convergence curve is possible for GMRES. SIAM J. Matrix Anal. Appl. 17, (1996) 1. Ipsen, I.C.F.: Expressions and bounds for the GMRES residual. BIT 40, (000) 13. Li, R.-C.: Sharpness in rates of convergence for CG and symmetric Lanczos methods, Technical Report (Department of Mathematics, University of Kentucky, 005). MAreport/ 14. Li, R.-C.: Convergence of CG and GMRES on a tridiagonal Toeplitz linear system. BIT 47, (007) 15. Li, R.-C.: Hard cases for conjugate gradient method. Int. J. Info. Sys. Sci. 4, 15 9 (008) 16. Li, R.-C.: On Meinardus examples for the conjugate gradient method. Math. Comp. 77, (008) (Electronically published on September 17, 007) 17. Liesen, J., Rozlozník, M., Strakoš, Z.: Least squares residuals and minimal residual methods. SIAM J. Sci. Comput. 3, (00) 18. Liesen, J., Strakoš, Z.: Convergence of GMRES for tridiagonal Toeplitz matrices. SIAM J. Matrix Anal. Appl. 6, (004) 19. Liesen, J., Strakoš, Z.: GMRES convergence analysis for a convection-diffusion model problem. SIAM J. Sci. Comput. 6, (005) 0. Saad, Y.: Iterative Methods for Sparse Linear Systems, nd edn. SIAM, Philadelphia (003) 1. Saad, Y., Schultz, M.: GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput. 7, (1986). Smith, G.D.: Numerical Solution of Partial Differential Equations, nd edn. Clarendon Press, Oxford (1978) 3. Trefethen, L.N., Pseudospectra of matrices. In: Griffiths, D.F., Watson, G.A. (eds.) Numerical Analysis 1991: Proceedings of the 14th Dundee Conference, June, Research Notes in Mathematics Series. Longman Press, London (199) 4. Trefethen, L.N., Bau, D. III.: Numerical Linear Algebra. SIAM, Philadelphia (1997) 5. Zavorin, I., O Leary, D.P., Elman, H.: Complete stagnation of GMRES. Linear Algebra Appl. 367, (003) 13