ON MEINARDUS EXAMPLES FOR THE CONJUGATE GRADIENT METHOD

Transcription

1 MATHEMATICS OF COMPUTATION Volume 77 Number 61 January 008 Pages S (07)019-9 Article electronically published on September ON MEINARDUS EXAMPLES FOR THE CONJUGATE GRADIENT METHOD REN-CANG LI Abstract. The conjugate gradient (CG) method is widely used to solve a positive definite linear system Ax b of order N. It is well known that the relative residual of the kth approximate solution by CG (with the initial approximation x 0 0) is bounded above by [ ] 1 +1 k + k with 1 where (A) A A 1 is A s spectral condition number. In 1963 Meinardus (Numer. Math. 5 (1963) pp. 14 3) gave an example to achieve this bound for k N 1 but without saying anything about all other 1 k< N 1. This very example can be used to show that the bound is sharp for any given k by constructing examples to attain the bound but such examples depend on k and for them the (k + 1)th residual is exactly zero. Therefore it would be interesting to know if there is any example on which the CG relative residuals are comparable to the bound for all 1 k N 1. There are two contributions in this paper: (1) A closed formula for the CG residuals for all 1 k N 1 on Meinardus example is obtained and in particular it implies that the bound is always within a factor of of the actual residuals; () A complete characterization of extreme positive linear systems for which the kth CG residual achieves the bound is also presented. 1. Introduction The conjugate gradient (CG) method is widely used to solve a positive definite linear system Ax b (often with certain preconditioning). The basic idea is to seek approximate solutions from the so-called Krylov subspaces. While different implementation may render different numerical behavior mathematically 1 the kth approximate solution x k by CG is the optimal one in the sense that the kth approximation error A 1 b x k satisfies [11 Theorem 6:1] (1.1) A 1 b x k A min x K k A 1 b x A Received by the editor September Revised January Mathematics Subject Classification. Primary 65F10. Key words and phrases. Conjugate gradient method Krylov subspace rate of convergence Vandermonde matrix condition number. This work was supported in part by the National Science Foundation CAREER award under Grant No. CCR and by the National Science Foundation under Grant No. DMS Without loss of generality we assume that A is already preconditioned and the initial approximation x c 007 American Mathematical Society Reverts to public domain 8 years from publication

2 336 REN-CANG LI Residual bounds over actual residuals Equidist eigenvalues on [δ 1] δ10-1 [minλ j max λ j ][0.11] δ10 - [min λ j maxλ j ][0.011] N k Residual bounds over actual residuals Random eigenvalues on [δ1] δ 10-1 [minλ j maxλ j ][ ] δ 10 - [minλ j maxλ j ][ ] N k Figure 1.1. Conjugate gradient method for Ax b with A diag(λ 1 λ...λ n )andb the vector of all ones Ratios of (1.4) over the actual residuals for equidistant or random distributed λ j s or equivalently the kth residual r k b Ax k satisfies (1.) r k A 1 min b Ax A 1 x K k where K k K k (A b) isthekth Krylov subspace of A on b defined as (1.3) K k K k (A b) def span{bab...a k 1 b} def and M-vector norm z M z Mz. Here the superscript takes conjugate transpose. In practice x k is computed recursively from x k 1 via short term recurrences [ ]. But exactly how it is computed though extremely crucial in practice is not important to our analysis here in this paper. CG always converges for positive definite A. In fact we have the following well known and frequently referenced error bound (see e.g. [ ]): (1.4) r k A 1 r 0 A 1 A 1 b x k A A 1 b A [ k + k ] 1 where (A) A A 1 is the spectral condition number generic notation is for either the spectral norm (the largest singular value) of a matrix or the euclidian length of a vector and (1.5) t def t +1 t 1 for t>0 that will be used frequently later for different t. The widely cited Kaniel [13] (1966) gave a proof of (1.4) while saying This result is known with a pointer to Meinardus [18] (1963). The same bound was proved to hold for Richardson-like processes [5 pp. 8 31] making it likely that (1.4) could be known before 1963 because their proofs do not differ much. This paper is concerned with how sharp this well-known error bound is. Consider A diag(λ 1 λ...λ n )andb the vector of all ones where λ j [δ 1] is either randomly or equidistantly distributed on the interval. Figure 1.1 plots the ratios of the residual bounds by (1.4) over the actual residuals. What it shows is that initially for small k bounds by (1.4) are good indications of actual residuals but

3 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 337 as k becomes larger and larger this bound overestimates the actual ones too much to be of any use. Are the phenomena in Figure 1.1 representative? Often this is what people observed [0] known as superlinear convergence. In [18] Meinardus devised an N N positive definite linear system Ax b for whichheprovedthat r N 1 A 1 r 0 A 1 [ N 1 ] 1 + (N 1) but without saying anything about all other 1 k<n 1. This example of Meinardus can be easily modified to give examples which achieve the error bound in (1.4) for any given 1 k<n 1; e.g. trivially embed an example of Meinardus of dimension k + 1 with zero blocks to make it of dimension N. Greenbaum [10 p. 5] and [9] claimed that the error bound for a given k can be attained for A having eigenvalues at the extreme points of a translated Chebyshev polynomial of degree k and some particular b whose explicit form was not given however. The same conclusion can be reached from [1 p. 561] if A s eigenvalues are chosen as Greenbaum s. This in a sense shows that the error bound in (1.4) is sharp and cannot be improved in general. But examples i.e. A and b constructed as such depend on the given step-index k and CG on any of these examples for k other than the example it was constructed for behaves much differently and in particular r k+1 0 exactly. So this only proves that the error bound is locally sharp: for given k [ ] k γ + k 1 γ (1.6) max 1. (A)γ r k A 1/ r 0 A 1 What about its global sharpness? For example Is there any positive definite system Ax b for which relative (1.7) residuals r k A 1/ r 0 A 1 achieve the error bounds by (1.4) for all 1 k<n 1? This question turns out to be too strong and the answer is no by Kaniel [13 Theorem 4.4] who showed that if r k attains the bound then it must be r k+1 0 (see also Theorem. below). So instead we ask (1.8) Is modestly bounded? sup (A)γ max 1 k n 1 [ ] k γ + k 1 γ r k A 1/ r 0 A 1 This question has been recently answered positively in Li [14] using A with eigenvalues being the translated zeros of Nth Chebyshev polynomial of the first kind. It is proved there that the ratio in (1.8) can be bounded from above by a bound that asymptotically approaches to γ / γ 1ask goes to infinity. It depends on (A) γ and unfortunately can be arbitrarily large as γ 1 +.Amuchstronger bound on the ratio namely is implied later in this paper. In what follows we shall compute the CG residuals on Meinardus examples for all 1 k N 1 and investigate extreme positive linear systems for which the kth CG residual achieves the error bound in (1.4). Before we set out to do so let us look at some numerical examples. Figure 1. plots the ratios of the error bounds

4 338 REN-CANG LI Residual bounds over actual residuals N50 Residual bounds over actual residuals N k k Figure 1.. Ratios of the error bound in (1.4) over the exact CG residuals for Meinardus example. by (1.4) over the actual CG relative residuals i.e. the right-hand side of (1.4) over its left-hand side on a Meinardus example where the exact CG residuals were carefully computed within MAPLE with a sufficiently high precision. While it is not surprising at all to see that the ratios are not smaller than 1 they seem not to be bigger than as well. This in fact will be confirmed by one of our main results which will also furnish another example for the global sharpness question (1.8) in addition to the one in [14]. The rest of this paper is organized as follows. Section explains Meinardus examples and gives our main results the closed formula for CG residuals for a Meinardus example and a complete characterization of extreme positive linear systems for which the kth CG residual achieves the error bound in (1.4). Proofs for our main results are rather long and thus are given separately in Section 3 and Section 4. Concluding remarks are given in Section 5. Notation. Throughout this paper C n m is the set of all n m complex matrices C n C n 1 andc C 1. Similarly define R n m R n andrexcept replacing the word complex by real. I n (or simply I if its dimension is clear from the context) is the n n identity matrix and e j is its jth column. The superscript T takes transpose only. We shall also adopt MATLAB-like convention to access the entries of vectors and matrices. i : j is the set of integers from i to j inclusive. For vector u and matrix X u (j) is u s jth entry X (ij) is X s (i j)th entry diag(u) isthe diagonal matrix with (diag(u)) (jj) u (j) ; X s submatrices X (k:li:j) X (k:l:) and X (:i:j) consist of intersections of row k to row l and column i to column j rowk to row l andcolumni to column j respectively.. Meinardus examples and main results The mth Chebyshev polynomial of the first kind is (.1) T m (t) cos(m arccos t) for t 1 1 ( t + ) m t 1 ( 1 + t m (.) t 1) for t 1. It frequently shows up in numerical analysis and computations because of its numerous nice properties for example T m (t) 1for t 1and T m (t) grows extremely

5 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 339 fast for t > 1. Later we will need ( ) ( ) (.3) 1+t T m t +1 1 t T m 1 [ m t 1 t + m ] t for 1 t>0. The first equality holds because T m ( t) ( 1) m T m (t). We shall prove the second equality for 0 <t<1 only and a proof for t>1 is similar. For 0 <t<1 we have ( ) 1+t 1+t 1 t t + t 1+ t 1 t 1 t 1 t t which proves (.3) for 0 <t<1. T m (t)hasm+1 extreme points in [ 1 1] so-called the mth Chebyshev extreme nodes: (.4) τ jm cosϑ jm ϑ jm j π 0 j m m at which T m (τ jm ) 1. Givenα<βset (.5) ω β α The linear transformation (.6) t(z) z ω + τ β α > 0 τ α + β β α. ( z α + β ) maps z [α β] one-to-one and onto t [ 1 1]. With its inverse transformation x(t) ω(t τ) we define the so-called mth translated Chebyshev extreme nodes on [α β]: (.7) τjm tr ω(τ jm τ) 0 j m. It can be verified that τ 0m β and τ mm α. Now we are ready to state Meinardus examples. Assume 0 <α<β. For the sake of presentation set n N 1. Let Q be any N N unitary matrix. A Meinardus example is a positive definite system Ax b with (.8) A QΛQ b QΛ 1/ g where (.9) Λ def diag(τ tr 0nτ tr 1n...τ tr nn) g (j+1) def 1/τjn tr for j {0n} /τjn tr for 1 j n 1. So an example of Meinardus is any member of the family parameterized by unitary Q. Theorem.1 is one of the two main results of this paper. Theorem.1. Let 0 <α<βand let A and b be given by (.8) and (.9). r k is the kth CG residual with initially r 0 b. Then (.10) r k A 1 r 0 A 1 ρ k [ k + k for 1 k n where (A) β/α and 1 (.11) < 1 ( ) ( 1+ n n +1 ρ k 1 ] 1 1+ k + (n k) n +1 ) 1.

6 340 REN-CANG LI Remark.1. (1) As far as the equality is concerned (.10) is valid for k 0 as well which corresponds to the very beginning of CG. () The factor ρ k is symmetrical in k about n/ i.e. ρ k ρ n k. This phenomenon certainly showed up in Figure 1. which equivalently plotted ρ 1 k. (3) ρ k 1 with equality if and only if k 0orn. (4) ρ k is strictly decreasing for k n/ (the largest integer that is no bigger than n/) and strictly increasing for k n/ (the smallest integer that is no less than n/) and 1 < min ρ k ρ n/ 1 as n. 0 k n The fact that ρ n 1 has already been established by Meinardus [18]. With it one can easily construct a positive definite linear system Ax b for which the kth CG residual achieves the error bound in (1.4). For example A and b are given by (.8) and (.9) where Λ diag(τ0k tr τtr 1k...τtr kk...) i.e. k +1 of A s eigenvalues are τ0k tr τtr 1k...τtr kk andg (j+1) is 1/τjk tr for j {0k} and /τjk tr for 1 j k 1 and zero for all other j then r k A 1/ r 0 A 1 [ ] 1. k + k For this example r k+1 0 i.e. convergence occurs at the (k + 1)th step! This is not a coincidence as it must be due to Kaniel [13 Theorem 4.4]. The following theorem characterizes all extreme linear systems as such. Theorem.. Let Ax b 0be a positive definite linear system of order N and 1 k<n.ifthekth CG residual r k (initially r 0 b) achieves the error bound in (1.4) i.e. (.1) r k A 1 r 0 A 1 [ k + k ] 1 where (A) A A 1 then the following statements hold. (1) A QΛQ and b QΛ 1/ g for some unitary Q C N N Λdiag(λ 1 λ...λ N ) with 0 <λ 1 λ λ N andg R N with all g (j) 0. () λ j λ 1 g(j) > 0 and λ j λ N g(j) > 0. (3) Let α min j λ j andβ max j λ j andletτjk tr be the translated Chebyshev extreme nodes on [α β]. The distinct λ j s in {λ j : g (j) > 0} consist of exactly τjk tr 0 j k i.e. {τjk tr 0 j k} {λ j : g (j) > 0} and λ i {τjk tr 0 j k} if g (i) > 0. (4) [13 Theorem 4.4] r k+1 0. (5) Let J l {j : λ j τlk tr g (j) > 0}. For some constant µ>0 { 1/τ tr (.13) g Jl µ lk for l {0k} /τ tr lk for 1 l k 1. Remark.. Any Ax b described by items (1) () (3) and (5) is essentially equivalent to an example of Meinardus (.8) and (.9) with N k + 1. Therefore this theorem practically says that the kth CG residual r k (initially r 0 b) achieves the error bound in (1.4) if and only if Ax b is an example of Meinardus. Thus

7 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 341 unless N there is no positive linear system whose kth CG residual achieves the error bound in (1.4) for all 1 k<n. 3. Proof of Theorem.1 We will adopt in whole the notation introduced in Section and assume 0 < α<β. Recall in particular n N 1andA is N N. Theorem.1 will be proved through a restatement. For A as in (.8) and (.9) (3.1) min x K k b Ax A 1 min φ k (0)1 φ k(a)b A 1 min φ k (0)1 φ k(λ)λ 1/ Q b min φ k (0)1 φ k(λ)g min u (1) 1 diag(g) V T k+1nu where φ k (t) is a polynomial of degree k u C k+1 and with α j+1 0 j n def α 1 α α N (3.) V k+1n α1 k α k αn k τ tr jn for a(k +1) N rectangular Vandermonde matrix. Note also that r 0 A 1 g. Therefore after substitution k +1 k Theorem.1 can be equivalently stated as follows. Theorem 3.1. Let 0 <α<β g as in (.9) andv kn as in (3.) with α j+1 τ tr jn for 0 j n. Then diag(g) VkN T (3.3) min u [ ρ k 1 k 1 u (1) 1 g for 1 k N n +1where β/α. ] 1 + (k 1). The rest of this section is devoted to the proof of this theorem. T j (t(z)) T j (z/ω + τ) is a polynomial of degree j in z; sowewrite Notice that T j (z/ω + τ) a jj z j + a j 1 j z j a 1j z + a 0j where a ij a ij (ω τ) are functions of ω and τ in (.5). Their explicit dependence on ω and τ is often suppressed for convenience. For integer m 1 define upper triangular R m R m m a matrix-valued function in ω and τ as (3.4) R m R m (ω τ) def a 00 a 01 a 0 a 0 m 1 a 11 a 1 a 1 m 1 a a m 1....ȧm 1 m 1

8 34 REN-CANG LI i.e. the jth column consists of the coefficients of T j 1 (z/ω + τ). Write V N V NN for short and set T 0 (τ 0n ) T 0 (τ 1n ) T 0 (τ nn ) def T 1 (τ 0n ) T 1 (τ 1n ) T 1 (τ nn ) (3.5) S N.... T n (τ 0n ) T n (τ 1n ) T n (τ nn ) Then V T N R N S T N.SinceR N is upper triangular we have (3.6) V T kn S T knr 1 k a key decomposition of VkN T that will play a vital role later in our proofs where S kn (S N ) (1:k:) is S N s first k rows. Set (3.7) Ω diag( ) R N N Υ def S N ΩS T N. Lemma 3. below says Υ is diagonal. So essentially (3.6) gives a QR-like decomposition of VkN T. Lemma 3.1 below is probably well known but a precise reference is hard to find. However it can be proved by using either Euler identity cos θ (e ιkθ + e ιkθ )/ where ι 1 the imaginary unit or the identities in [8 p. 30]. Detail is omitted. Lemma 3.1. Let ϑ kn πk/n as in (.4). Then N if l mn for some integer m (3.8) cos lϑ kn 0 if l is odd k0 1 if l is even but l mn for any integer m. Lemma 3. is known to [ p. 33] and probably long before that. A proof can be given with the help of Lemma 3.1 and again detail is omitted. Lemma 3.. Let S N Ω andυ be defined as in (3.5) and (3.7). ThenΥ n Ω 1. Lemma 3.3. Let Γdiag(µ+ν cos ϑ 0n µ+ν cos ϑ 1n...µ+ν cos ϑ nn ) and define Υ µν def S N ΩΓS T N whereµ ν C. We have (3.9) Υ µν n 4 Ω 1 (µω+νh)ω 1 where H R N N Proof. Notice that Υ µν µυ 10 + νυ 01 and Υ 10 S N ΩS T N Υ n Ω 1 by Lemma 3.. It is enough to calculate Υ 01. Using to mean the first and last

9 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 343 terms halved we have for 0 i j n (3.10) (S N ΩΓS T N) (i+1j+1) (S N ) (i+1k) (µ + ν cos ϑ kn )(S T N ) (kj+1) µ k0 T i (τ kn )(µ + ν cos ϑ kn )T j (τ kn ) k0 cos iϑ kn (µ + ν cos ϑ kn )cosjϑ kn k0 +ν cos iϑ kn cos jϑ kn k0 cos iϑ kn cos ϑ kn cos jϑ kn. k0 So (Υ 01 ) (i+1j+1) n k0 cos iϑ kn cos ϑ kn cos jϑ kn.now 4 cos iϑ kn cos ϑ kn cos jϑ kn k0 cos(i + j +1)ϑ kn + k0 + cos(i + j 1)ϑ kn k0 cos(i j +1)ϑ kn + k0 cos(i j 1)ϑ kn. Apply Lemma 3.1 to conclude Υ 01 n 4 Ω 1 HΩ 1 whose verification is straightforward albeit tedious. Lemma 3.4. Let m n and ξ C such that ( ξω+h) (1:m1:m) is nonsingular. Then the first entry of the solution to ( ξω+h) (1:m1:m) y e 1 is y (1) k0 γ m γ m + ξ 1(γ m + γm + ) where γ ± ξ ± ξ 1. Proof. Expand y to have a 0th entry y (0) and a (m + 1)th entry y (m+1) satisfying (3.11) y (0) ξy (1) 1 y (m+1) 0. Entry-wise we have y (i 1) ξy (i) + y (i+1) 0 for 1 i m. The general solution has form y (i) c + γ i + + c γ i whereγ ± are the two roots of 1 ξγ + γ 0 i.e. γ ± ξ ± ξ 1. We now determine c + and c by the edge conditions (3.11): (1 ξγ + ) c + + (1 ξγ ) c 1 γ m+1 + c + + γ m+1 c 0.

10 344 REN-CANG LI Notice γ + γ 1and (1 ξγ + )γ m+1 (1 ξγ )γ m+1 + (γ ξ)γ m (γ + ξ)γ m + ξ 1(γ m + γ m + ) to get γ m+1 c + ξ 1(γ m + γ+ m ) c +γ+ m+1 ξ 1(γ m + γ+ m ). Finally y (1) c + γ + + c γ. In its present general form the next lemma was proved in [14]. implied by the proof of [1 Theorem.1]. See also [16]. It was also Lemma 3.5. If Z has full column rank then (3.1) min u (1) 1 Zu [ e T 1 (Z Z) 1 e 1 ] 1/. Proof. Set v Zu. SinceZ has full column rank its Moore-Penrose pseudo-inverse is Z (Z Z) 1 Z [1] and thus u Z v. This gives a one-one and onto mapping between u C m and the column space v span(z). Now (3.13) min Zu Zu min min u (1) 1 u u (1) v span(z) where the last min is achieved at v e T 1 Z v min v v opt ( e T 1 Z ) Z(Z Z) 1 e 1 span(z) v e T 1 Z v et 1 Z 1 which implies the in (3.13) is actually an equality and u opt Z v opt /e T 1 Z v opt. Finally e T 1 Z e T 1 Z (Z ) e 1 e T 1 (Z Z) 1 e 1. This completes the proof. Proof of Theorem 3.1. By Lemma 3.5 diag(g)vkn T (3.14) min u u (1) 1 g [ ( ) ] 1 1/ e T 1 V kn [diag(g)] VkN T e1. g Let Γ diag(τ0n tr τtr 1n...τtr nn) diag(µ + ν cos ϑ 00 µ+ ν cos ϑ 01...µ+ ν cos ϑ nn ) where µ ωτ and ν ω as in (.5). Then (3.15) V kn [diag(g)] VkN T V kn Γ 1 ΩVkN T ( ) e T Γ 1 Ω ( e V k 1N Γ ) ΓVk 1N T ( e T Γ 1 Ωe e T ΩVk 1N T V k 1N Ωe V k 1N ΓΩV T k 1N )

11 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 345 where e ( ) T.NoticeVk 1N T k 1NR 1 k 1 by (3.6) to get (3.16) V k 1N Ωe V k 1N ΩVk 1N T e 1 R T k 1 (Υ 10) (1:k 11:k 1) R 1 k 1 e 1 (3.17) R T k 1 (Υ 10) (1:k 11:k 1) e 1 V k 1N ΓΩVk 1N T R T k 1 S k 1N ΓΩS T k 1NR 1 k 1 R T k 1 (Υ µν) (1:k 11:k 1) R 1 k 1 in the notation introduced in Lemma 3.3. Recall 3 ( ) 1 ( B11 B 1 B B 1 C11 1 C11 1 B 1B 1 B 1 B 1C11 1 B 1 + B 1 B 1C11 1 B 1B 1 assuming all inversions exist where C 11 B 11 B 1 B 1 B 1. We have from (3.15) e T ( 1 VkN [diag(g)] VkN) T 1 (3.18) e1 1 [ ζ e T ΩV T ( k 1N Vk 1N ΓΩV T 1 1 k 1N) Vk 1N Ωe] where ζ e T Γ 1 Ωe. But from (3.16) and (3.17) ) (3.19) e T ΩV T k 1N ( Vk 1N ΓΩV T k 1N) 1 Vk 1N Ωe ] 1 e T 1 (Υ 10 ) (1:k 11:k 1) [(Υ µν ) (1:k 11:k 1) (Υ10 ) (1:k 11:k 1) e 1 [ ] 1 n e T 1 (Υ µν ) (1:k 11:k 1) e1 and for k N by Lemma 3.4 with m k 1andξ τ [ ] 1 (3.0) e T 1 (Υ µν ) (1:k 11:k 1) e1 n 1 e T [ ] 1 1 (µω+νh)(1:k 11:k 1) e1 1 [ ] 1 nω et 1 ( τω+h)(1:k 11:k 1) e1 1 nω γ k 1 τ 1(γ k 1 γ k γ k 1 + ) where γ ± τ ± τ 1. The conditions of Lemma 3.4 are fulfilled because τ > 1 and τω+h is diagonally dominant and thus nonsingular. Since ζ g we have by (3.14) and (3.18) (3.0) diag(g)vkn T (3.1) min u u (1) 1 g [ n 1 ωζ τ 1 γ k 1 γ k 1 We now compute ωζ τ 1. Let f(z) def n tr j0 (z τjn ). Then f(z) η(z τ0n)(z tr τnn) tr U n 1 (z/ω + τ) γ k γk 1 + ] 1/. 3 This is well known. See e.g. [6 pp ] [3 p. 3].

12 346 REN-CANG LI for some constant η where U n 1 (t) isthe(n 1)th Chebyshev polynomial of the second kind. This is because the zeros of U n 1 (z/ω + τ) are precisely τjn tr ω(τ jn τ) j 1...n 1. Then upon noticing τ0n tr β and τnn tr α ζ τjn tr 1 1 τ0n tr + τjn tr 1 ( 1 τnn tr α + 1 ) f (0) β f(0) j0 α + β αβ α + β αβ Recall [3 p. 37] (3.) They yield j0 (α + β) U n 1(τ)+α + βu n 1(τ)/ω αβ U n 1 (τ) ω U n 1(τ) U n 1 (τ). U n 1 (t) (t + t 1) n (t t 1) n t 1 U n 1(t) n (t + t 1) n +(t t 1) n t 1 t [ (t + t 1) n (t t 1) n] (t 1). t 1 U n 1 (τ) γn + γ n τ 1 U n 1 (τ) nγn + + γ n τ 1 τ(γn + γ n ) (τ 1) τ 1. Therefore upon noticing ω (α + β)/ andτ (β + α)/(β α) ζ α + β αβ + n γ n + γ+ n ω τ 1 γ n + τ (3.3) γn + ω τ 1 n γ n + γ+ n ω τ 1 γ n γ+ n ωζ τ 1 n γn + γ+ n (3.4) γ n. γn + Equation (3.1) and (3.4) imply diag(g)vkn T (3.5) min u [ 1 γn γ+ n γ k 1 ] 1/ γk 1 + u (1) 1 g γ n + γn + γ k 1 + γ+ k 1. Because τ ( +1)/( 1) γ k 1 and therefore diag(g)vkn T (3.6) min u [ u (1) 1 g ( 1) k 1 k 1 γ k 1 + ( 1) k 1 (k 1) 1 n n n + n k 1 k 1 (k 1) + (k 1) ] 1/. For k N n + 1 the right-hand side of (3.6) is [ n + n ] 1 aswasshown by Meinardus [18]. For any other k wehave diag(g)vkn T (3.7) min u ρ k 1 u (1) 1 g [ k 1 ] 1 + (k 1).

13 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 347 where ρ k 1 def Right-hand side of (3.6) [ ] 1 k ( k 1 ( k 1 + (k 1) ) + (k 1) 4 )( + (k 1) n (k 1) ( ( n n ) (k 1) n + n ( )( (k 1) +1 [n (k 1)] +1 n +1 4 ( n + n + [n (k 1)] ) 1/ ) (k 1) ) ) 1/ 1/ which yields (.11). 4. Proof of Theorem. We first prove two general lemmas for the Vandermonde matrix V N V NN as defined in (3.) with arbitrary possibly complex nodes α j. Lemma 4.1. Assume one or more of 1) there are fewer than n distinct α j ) some α j 0and3) some g (j) 0occur. Then { 0 min diag(g)v N T u if all αj 0; u (1) 1 αj0 g (j) otherwise. Proof. If all α j 0 only cases 1) and 3) are possible. Let l be the number of distinct α j s exclude those corresponding to g (j) 0. Thenl<n.Bypermuting the rows of diag(g)vn T we may assume that α 1α...α l are distinct and for α j (j >l)eitheritisequaltosomeα i (i l) or corresponding g (j) 0. Setv C N whose v (j) is the coefficient of z j 1 in the polynomial φ(z) l j1 (z α j). Then v (1) l j1 ( α j) 0and min N T u diag(g)vn T (v/v (1) ) 0 u (1) 1 as expected. If some α j 0thensince diag(g)vn T u α j 0 g (j) always for any vector u with u (1) 1 it suffices to find a vector u to annihilate all other rows corresponding to α j 0. Such u can be constructed similarly to what we just did. The next lemma is essentially [17 Theorems.1 and 3.1] but stated differently. The proof below has a slightly different flavor. Lemma 4. ([17]). Let V N V NN be as defined in (3.) with all nodes α j (possibly complex) distinct and let f(z) N j1 (z α j).

14 348 REN-CANG LI (1) If all g (j) 0then diag(g)vn T (4.1) min u u (1) 1 g () (4.) max g min u (1) 1 N j1 diag(g)v T N u g ( ) f(0) N α j f g (j) (α j ) j1 N f(0) α j f (α j ) j1 1/ g (j) where the maximum is achieved if and only if for some constant µ>0 [ ] 1/ f(0) (4.3) g (j) µ α j f for 1 j N. (α j ) Proof. In Lemma 3.5 take Z diag(g)vn T. The assumptions make this Z nonsingular. Therefore [ ] min diag(g)v N T u e T 1 ( V N ΦVN T ) 1 e 1 u (1) 1 e T 1 (V N ΦVN) 1 e 1 (V 1 N e 1) Φ 1 (V 1 N e 1) where Φ [diag(g)] diag(g) and V N is the complex conjugate of V N. Let y V 1 N e 1 the first column of V 1 N which consists of the constant terms of the Lagrangian basis functions l j (z) z α i α i α j i j 1 j N since l j (α i )1fori j and 0 otherwise which means the jth row of V 1 N consists of the coefficients of l j (z). Therefore e T 1 ( V N ( ) f(0) N ΦVN T ) 1 e 1 α j1 j f g (j) (α j ) diag(g)vn T min N ( ) f(0) 1/ (4.4) u (1) 1 g α j1 j f g (j) g (j) (α j ) j1 1/ N f(0) α j f (α j ) 1/ j1 where it is an equality if and only if g (j) are given by (4.3). Remark 4.1. This lemma closely relates to a result of Greenbaum [9 (.) and Theorem 1] which in our notation essentially proved that if all nodes α j > 0 there exist k of α j s: α j1...α jk such that max g min u (1) 1 diag(g)v T kn u g max h min u (1) 1 diag(h)v T k u h.

15 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 349 where V k is the k k Vandermonde matrix with nodes α ji (1 i k). Notice the difference in conditions: Lemma 4.3 only covers k N while this result of Greenbaum s is for all 1 k N but requires all α j > 0. Greenbaum [9 Theorem 1] also obtained an expression for the optimal h but a bit of more complicated than we get from applying Lemma 4.3. It is not clear how to find out the most relevant nodes α ji. Lemma 4.3. Let ω τ C (not necessarily associated with any interval [α β] as previously required) and let n N 1 and τjn tr as in (.7) with any given ω and τ. Suppose the Vandermonde matrix V N has nodes α j+1 τjn tr for 0 j n. (1) If all g (j) 0then (4.5) diag(g) VN T min u u (1) 1 g () 1 (τ0n tr g (1) + ) τ tr 0n τ tr nω N nnu n 1 (τ) g (j) N 1 j j1 1 (τ tr jn ) g (j+1) + 1 (τ tr nn) g (N) 1/ where U n 1 (t) is the (n 1)th Chebyshev polynomial of the second kind as in (3.). (4.6) max g min u (1) 1 diag(g) VN T u g nω τ tr 0n τ tr nnu n 1 (τ) 1 τ tr 0n + n 1 j1 1 τ tr jn + 1 τ tr nn 1 where the maximum is achieved if and only if for some µ>0 µ 1/ τjn tr (4.7) g (j+1) for j {0n} µ for 1 j n 1. / τ tr jn Proof. f(z) N j1 (z α j)admits f(z) η (z τ0n)(z tr τnn)u tr n 1 (z/ω + τ) where η 1 is the coefficient of z n 1 in U n 1 (z/ω + τ). We have f(0) ητ0nτ tr nn tr U n 1 (τ) f (τ0n) tr η (τ0n tr τnn) tr U n 1 (1) η (τ0n tr τnn) tr n η nω f (τnn) tr η (τnn tr τ0n) tr U n 1 ( 1) ( 1) n η (τnn tr τ0n) tr n ( 1) n η nω

16 350 REN-CANG LI and for 1 j n 1 f (τjn) tr η (τjn tr τ0n)(τ tr jn tr τnn)u tr n 1(τ jn )/ω η (τjn tr τ0n)(τ tr jn tr τnn)n/[ω(1 tr τjn)] ηnω. Therefore by Lemma 4. we have (4.5) and (4.6). Remark 4.. As a corollary to (4.6) and the error bound in (1.4) we deduce that the right-hand side of (4.6) is equal to T n (τ) [ n + n ] 1. Proof of Theorem.. Item (1) is always true for any given positive definite system Ax b. In fact let A QΛ Q be its eigendecomposition where Q is unitary and Λ as in the theorem since A is positive definite. Set g Λ 1/ Q b. Define g ( g (1) g ()... g (N) ) T R N. Then g Dg for some diagonal D with D (jj) 1. Finally A QΛQ and b QΛ 1/ g with Q QD still unitary. Next we notice that (4.8) r k A 1 min b Ax A 1 x K k min p k(λ)g p k (0)1 min N p k (λ j ) g(j) p k (0)1 where p k (z) denotes a polynomial of degree no more than k. If either inequality in item () is violated the effective condition number <(A) as far as CG is concerned and the error bound in (1.4) gives r k A 1 r 0 A 1 [ k + k j1 ] 1 [ < k + k ] 1 contradicting (.1). This proves item (). For item (3) we first claim that λ j for which g (j) > 0isin{τjk tr 0 j k}. Otherwise if there was a j 0 such that g (j0 ) > 0andλ j0 {τjk tr 0 j k} then T k (λ j0 /ω + τ) < 1 where ω and τ are given by (.5). Now take p k (z) q k (z) in (4.8) where q k (z) T k (z/ω + τ)/t k (τ) to get r k A 1 q k (λ j0 ) g(j 0 ) + q k (λ j ) g(j) j j 0 < T k (τ) 1 g (j 0 ) + j j 0 g (j) T k (τ) 1 r 0 A 1 contradicting (.1). This proves the claim. On the other hand since r k 0there are at least k + 1 distinct values in {λ j : g (j) > 0} and therefore {λ j : g (j) > 0} {τjk tr 0 j k}. Item(3)isproved. Item (3) says effectively A has k+1 distinct eigenvalues as far as CG is concerned and thus r k+1 0. This is item (4). Kaniel [13] gave a different proof of this fact.

17 MEINARDUS EXAMPLES FOR CONJUGATE GRADIENT METHOD 351 Define ĝ R k+1 by ĝ (l+1) g Jl. (4.8) gives r k A 1 min k p k (τjk tr p k (0)1 ) ĝ(j+1) min diag(ĝ)v k+1u T u (1) 1 j0 where V k+1 V k+1k+1 is the (k +1) (k + 1) Vandermonde matrix as defined in (3.) with nodes α j+1 τjk tr for 0 j k. The condition (.1) and the error bound in (1.4) imply that for ĝ min diag(ĝ)v k+1u T max min diag(h)v k+1u T. u (1) 1 h u (1) 1 Lemma 4.3 shows ĝ (l+1) g Jl must take the form of (.13). 5. Concluding remarks We have found a closed formula for the CG residuals for Meinardus examples. These residuals may deviate from the well-known error bounds in (1.4) by a factor no bigger than 1/ indicating the error bounds by (1.4) governing the CG convergence rate is very tight in general. Three key technical components that made our computations possible are as follows: (1) transforming CG residual computations as minimization problems involving rectangular Vandermonde matrices () the QR-like decomposition VN T ST NR 1 N and (3) the solution to min u(1) 1 Zu. It turns out that QR-like decompositions exist for quite a few Vandermonde matrices and the combination of the three technical components have been used in [14 15] for arriving at the asymptotically optimally conditioned real Vandermonde matrices analyzing the sharpness of existing error bounds for CG and the symmetric Lanczos method for eigenvalue problems. We completely characterized the extreme positive linear systems for which the kth CG residual achieves the error bound in (1.4). Roughly speaking as far as CG is concerned these extreme examples are nothing but one of Meinardus examples of order k + 1. As a consequence unless N there is no positive linear system whose kth CG residual achieves the error bound in (1.4) for all 1 k<n. Acknowledgment The author wishes to acknowledge the anonymous referee to draw his attention to Chapter II in [5] by H. Rutishauser. References [1] O. Axelsson Iterative solution methods Cambridge University Press New York MR (95f:65005) [] B. Beckermann On the numerical condition of polynomial bases: Estimates for the condition number of Vandermonde Krylov and Hankel matrices Habilitationsschrift Universität Hannover April 1996; see Beckermann.pdf. [3] P. Borwein and T. Erdélyi Polynomials and polynomial inequalities Graduate Texts in Mathematics vol. 161 Springer New York MR (97e:41001) [4] J. Demmel Applied numerical linear algebra SIAM Philadelphia MR (98m:65001)

18 35 REN-CANG LI [5] M. Engeli Th. Ginsburg H. Rutishauser and E. Stiefel Refined iterative methods for computation of the solution and the eigenvalues of self-adjoint boundary value problems Birkhäuser Verlag Basel/Stuttgart MR (6:318) [6] V. N. Faddeeva Computational methods of linear algebra Dover Publications New York 1959 Translated from the Russia by Curtis D. Benster. MR (53:4500) [7] G. H. Golub and C. F. Van Loan Matrix computations 3rd ed. Johns Hopkins University Press Baltimore Maryland MR (97g:65006) [8] I. S. Gradshteyn and I. M. Ryzhik Table of integrals series and products Academic Press New York 1980 Corrected and Enlarged Edition prepared by A. Jeffrey incorporated the fourth edition prepared by Yu. V. Geronimus and M. Yu. Tseytlin translaed from the Russian by Scripta Technica Inc. MR (001c:0000) [9] A. Greenbaum Comparision of splittings used with the conjugate gradient algorithm Numer. Math. 33 (1979) MR (80k:65035) [10] Iterative methods for solving linear systems SIAM Philadelphia MR (98j:6503) [11] M. R. Hestenes and E. Stiefel Methods of conjugate gradients for solving linear systems J. Res. Nat. Bur. Standards 49 (195) MR (15651a) [1] I. C. F. Ipsen Expressions and bounds for the GMRES residual BIT 40 (000) no MR (001g:6503) [13] S. Kaniel Estimates for some computational techniques in linear algebra Math. Comp. 0 (1966) no MR (38:934) [14] R.-C. Li Sharpness in rates of convergence for CG and symmetric Lanczos methods Technical Report Department of Mathematics University of Kentucky 005 Available at [15] Vandermonde matrices with Chebyshev nodes Technical Report Department of Mathematics University of Kentucky 005 Available at Lin. Alg. Appl. to appear. [16] J. Liesen M. Rozlozník and Z. Strakos Least squares residuals and minimal residual methods SIAMJ.Sci.Comput.3 (00) no MR (003a:65033) [17] J. Liesen and P. Tichý The worst-case GMRES for normal matrices BIT44 (004) no MR (005d:65046) [18] G. Meinardus Über eine Verallgemeinerung einer Ungleichung von L. V. Kantorowitsch Numer. Math. 5 (1963) MR (8:355) [19] Y. Saad Iterative methods for sparse linear systems nd ed. SIAM Philadelphia 003. MR (004h:6500) [0] G. L. G. Sleijpen and A. van der Sluis Further results on the convergence behavior of conjugate-gradients and Ritz values Linear Algebra Appl. 46 (1996) MR (97j:65067) [1] G. W. Stewart and J.-G. Sun Matrix perturbation theory Academic Press Boston MR (9a:65017) [] L. N. Trefethen and D. Bau III Numerical linear algebra SIAM Philadelphia MR (98k:6500) [3] K.ZhouJ.C.DoyleandK.GloverRobust and optimal control Prentice Hall Department of Mathematics University of Texas at Arlington P.O. Box Arlington Texas address: rcli@.uta.edu URL: