ABSTRACT OF DISSERTATION. Wei Zhang. The Graduate School. University of Kentucky

Transcription

1 ABSTRACT OF DISSERTATION Wei Zhang The Graduate School University of Kentucky 2007

2 GMRES ON A TRIDIAGONAL TOEPLITZ LINEAR SYSTEM ABSTRACT OF DISSERTATION A dissertation submitted in partial fulfillment of the requirements of the degree of Doctor of Philosophy in the College of Arts and Sciences at the University of Kentucky By Wei Zhang Lexington, Kentucky Director: Dr. Ren-Cang Li, Dr. Qiang Ye, Department of Mathematics Lexington, Kentucky 2007 Copyright c Wei Zhang 2007

3 ABSTRACT OF DISSERTATION GMRES ON A TRIDIAGONAL TOEPLITZ LINEAR SYSTEM The Generalized Minimal Residual method (GMRES) is often used to solve a nonsymmetric linear system Ax = b. But its convergence analysis is a rather difficult task in general. A commonly used approach is to diagonalize A = XΛX and then separate the study of GMRES convergence behavior into optimizing the condition number of X and a polynomial minimization problem over A s spectrum. This artificial separation could greatly overestimate GMRES residuals and likely yields error bounds that are too far from the actual ones. On the other hand, considering the effects of both A s spectrum and the conditioning of X at the same time poses a difficult challenge, perhaps impossible to deal with in general but only possible for certain particular linear systems. This thesis will do so for a (nonsymmetric) tridiagonal Toeplitz system. Sharp error bounds on and sometimes exact expressions for residuals are obtained. These expressions and/or bounds are in terms of the three parameters that define A and Chebyshev polynomials of the first kind or the second kind. KEYWORDS: GMRES, rate of convergence, tridiagonal Toeplitz matrix, linear system, Chebyshev Polynomials Wei Zhang 23 July 2007

4 GMRES ON A TRIDIAGONAL TOEPLITZ LINEAR SYSTEM By Wei Zhang Dr. Ren-Cang Li, Dr. Qiang Ye Director of Dissertation Dr. Serge Ochanine Director of Graduate Studies 23 July 2007

5 RULES FOR THE USE OF DISSERTATIONS Unpublished dissertations submitted for the Master s and Doctor s degrees and deposited in the University of Kentucky Library are as a rule open for inspection, but are to be used only with due regard to the rights of the authors. Bibliographical references may be noted, but quotations or summaries of parts may be published only with the permission of the author, and with the usual scholarly acknowledgments. Extensive copying or publication of the dissertation in whole or in part requires also the consent of the Dean of the Graduate School of the University of Kentucky.

6 DISSERTATION Wei Zhang The Graduate School University of Kentucky 2007

7 GMRES ON A TRIDIAGONAL TOEPLITZ LINEAR SYSTEM DISSERTATION A dissertation submitted in partial fulfillment of the requirements of the degree of Doctor of Philosophy in the College of Arts and Sciences at the University of Kentucky By Wei Zhang Lexington, Kentucky Director: Dr. Ren-Cang Li, Dr. Qiang Ye, Department of Mathematics Lexington, Kentucky 2007 Copyright c Wei Zhang 2007

8 ACKNOWLEDGMENTS I would like to express my gratitude to all those who gave me the possibility to complete this thesis. I would like to thank Dr. Zhongwei Shen, Dr. David Johnson, Dr. Robert Molzon, Dr. Russell Brown, Dr. Peter Hislop and Dr. Serge Ochanine from Department of Mathematics of University of Kentucky, and Dr. Yuming Zhang from Department of Electrical Engineering of University of Kentucky, for their continuous support and encouragement since I joined the mathematics program. I am deeply indebted to my advisor Dr. Ren-Cang Li and Dr. Qiang Ye from Department of Mathematics of University of Kentucky, who helped and encouraged me in all the time of research for and writing of this thesis and I would like to give my special thanks to my parents whose support enabled me to complete this work. iii

9 Contents Acknowledgments List of Tables List of Figures iii vi vii Chapter Introduction. Introduction Objective Notation Chapter 2 Preliminary 6 2. Projection Method Basic Idea Properties Krylov Subspace GMRES method Arnoldi Algorithm GMRES Algorithm Practical Implementation of GMRES Chebyshev Polynomials Chebyshev Polynomials of the First Kind Chebyshev Polynomials of the Second Kind Chapter 3 Convergence Analysis Using Chebyshev Polynomials of the First Kind Residual Formulation for a Diagonalizable Linear System iv

10 3.2 Residual Reformulation Using Chebyshev Polynomials of the First Kind Estimation of Residual in General Case Residual with General Right-hand Sides The Second Part of the Proof Numerical Examples Special Right-hand Sides: b = e, b = e N Right-hand Sides b = e Numerical Examples for b = e Right-hand Sides b = e N Right-hand Sides b () e + b (N) e N Worst Convergence Speed Chapter 4 Convergence Analysis Using Chebyshev Polynomials of the Second Kind Residual Reformulation Using Chebyshev Polynomials of the Second Kind Estimation of Residual in General Case Residual with General Right-hand Sides Numerical Examples Exact Residual for Special Right-hand Sides: b = e, b = e N The Structure of Ml Chapter 5 Conclusions 84 Bibliography 89 Vita 94 v

11 List of Tables 3.3. Parameters Setting vi

12 List of Figures 2.3. AV m = V m H m + h m+,m v m+ e T m AV j = V j H j The structure of M l with N = 8 for L =, 2, 3, The structure of M l with N = 8 for L = 5, 6, 7, Computation of M (i,j) GMRES residuals for random b with τ = 0.8, and the upper bounds by Theorem GMRES residuals for random b with τ =.2, and the upper bounds by Theorem GMRES residuals for random b with τ =, and the upper bounds by Theorem GMRES residuals for random b with τ = 0.8, and the upper bounds by Theorem GMRES residuals for random b with τ =.2, and the upper bounds by Theorem GMRES residuals for random b with τ =, and the upper bounds by Theorem GMRES residuals for b = e, τ = 0.8, and the bounds by Theorem GMRES residuals for b = e, τ =.2, and the bounds by Theorem GMRES residuals for b = e, τ =, and the bounds by Theorem GMRES residuals for b = e + e N, τ = 0.8, and the bounds by Theorem GMRES residuals for b = e + e N, τ =.2, and the bounds by Theorem GMRES residuals for b = e + e N, τ =, and the bounds by Theorem vii

13 4.2. GMRES residuals for random b with τ = 0.8, and the upper bounds by Theorem 3.3. and Theorem GMRES residuals for random b with τ =.2, and the upper bounds by Theorem 3.3. and Theorem GMRES residuals for random b with τ =, and the upper bounds by Theorem 3.3. and Theorem Upper bound ratios between the bounds obtained from Theorem 4.2. and those from Theorem viii

14 Chapter Introduction. Introduction Iteration methods are often used to solve large sparse systems of linear equations. The Generalized Minimal Residual (GMRES) method [32, 4] is such an algorithm and is often used for solving a non-symmetric linear system Ax = b, (..) where A is an N N nonsingular matrix, and b is a vector with dimension N. The basic idea is to seek approximate solutions, which minimize the residual norm, within the Krylov subspaces. Specifically, the kth approximation, x k, is sought so that the kth residual, r k = b Ax k, satisfies [32] (without loss of generality, we take initially x 0 = 0 and thus r 0 = b) r k 2 = min y K k b Ay 2, where the kth Krylov subspace K k K k (A, b) of A on b is defined as K k K k (A, b) def = span{b, Ab,..., A k b}, (..2) and generic norm 2 is the usual l 2 norm of a vector or the spectral norm of a matrix. According to R. W. Freund, N. M. Nachtigal [2] and M. Embree [0], the residual norms by other algorithms, such as QMR [2, 3] and BiCGSSTAB [39, 20], are somehow

15 related to the GMRES residual norm. Hence, understanding convergence for GMRES helps us to study convergence of other algorithms. Roughly speaking, there are three kinds of convergence bounds for GMRES:. The bound based on eigenvalues with the eigenvector condition number [8, 32]; 2. The bound based on the field of values [6, 7]; 3. The bound based on pseudospectra [37, 36]. In this thesis we are most interested in the first kind. It starts by diagonalizing A = XΛX and then separating the study of GMRES convergence behavior into optimizing the condition number of X and a polynomial minimization problem over A s spectrum. This artificial separation could greatly overestimate GMRES residuals and likely yields error bounds that are too far from the actual ones. On the other hand, considering the effects of both A s spectrum and the conditioning of X at the same time poses a difficult challenge, perhaps impossible to deal with in general but possible for certain particular linear systems..2 Objective This thesis is concerned with the convergence analysis of GMRES on a linear system Ax = b whose coefficient matrix A is a (nonsymmetric) tridiagonal Toeplitz coefficient matrix λ µ ν A = µ, ν λ where λ, µ, ν are assumed nonzero and possibly complex. Linear systems as such have been studied quite extensively in the past. For the nonsymmetric case, i.e., µ ν as we 2

16 are interested in here, most up-to-date and detailed studies are due to Liesen and Strakoš [26] and Ernst []. Motivated to better understand the convergence behavior of GMRES on a convectiondiffusion model problem [27], Liesen and Strakoš and Ernst established various bounds on residual ratios. Most results in Liesen and Strakoš [26] are of a qualitative nature, intended to explain GMRES convergence behaviors for such linear systems. In particular, Liesen and Strakoš showed that GMRES for tiny µ behaves much like GMRES after setting µ to 0. Instead of eigenvalue information, Ernst used the field of values to assess the convergence rate. Our object here is to analyze the kth residual for the GMRES on tridiagonal Toeplitz A directly and arrive at simple quantitative results. The remainder of this thesis is organized as follows. Chapter 2 reviews necessary material about GMRES method and Chebyshev polynomials. Section 2. reviews projection methods and some of their properties. Section 2.2 introduces Krylov subspaces. Based on projection methods and Krylov subspaces, Section 2.3 explains the Arnoldi process and GMRES algorithm. Section 2.4 introduces Chebyshev polynomials of the first kind and the second kind. Chapter 3 covers the rate of convergence analysis of GMRES Method on a tridiagonal Toeplitz linear system by applying the Chebyshev polynomial of the first kind. Section 3. gives the calculation of the kth residual of GMRES method. Section 3.2 analyzes the norm of the residual based on rectangular Vandermonde matrices and the Chebyshev polynomial of the first kind. Section 3.3 follows the calculation of the residual in Section 3.2, and shows an estimation of the upper bound of the kth residual in a general case, which is given by Theorem Section 3.3 finishes the proof of Theorem 3.3. by analyzing the decomposition of the residual. Some numerical examples are also given 3

17 in Section 3.3. Section 3.4 lists the estimation result with some special right-hand sides: e, e N, or b () e + b (N) e N. The estimation comes from Theorem 3.3. and gives a tight upper bound. Section 3.5 estimates what the worst convergence speed could be. Chapter 4 applies Chebyshev Polynomials of the second kind to estimate the kth residual, r k, of GMRES method on a tridiagonal Teoplitz linear system. Section 4. calculates the residual by applying rectangular Vandermonde matrices and the Chebyshev polynomial of the second kind. The computation is similar to those in Chapter 3, and similar bounds are obtained. Section 4.2 follows the computation in the previous section, and gives an estimation of residuals of GMRES with a general right-hand side. The residuals with special right-hand side: b = e or b = e N are calculated exactly in Section 4.3. Some of the complicated computations, needed in Section 4.2, are presented in Section 4.4. Chapter 5 presents our concluding remarks..3 Notation Throughout this thesis, K n m is the set of all n m matrices with entries in K, where K is C (the set of complex numbers) or R (the set of real numbers), K n = K n, and K = K. I n (or simply I if its dimension is clear from the context) is the n n identity matrix, and e j is its jth column. The superscript takes conjugate transpose while T takes transpose only. The smallest singular value of X is denoted by σ min (X). We shall also adopt MATLAB-like convention to access the entries of vectors and matrices. The set of integers from i to j inclusive is i : j. For vector u and matrix X, u (j) is u s jth entry, X (i,j) is X s (i, j)th entry, diag(u) is the diagonal matrix with (diag(u)) (j,j) = u (j) ; X s submatrices X (k:l,i:j), X (k:l,:), and X (:,i:j) consists of intersections of row k to row l and column i to column j, row k to row l and all columns, and all rows 4

18 and column i to column j, respectively. Finally p ( p ) is the l p norm of a vector or the l p operator norm of a matrix, defined as ( ) /p u p = u (j) p j, X p = max Xu p. u p= α be the largest integer that is no bigger than α, and α the smallest integer that is no less than α. Throughout this thesis, exact arithmetic is assumed, A is N-by-N, and k is GMRES iteration index. Since in exact arithmetic GMRES computes the exact solution in at most N steps, r N = 0. For this reason, we restrict k < N at all times. This restriction is needed to interpret our later results concerning (worst) asymptotic speed in terms of certain limits of r k /k as k. Copyright c Wei Zhang

19 Chapter 2 Preliminary In this chapter, we briefly present the projection method, the Krylov subspace method, the GMRES method, and Chebyshev polynomials. The projection methods and some of their properties are presented in Section 2.. In Section 2.2, the Krylov subspace method is introduced as a special case of the Projection method. Section 2.3 introduces the GMRES method, including Arnoldi process and the general GMRES algorithm. Finally Section 2.4 presents Chebyshev polynomials of the first kind and the second kind. 2. Projection Method 2.. Basic Idea A projection method [2, 5] is to find an approximation to the solution of a linear system from a subspace, and is widely used for solving large linear systems. Without loss of generality, we take initially x 0 = 0. A general projection method for solving the linear system Ax = b is a method which seeks an approximate solution x m from a subspace J m of dimension m by imposing the Petrov-Galekin condition: b Ax m L m, 6

20 i.e., b Ax m, w = 0, w L m, (2..) where L m is another subspace of dimension m, and, is the inner product. The subspace J m is the search subspace, which contains the approximate solution x m. L m is called the subspace of constraints, i.e. the constraints in the Petrov-Galekin condition. J m and L m can be the same subspace, which results in an orthogonal projection method. If they are different, it is called an oblique projection method. Let J m have a basis {v, v 2,..., v m }, and V = [v, v 2,..., v m ] be an n m matrix. Let L m have a basis {w, w 2,..., w m }, and W = [w, w 2,..., w m ] be an n m matrix. Then let x m = V y, where y is a vector with dimension m. According to (2..), we have W T AV y = W T b. (2..2) If W T AV is nonsingular, then x m can be written as x m = V y = V (W T AV ) W T b. (2..3) The projection method can be presented by Algorithm 2.. [29]. Algorithm 2.. Projection Method : repeat 2: Select a pair of subspace J m and L m ; 3: Choose bases V = [v, v 2,..., v m ] and W = [w, w 2,..., w m ] for J m and L m ; 4: r := b Ax; 5: y := (W T AV ) W T r; 6: x := x + V y; 7: until convergence 2..2 Properties Obviously, Algorithm 2.. works only if W T AV is nonsingular. A special case, where W T AV is nonsingular, is presented by [29, Proposition 5.]. 7

21 Proposition 2... If A is nonsingular and L m = AJ m, then the matrix W T AV is nonsingular for any basis matrices V and W of J m and L m, respectively. Proof: Since L m = AJ m, then we have W = AV G, where G is a nonsingular m m matrix. Hence W T AV = G T (AV ) T AV. Since A is a nonsingular N N matrix, and V is a basis of J m with dimension N m, then N m matrix AV is of full column rank and (AV ) T AV is nonsingular.. [29, Proposition 5.] also presents another particular case as follows, Proposition If A is positive definite and L m = J m, then the matrix W T AV is nonsingular for any basis matrices V and W of J m and L m, respectively. Proof: Since L m = J m, then W = V G, where G is a nonsingular m m matrix. Hence W T AV = G T V T AV. Since A is positive definite, V T AV is also positive definite. Hence W T AV = G T V T AV is nonsingular.. According to the above propositions, L m is chosen as J m or AJ m very often. Let L m = AJ m. The projection method minimizes 2-norm of the residual r = b Ax [29, Proposition 5.3]. 8

22 Proposition Let A be an arbitrary square matrix, L m = AJ m. Given an initial solution x 0 = 0, a vector x is the result of an projection method onto J m, orthogonally to L m if and only if it minimizes the 2-norm of the residual vector b A x over x J m, i.e. b A x 2 = min x K m b Ax 2. Proof: If x is the minimizer of b Ax 2, it is necessary and sufficient that b A x is orthogonal to all vectors of the form v = Ay, where y J m. Since L m = AJ m, v = Ay L m. Hence x is the minimizer of b Ax 2, if and only if, b A x, v = 0, v L m, i.e. the Petrov-Galerkin condition is satisfied, and x is the approximated solution. Proposition 2..3 is applied in GMRES method. For L m = J m, there is a similar property, which is used in this thesis and omitted here. 2.2 Krylov Subspace A Krylov subspace method [29, 3] is a projection method for which the subspace J m is the Krylov subspace J m = K m (A, b) = span{b, Ab, A 2 b,..., A m b}. In this thesis, K m (A, b) will be denoted by K m. The approximations obtained from a Krylov subspace method are of the form A b x m = q m (A)b, where q m is a certain polynomial of degree (m ). 9

23 The dimension of the subspace increases by one at each step of the iterative process. The subspace K m is the subspace of all vectors in R N which can be written as x = p(a)b, where p is a polynomial of degree not exceeding m. 2.3 GMRES method The Generalized Minimum Residual Method(GMRES) [32, 4] is a projection method based on taking J m = K m and L m = AK m, in which K m is the m-th Krylov subspace with v = r 0 / r 0 2. The GMRES method minimizes the residual norm over all vectors in the Krylov subspace K m Arnoldi Algorithm Arnoldi s process [2, 30] is applied to build an orthogonal basis of the Krylov subspace K m. In exact arithmetic, the Arnoldi algorithm is described as Algorithm Algorithm 2.3. Arnoldi algorithm : Choose a vector v of norm ; 2: for j =,2,...,m do 3: Compute h ij = Av j, v i for i =,2,...,j; 4: Compute w j = Av j j i= h ijv i ; 5: h j+,j = w j 2 ; 6: if h j+,j = 0 then stop; 7: v j+ = w j /h j+,j ; 8: end for According to Algorithm 2.3., {v, v 2,...v m } forms an orthogonal basis of the Krylov subspace K m. At each step, the algorithm multiplies the previous vector v j by A and then orthonormalizes the resulting vector w j against all previous v i s by a standard Gram- Schmidt procedure. If w j vanishes, then the process stops. If w j does not vanish, then another vector v j+ is obtained, and the algorithm produces a bigger Krylov subspace K j+. If A m b K m, then the dimension of K m+ is equal to the dimension of K m. 0

24 0 A V m = V m H m +h m+,m v m+ e T m Figure 2.3.: AV m = V m H m + h m+,m v m+ e T m. According to Algorithm 2.3., the following property is obtained [29]. Proposition Let V m = [v, v 2,..., v m ], where {v,..., v m } are obtained from Algorithm 2.3., and H m be the (m + ) m Hessenberg matrix whose nonzero entries h ij are defined by Algorithm 2.3., and H m obtained from H m by deleting the last row. Then AV m = V m+ Hm (2.3.4) = V m H m + h m+,m v m+ e T m, (2.3.5) and hence V T m AV m = H m. (2.3.6) The proof just follows from Algorithm 2.3., and is omitted here. Obviously, the relation (2.3.4) and (2.3.5) are equivalent, the relation (2.3.6) is obtained by multiplying both sides of (2.3.5) by Vm T. The relation (2.3.5) is represented in Figure In practice, the Modified Gram-Schmidt [34] or the Householder Arnoldi [40] is used instead of the standard Gram-Schmidt algorithm. The Modified Gram-Schmidt results in Algorithm [34].

25 Algorithm Arnoldi-Modified Gram-Schmidt : Choose a vector v of norm ; 2: for j =,2,...,m do 3: Compute w j = Av j ; 4: for i =,2,...,j do 5: h ij = w j, v i ; 6: w j = w j h ij v i ; 7: end for 8: h j+,j = w j 2. If h j+,j = 0 then stop; 9: v j+ = w j /h j+,j ; 0: end for The Arnoldi-Modified Gram-Schmidt and Algorithm 2.3. are mathematically equivalent in exact arithmetic. If we consider the round-off in practice, Algorithm is more reliable. If even Algorithm is inadequate, then double orthogonalization or the Housholder Arnoldi can be used. The Arnoldi-Modified Gram-schmidt algorithm is applied in the GMRES algorithm in the next subsection GMRES Algorithm By using the Arnoldi-Modified Gram-schmidt algorithm [34] to build an orthogonal basis of the Krylov subspace K m, the GMRES method seek an approximation minimizing the residual norm over all vectors in the Krylov subspace K m. Any vector x in the Krylov subspace K m can be written as: x = V m y, where y is an m-vector. Then b Ax 2 = b AV m y 2. 2

26 Furthermore, we have b Ax = b AV m y = βv V m+ Hm y = V m+ (βe H m y), where β = b 2. Since the column-vectors of V m+ are othonormal, then b Ax 2 = βe H m y 2. (2.3.7) The GMRES approximation gives the unique vector in K m, which minimizes b Ax 2. According to (2.3.7), this approximation can be obtained by seeking an optimal y m that minimizes βe H m y 2, hence the approximation is x m = V m y m. The y m is easier to compute since it is the solution of an (m + ) m least-squares problem where m is not big generally. The algorithm is described as Algorithm [32]. Algorithm GMRES : Compute r 0 = b, β = r 0 2, and v = r 0 /β; 2: Define the (m + ) m matrix H m = {h ij } i m+, j m ; 3: for j =,2,...,m do 4: Compute w j = Av j ; 5: for i =,2,...,j do 6: h ij = w j, v i ; 7: w j = w j h ij v i ; 8: end for 9: h j+,j = w j 2. If h j+,j = 0 set m = j and go to 2; 0: v j+ = w j /h j+,j ; : end for 2: Compute y m, the minimizer of βe H m y 2, and x m = V m y m. Algorithm calculates the GMRES approximation as: x m = V m y m, (2.3.8) where y m = argmin y βe H m y 2. (2.3.9) 3

27 Similar to Algorithm 2.3., Algorithm will stop if w j vanishes, i.e., when h j+,j = 0 at some step j. If the algorithm stops in this way, that means the residual vector is zero, hence the approximation of GMRES will be the exact solution. The following proposition is Proposition 2 in [32]. Proposition The solution x j produced by GMRES at step j is exact if and only if the following four equivalent conditions hold:. The algorithm breaks down at step j. 2. v j+ = h j+,j = The degree of the minimal polynomial of A on the initial residual vector r 0 is equal to j, where the minimal polynomial of A on r 0 is the monic polynomial of least degree such that p(a)r 0 = 0. The breakdown results in the exact solution. Because the degree of the minimal polynomial of v can not exceed N for an N-dimensional problem, GMRES terminates in at most N steps [32]. We also can illustrate the above property by Figure 2.3.2, which is obtained by letting h j+,j = 0 in Figure Practical Implementation of GMRES In order to solve the least-squares problem min βe H m y 2 in the last step of Algorithm 2.3.3, the Hessenberg matrix is transformed into upper triangular form by using 4

28 0 A V j = V j H j Figure 2.3.2: AV j = V j H j + 0. Givens rotations [6, 32]. Define the rotation matrices as Q i =... c i s i s i c i..., (2.3.0) where c 2 i + s 2 i =. Given H m as an (m + ) m matrix, Q i s are (m + ) (m + ) matrices for i m. The idea can be best explained for m = 4. We have h h 2 h 3 h 4 β h 2 h 22 h 23 h 24 H 4 = h 32 h 33 h 34 h 43 h 44, γ = (2.3.) h 54 0 Multiply H 4 and γ by where c s s c Q =, c = s = h, h 2 + h 2 2 h 2. h 2 + h 2 2 5

29 Then we have H () 4 = h () h () 2 h () 3 h () 4 h () 22 h () 23 h () 24 h 32 h 33 h 34 h 43 h 44 h 54 c β, s β γ() = 0 0. (2.3.2) 0 Now h 2 has been eliminated, and we can multiply Q 2 to eliminate h 32, and continue the elimination process until the upper triangular H (4) 4 is obtained, h (4) h (4) 2 h (4) 3 h (4) 4 γ c β h (4) 22 h (4) 23 h (4) 24 (4) R 4 = H 4 = h (4) 33 h (4) γ 2, γ (4) = 34 γ 3 h (4) γ 4 = c 2 s β c 3 s 2 s β c 4 s 3 s 2 s β, (2.3.3) 44 0 γ 5 s 4 s 3 s 2 s β where c i and s i are defined as c i = s i = h (i ) ii (h (i ) ii h i+,i (h (i ) ii ) 2 + h 2 i+,i ) 2 + h 2 i+,i, (2.3.4). (2.3.5) Let Q = Q 4 Q 3 Q 2 Q, then R 4 = Q H 4, (2.3.6) γ (4) = Q γ = Qβe, (2.3.7) and γ (4) R 4 y = Qβe Q H 4 y (2.3.8) = Q(βe H 4 y). (2.3.9) Since Q is unitary, min γ (4) R 4 y 2 = min βe H 4 y 2. (2.3.20) 6

30 The solution to min γ (4) R 4 y 2 can be calculated by solving the upper triangular system, Moreover, min γ (4) R 4 y 2 = γ 5. h (4) h (4) 2 h (4) 3 h (4) 4 h (4) 22 h (4) 23 h (4) 24 h (4) 33 h (4) 34 h (4) 44 y = γ γ 2 γ 3 γ 4. (2.3.2) 2.4 Chebyshev Polynomials Chebyshev polynomials [28, ], named after Pafnuty Chebyshev, are a sequence of orthogonal polynomials which are related to de Moivre s formula and which are easily defined recursively. In this thesis, we calculate Chebyshev polynomials of the first kind which are denoted by T n (x) and Chebyshev polynomials of the second kind which are denoted by U n (x) Chebyshev Polynomials of the First Kind The Chebyshev polynomials of the first kind [28] are a set of orthogonal polynomials defined as the solutions to the Chebyshev differential equation. They are used as an approximation to a least squares fit. They are also intimately connected with trigonometric multiple-angle formulas. They are normalized such that T n () =. The Chebyshev polynomials of the first kind can be obtained from the generating function g(t, x) = xt 2xt + t = T 2 n (x)t n, n=0 for x < and t <. 7

31 The Chebyshev polynomials of the first kind are: T 0 (x) =, T (x) = x, T 2 (x) = 2x 2,..., T n+2 (x) = 2T n+ x T n (x). The Chebyshev polynomials of the first kind can also be defined through the identity T n (cos θ) = cos(nθ). Let x = cos(θ). Then T n (x) = cos(nθ) = cos(n arccos(x)). The sequence of Chebyshev polynomials T n (x) composes a sequence of orthogonal polynomial. Two different polynomials T n (x), T m (x) in the sequence are orthogonal to each other in the following sense dx 0, if m n, T n (x) T m (x) = π, if m = n = 0, x 2 π/2, if m = n 0. (2.4.22) Chebyshev Polynomials of the Second Kind The Chebyshev polynomials of the second kind [28] are denoted U n (x) and are defined by a different generating function g(t, x) = 2xt + t = U 2 n (x)t n, n=0 for x < and t <. 8

32 The Chebyshev polynomials of the second kind are: U 0 (x) =, U (x) = 2x, U 2 (x) = 4x 2,..., U n+2 (x) = 2U n+ x U n (x). Letting x = cos(θ) allows the Chebyshev polynomials of the second kind to be written as U n (x) = sin[(n + )θ)]. sin(θ) The sequence of U n (x) also composes a orthogonal polynomial sequence. Two different polynomials U n (x), U m (x) in the sequence are orthogonal to each other in the following sense U n (x) U m (x) x 2 dx = { 0, if m n, π/2, if m = n. (2.4.23) Copyright c Wei Zhang

33 Chapter 3 Convergence Analysis Using Chebyshev Polynomials of the First Kind In this and next chapter, we will present our main result of the convergence analysis of GMRES, i.e. estimation of the residuals of a tridiagonal Toeplitz matrix. Section 3. introduces how the residuals are calculated. Section 3.2 applies Chebyshev polynomial to calculate the residuals. In Section 3.3, the upper bounds of estimated residuals in general case are given, and numerical examples are provided to show how sharp our error bounds are. More error bounds of special cases are obtained in Section 3.4 and 3.5. In this chapter, a general case means a linear system with a general right hand side; special cases are ones with special right hand sides, such as b = e, b = e N. 3. Residual Formulation for a Diagonalizable Linear System The result in this section applies to any Ax = b with diagonalizable A. Suppose A = XΛX, (3..) Λ = diag(λ,..., λ N ), (3..2) and X is an N N nonsingular matrix. Recall we assumed, without loss of generality, the initial approximation x 0 = 0 and 20

34 thus the initial residual r 0 = b Ax 0 = b. The kth GMRES residual is r k 2 = min p k (A)b 2 p k (0)= = min Xp k (Λ)X b 2 p k (0)= = min u () = Y V T k+,nu 2, (3..3) where Y = X diag(x b), (3..4) V k+,n is the (k + ) N rectangular Vandermonde matrix def λ λ 2 λ N V k+,n =......, (3..5) λ k λ k 2 λ k N having nodes {λ j } N j=[4, Lemma 2.] and the coefficients of p k (A) forms a vector u with u () =. 3.2 Residual Reformulation Using Chebyshev Polynomials of the First Kind An N N tridiagonal Toeplitz A takes this form λ µ ν A = µ CN N. (3.2.) ν λ Given parameters ν, λ, and µ are nonzero, A is diagonalizable. In fact [33, pp.3-5] (see also [26]), A = XΛX, (3.2.2) 2

35 where Λ = diag(λ,..., λ N ), (3.2.3) λ j = λ 2 µν t j, (3.2.4) t j = cos θ j, θ j = jπ N +, (3.2.5) X = ΩZ, (3.2.6) µν Ω = diag(ξ 0, ξ,..., ξ N+ ), ξ = ν, (3.2.7) 2 Z (:,j) = N + (sin jθ,..., sin jθ N ) T. (3.2.8) It can be verified that Z T Z = I N. So A is normal if ξ =, i.e., µ = ν > 0. By (3.2.4), we have λ j = ω(t j τ), j N, (3.2.9) where ω = 2 µν, (3.2.0) τ = λ 2 µν. (3.2.) Any branch of µν, once picked and fixed, is a valid choice in this thesis. According to (3..3), we have r k 2 = min u () = Y V T k+,nu 2 Y 2 min u () = V T k+,nu 2 X 2 max (X b) (i) min V i u () k+,nu T 2 = X 2 X b 2 min u () = V T k+,nu 2 X 2 X 2 b 2 min u () = V T k+,nu 2 κ(x) b 2 min max p k (0)= i 22 p k (λ i ), (3.2.2)

36 where p k is a polynomial of degree no higher than k. Thus, together with r 0 = b, they imply r k 2 / r 0 2 κ(x) min max p k (λ i ). (3.2.3) p k (0)= i Inequality (3.2.3) is often the starting point in existing quantitative analysis on GMRES convergence [8, Page 54], as it seems that there is no easy way to do otherwise. It simplifies the analysis by separating the study of GMRES convergence behavior into optimizing the condition number of X and a polynomial minimization problem over A s spectrum, but it could potentially overestimate GMRES residuals. This is partly because, as observed by Liesen and Strakoš [26], possible cancelations of huge components in X and/or X were artificially ignored for the sake of the convergence analysis. However, for tridiagonal Toeplitz matrix A the rich structure (the Chebyshev polynomial) allows us to estimate differently, namely starting with (3..3) directly. We define the mth Translated Chebyshev Polynomial of the first kind in z of degree m as T m (z; ω, τ) def = T m (z/ω + τ) (3.2.4) = a mm z m + a m m z m + + a m z + a 0m, (3.2.5) where the coefficients a jm a jm (ω, τ) are functions of ω and τ, and forms an upper triangular R m+ in terms of ω and τ as R m+ R m+ (ω, τ) def = a 00 a 0 a 02 a 0 m a a 2 a m a 22 a 2 m, (3.2.6).... a m m 23

37 i.e., the (m + )th column consists of the coefficients of T m (λ; ω, τ). Set T 0 (t ) T 0 (t 2 ) T 0 (t N ) def T (t ) T (t 2 ) T (t N ) T N =..., (3.2.7) T N (t ) T N (t 2 ) T N (t N ) and V N = V N,N for short. Then V T N R N = T T N. (3.2.8) Equation (3.2.8) yields V T N = T T NR N. Extracting the first k + columns from both sides of V T N = T T NR N yields V T k+,n = (T T N) (:,:k+) R k+. (3.2.9) Now notice Y = X diag(x b) and X = ΩZ with Z in (3.2.6) being real and orthogonal to get Y V T k+,n = ΩZ diag(z T Ω b) (T T N) (:,:k+) R k+ = ΩM (:,:k+) R k+ (3.2.20) = ΩM (:,:k+) Ω k+ Ω k+r k+. (3.2.2) where Ω k+ = Ω (:k+,:k+), the (k + )th leading submatrix of Ω, M = Z diag(z T Ω b) T T N. (3.2.22) Now we can estimate GMRES residual Then r k 2 = min u () = Y V T k+,nu 2 = min u () = ΩM (:,:k+)ω k+ Ω k+r k+ u 2. (3.2.23) σ min (ΩM (:,:k+) Ω k+ ) r k 2 min u() = Ω k+ R k+ u 2 ΩM (:,:k+) Ω k+ 2. (3.2.24) Hence, the upper bound and lower bound of the residual r k 2 could be estimated. 24

38 3.3 Estimation of Residual in General Case Set { ζ = min ξ, }, (3.3.) ξ { τ ρ = max + τ 2, τ } τ 2. (3.3.2) Note ρ always because (τ + τ 2 )(τ τ 2 ) =. In particular if λ R, µ < 0 and ν > 0, then ρ = τ + τ 2 +. Recall Chebyshev polynomials of the first kind Define T m (t) = cos(m arccos t), for t, (3.3.3) = 2 ( t + ) m ( t 2 + t m t 2 2 ), for t. (3.3.4) Φ (+) k+ (τ, ξ) def = Φ ( ) k+ (τ, ξ) def = Φ k+ (τ, ξ) def = k ξ 2j T j (τ) 2, (3.3.5) j=0 k ξ 2j T j (τ) 2, (3.3.6) j=0 k j=0 { } ζ 2j T j (τ) 2 min Φ (+) k+ (τ, ξ), Φ( ) k+ (τ, ξ), (3.3.7) where j means the first term is halved. Obviously, if ξ, then Φ k+ (τ, ξ) = Φ (+) k+ (τ, ξ); otherwise, Φ k+ (τ, ξ) = Φ ( ) k+ (τ, ξ) Residual with General Right-hand Sides Given a tridiagonal Toeplitz with a general right hand side, we have 25

39 Theorem For Ax = b, where A is tridiagonal Toeplitz as in (3.2.) with nonzero (real or complex) parameters ν, λ, and µ. Then the kth GMRES residual r k satisfies for k < N r k 2 [ ] /2 k + r Φ k+(τ, ξ). (3.3.8) The proof of Theorem 3.3. involves a complicated computation. We will prove the theorem for the case ξ first. Recall the computation in the previous section, the proof follows the inequality (3.2.24) σ min (ΩM (:,:k+) Ω k+ ) r k 2 min u() = Ω k+ R k+ u 2 Rewrite the second inequality in (3.2.24): ΩM (:,:k+) Ω k+ 2. r k 2 min u () = Ω k+r k+ u 2 ΩM (:,:k+) Ω k+ 2. (3.3.9) This is our foundation to prove Theorem There are two quantities to deal with min Ω k+r u () = k+ u 2 and ΩM (:,:k+) Ω k+ 2. (3.3.0) For the first part, the following lemma was proven in [23, 24], and also implied by the proof of [25, Theorem 2.]. See also [22]. Lemma If W has full column rank, then min W u 2 = [ ] e T (W W ) /2 e. (3.3.) u () = In particular if W is nonsingular, min u() = W u 2 = W e 2. Proof: Set v = W u. Since W has full column rank N, its Moore-Penrose pseudoinverse is W = (W W ) W [35], and thus u = W v. This gives a one-one and onto mapping between u C m and the column space v span(w ), where span(w ) = span{w (: 26

40 , ), W (:, 2),..., W (:, N)}. Then W u 2 min W u 2 = min u () = u u () where the last min is achieved at = min v span(w ) v 2 e T W v min v v 2 e T W v = et W 2, (3.3.2) v opt = (e T W ) = W (W W ) e span(w ), which implies the in (3.3.2) is an equality, and u opt = W v opt e T W v opt. Finally e T W 2 = e T W (W ) e = e T (W W ) e. The above proof is taken from [23]. By this lemma, we have (note a 00 = ) [ ] /2 min Ω k+r u () = k+ u 2 = Ω k+ R k+e 2 = 2 + Φ(+) k+ (τ, ξ). (3.3.3) This gives the first quantity in (3.3.0). Rewrite Theorem3.3. as [ ] /2 r k 2 r 0 2 k Φ k+(τ, ξ) [ ] /2 b 2 k Φ k+(τ, ξ) [ ] /2 2 + Φ k+(τ, ξ) b 2 k +. If we can show ΩM (:,:k+) Ω k+ 2 b 2 k +, (3.3.4) then according to inequality (3.3.9) and (3.3.3), Theorem 3.3. is proved. We prove the inequality (3.3.4) in the next subsection and finish the proof of Theorem

41 3.3.2 The Second Part of the Proof Now let s investigate ΩM (:,:k+) Ω k+ and prove the inequality (3.3.4). It can be seen that ΩM (:,:k+) Ω k+ = (ΩMΩ ) (:,:k+) since Ω is diagonal. To compute ΩMΩ, we shall investigate M in (3.2.22) first. M = Z diag(z T Ω b) T T N N = Z diag(zω b (l) e l ) T T N = = = l= N b (l) ξ l Z diag(ze l ) T T N l= N b (l) ξ l Z diag(z (:,l) ) T T N l= N b (l) ξ l M l, (3.3.5) l= where M l = Z diag(z (:,l) ) T T N and b (l) is the lth element of the right hand side b. It is not easy to obtain M directly, but M l can be calculated in Lemma In Lemma and in the proof of Lemma below, without causing notational conflict, we will temporarily use k as a running index, as opposed to the rest of the paper where k is reserved for GMRES step index. Lemma For θ j = N cos lθ k = k= j π and integer l, N+ N, if l = 2m(N + ) for some integer m,, if l is even, but l 2m(N + ) for any integer m, 0, if l is odd. (3.3.6) Proof: If l = 2m(N + ) for some integer m, then lθ k = 2mkπ and thus cos lθ k =. Assume that l 2m(N + ) for any integer m. Set Ω = lπ/(n + ). We have [7, p.30] N cos lθ k = k= N k= cos kω = cos N + Ω 2 NΩ sin 2. sin Ω 2 28

42 Now notice cos N+ 2 Ω = cos l 2 π = 0 for odd l and ( )l/2 for even l, and sin NΩ 2 = sin( l 2 π Ω) = ( )l/2 sin Ω for even l to conclude the proof. Lemma is Lemma 3. in [23]. Lemma Let M l def = Z diag(z (:,l) )T T N for l N. Then the entries of M l are zeros, except at those positions (i, j), graphically forming four straight lines: (a) i + j = l +, (b) i j = l, (c) j i = l +, (d) i + j = 2(N + ) l +. (3.3.7) (M l ) (i,j) = /2 for (a) and (b), except at their intersection (l, ) for which (M l ) (l,) =. (M l ) (i,j) = /2 for (c) and (d). Notice no valid entries for (c) if l N and no valid entries for (d) if l 2. Proof: For i, j N, 2(N + ) (M l ) (i,j) = 4 = 4 = 2 = N sin kθ i sin lθ k cos(j )θ k k= N sin iθ k sin lθ k cos(j )θ k k= N [cos(i l)θ k cos(i + l)θ k ] cos(j )θ k k= N [cos(i + j l )θ k + cos(i j l + )θ k k= cos(i + j + l )θ k cos(i j + l + )θ k ]. Since all i = i + j l, i 2 = i j l +, i 3 = i + j + l, i 4 = i j + l +, 29

43 are either even or odd at the same time, Lemma implies (M l ) (i,j) = 0 unless one of them takes the form 2m(N + ) for some integer m. We now investigate all possible situations as such, keeping in mind that i, j, l N.. i = i + j l = 2m(N + ). This happens if and only if m = 0, and thus i + j = l +. Then i 2 = 2j + 2, i 3 = 2l, i 4 = 2j + 2l + 2. They are all even. i 3 and i 4 do not take the form 2m(N + ) for some integers m. This is obvious for i 3, while i 4 = 2m(N + ) implies m = 0 and j = l +, and thus i = 0 which cannot happen. However if i 2 = 2m(N + ), then m = 0 and j =, and thus i = l. So Lemma implies (M l ) (i,j) = /2 for i+j = l+ and i l, while (M l ) (l,) =. 2. i 2 = i j l + = 2m(N + ). This happens if and only if m = 0, and thus i j = l. Then i = 2j 2, i 3 = 2j + 2l 2, i 4 = 2l. They are all even. i 3 and i 4 do not take the form 2m(N + ) for some integers m. This is obvious for i 4, while i 3 = 2m(N + ) implies m = and j = N + 2 l, and thus i = N + which cannot happen. However if i = 2m(N + ), then m = 0 and thus j = and i = l which has already been considered in Item. So Lemma implies (M l ) (i,j) = /2 for i j = l and i l, while (M l ) (l,) =. 3. i 3 = i + j + l = 2m(N + ). This happens if and only if m =, and thus i + j = 2(N + ) l +. Then i = 2(N + ) 2l, i 2 = 2(N + ) 2j 2l + 2, i 4 = 2(N + ) 2j

44 They are all even. i and i 2 do not take the form 2m(N + ) for some integers m. This is obvious for i, while i 2 = 2m(N + ) implies m = 0 and j = N + 2 l, and thus i = N + which cannot happen. However if i 4 = 2m(N + ), then m = and thus j = and i = 2(N + ) l which is bigger than N + 2 and not possible. So Lemma implies (M l ) (i,j) = /2 for i + j = 2(N + ) l i 4 = i j + l + = 2m(N + ). This happens if and only if m = 0, and thus j i = l +. Then i = 2j 2l 2, i 2 = 2l, i 3 = 2j 2. They are all even, and do not take the form 2m(N + ) for some integers m. This is obvious for i 2. i = 2m(N + ) implies m = 0 and j = l +, and thus i = 0 which cannot happen. i 3 = 2m(N + ) implies m = 0 and thus j = and i = l which cannot happen either. So Lemma implies (M l ) (i,j) = /2 for j i = l +. This completes the proof. Figure 3.3. gives the example for M l with N = 8 for L =, 2, 3, 4. The structure of M l, i.e. the nonzero entries, is showed in each graph. The entries on the red lines have the value 2 except on the first column with M l(l, ) = ; the entries on the black lines have the value. The lines are labeled with a, b, c, d according to Lemma Figure gives the example for M l with N = 8 for L = 5, 6, 7, 8 similarly. Now we know M l. We still need to find out ΩMΩ. Let us examine it for N = 5 in 3

45 0 M(): with N=8 0 a M(2): with N=8 2 3 c 2 3 c b b nz = nz = 3 0 M(3): with N=8 0 M(4): with N=8 2 a c 2 a c b d b d nz = nz = 3 Figure 3.3.: The structure of M l with N = 8 for L =, 2, 3, 4. 32

46 0 M(5): with N=8 0 M(6): with N=8 2 3 a c 2 3 a c b d 7 8 b d nz = nz = 3 0 M(7): with N=8 0 M(8): with N= a 3 4 a d 6 7 d 8 b nz = nz = 4 Figure 3.3.2: The structure of M l with N = 8 for L = 5, 6, 7, 8. 33

47 order to get some idea about what it may look like. ΩMΩ for N = 5 is b () 2 ξ2 b (2) 2 ξ2 b () + 2 ξ4 b (3) 2 ξ4 b (2) + 2 ξ6 b (4) 2 ξ6 b (3) + 2 ξ8 b (5) b (2) b 2 () + b 2 (3) ξ 2 b 2 (4) ξ 4 2 ξ2 b () + 2 ξ6 b (5) 2 ξ4 b (2) b (3) b 2 (2) + 2 ξ2 b (4) b 2 () + b 2 (5) ξ ξ2 b () 2 ξ6 b (5) b (4) 2 b (3) + b 2 (5) ξ 2 b 2 (2) b 2 () b 2 (5) ξ 4 b. 2 (4) ξ 4 b (5) b 2 (4) b 2 (3) b 2 (5) ξ 2 b 2 (2) 2 ξ2 b (4) b 2 () b 2 (3) ξ 2 We observe that for N = 5, the entries of ΩMΩ are polynomials in ξ with at most two terms. This turns out to be true for all N. Lemma The following statements hold.. The first column of ΩMΩ is b. Entries in every other columns taking one of the three forms: (b (n )ξ m + b (n2 )ξ m 2 )/2 with n n 2, b (n )ξ m /2, and 0, where n, n 2 N and m i 0 are nonnegative integer. 2. In each given column of ΩMΩ, any particular entry of b appears at most twice. As the consequence, we have ΩM (:,:k+) Ω k+ 2 k + b 2, if ξ. Proof: Notice M = N l= b (l)ξ l M l and consider M s (i, j)th entry which comes from the contributions from all M l. But not all of M l contribute as most of them are zero at the position. Precisely, with the help of Lemma 3.3.3, those M l that contribute nontrivially to the (i, j)th position are the following ones subject to satisfying the given inequalities. (a) if i + j N or equivalently i + j N +, M i+j gives a /2. (b) if i j + N or equivalently i j, M i j+ gives a /2. (c) if j i N or equivalently j i + 2, M j i gives a /2. 34

48 i + j N + (a) (d) i + j N + 3 i j 2 (c) (b) i j 0 (a) (c) (a). (c) (b) (d) (b) (d) Figure 3.3.3: Computation of M (i,j). (d) if 2(N + ) (i + j) + N or equivalently i + j N + 3, M 2(N+) (i+j)+ gives a /2. These inequalities, effectively 4 of them, are described in Figure The left graph shows the regions of entries as divided by inequalities in (a) and (d); the middle shows Regions of entries as divided by inequalities in (b) and (c). These four regions divided entries of M into nine possible regions showed as the right graph of Figure We shall examine each region one by one. Recall (ΩMΩ ) (i,j) = ξ i+ M (i,j) ξ j = ξ j i M (i,j), and let γ a = γ b = 2 b (i+j )ξ 2j 2, 2 b (i j+), γ c = 2 b (j i )ξ 2(j i ), γ d = 2 b (2(N+) (i+j)+)ξ 2(N+) (i+j). Each entry in the 9 possible regions in the rightmost plot of Figure is as follows.. (a) and (b): (ΩMΩ ) (i,j) = γ a + γ b. 2. (a) and (c): (ΩMΩ ) (i,j) = γ a + γ c. 35

49 3. (b) and (d): (ΩMΩ ) (i,j) = γ b + γ d. 4. (c) and (d): (ΩMΩ ) (i,j) = γ c + γ d. 5. (a) and i j = : (ΩMΩ ) (i,j) = γ a. 6. (b) and i + j = N + 2: (ΩMΩ ) (i,j) = γ b. 7. (c) and i + j = N + 2: (ΩMΩ ) (i,j) = γ c. 8. (d) and i j = : (ΩMΩ ) (i,j) = γ d. 9. i j = and i + j = N + 2: (ΩMΩ ) (i,j) = 0. In this case, i = (N + )/2 and j = (N + 3)/2. So there is only one such entry when N is odd, and none when N is even. With this profile on the entries of ΩMΩ, we have Item of the lemma immediately. Item 2 is the consequence of M = N l= b (l)ξ l M l and Lemma which implies that there are at most two nonzero entries in each column of M l. As the consequence of Item and Item 2, each column of ΩMΩ can be expressed as the sum of two vectors w and v such that w 2, v 2 b 2 /2 when ξ, and thus (ΩMΩ ) (:,j) 2 b 2 for all j N. Therefore as expected. ΩM (:,:k+) Ω k+ 2 k+ (ΩMΩ ) (:,j) 2 2 k + b 2, j= We can prove Theorem 3.3. now. Proof: First we can prove [ ] /2 r k 2 b 2 k Φ(+) k+ (τ, ξ), for ξ. (3.3.8) 36

50 Assume ξ. Inequality (3.3.8) is the consequence of estimation of residuals (3.2.24) σ min (ΩM (:,:k+) Ω k+ ) r k 2 min u() = Ω k+ R k+ u 2 the equation(3.3.3) and Lemma ΩM (:,:k+) Ω k+ 2, [ ] /2 min Ω k+r u () = k+ u 2 = Ω k+ R k+e 2 = 2 + Φ(+) k+ (τ, ξ), ΩM (:,:k+) Ω k+ 2 k + b 2, if ξ. For the other case, ξ >, the proof can be turned into this case as follows. Let Π = (e N,..., e 2, e ) R N N be the permutation matrix. Notice Π T AΠ = A T and thus Ax = b is equivalent to A T Π T x = (Π T AΠ)(Π T x) = Π T b. (3.3.9) Note K k (A T, Π T b) = K k (Π T AΠ, Π T b) = Π T K k (A, b), and r k 2 = min b Ay 2 = min Π T (b AΠ Π T y) 2 y K k (A,b) Π T y Π T K k (A,b) Since (3.3.8) is proven true, then for ξ > we have = min Π T b A T w 2. w K k (A T,Π T b) [ ] /2 r k 2 Π T b 2 k Φ( ) k+ (τ, ξ), because the ξ for A T is the reciprocal of the one for A. Remark The leftmost inequality in (3.2.24) gives a lower bound on r k 2 in terms of σ min (ΩM (:,:k+) Ω k+ ) which, however, is hard to bound from below because it can be as small as zero, unless we know more about b such as b = e or e N as in Theorems 3.4. and

51 The upper bounds can also be calculated according to the inequality (3.2.3). Theorem For Ax = b, where A is tridiagonal Toeplitz as in (3.2.) with nonzero (real or complex) parameters ν, λ, and µ. Then the kth GMRES residual r k satisfies for k < N Proof: According to (3.2.3), [ k ] /2 r k 2 κ(x) T j (τ) 2. (3.3.20) r 0 2 j=0 r k 2 / r 0 2 κ(x) min max p k (λ i ). (3.3.2) p k (0)= i Now we calculate min pk (0)= max i p k (λ i ) according to (3.2.8) Apply Lemma 3.3., we have min max p k (λ i ) = min V p k (0)= i u () k+,nu T 2 = = min u () = (T T N) (:,:k+) R k+ u 2. (3.3.22) min max p k (λ i ) = min (T T p k (0)= i u () N) (:,:k+) R = k+ u 2 = [ e T (((T T N) (:,:k+) R k+ ) (T T N) (:,:k+) R k+ ) e ] /2 = [ e T (R k+ R k+ ) e ] /2 = [ ] e T (R k+ Rk+)e /2 [ k ] /2 = T j (τ) 2. (3.3.23) j=0 Now (3.2.3) can be written as (3.3.20) Numerical Examples In all numerical examples in this paper, we always take N = 50 and λ =, and choose different τ and ξ, then calculate µ and ν as following: µ = ξ, µ = ± µ, and ν = ν = 2 τ 2 τξ. 38

52 0 0, upper bounds ( τ =0.8, ξ =0.7) N = for µ=0.4375, ν= for µ= , ν= k+ 0 0, upper bounds ( τ =0.8, ξ =.2) N = for µ=0.75, ν= for µ= 0.75, ν= k+ Figure 3.3.4: GMRES residuals for random b with τ = 0.8, and the upper bounds by Theorem

53 0 0, upper bounds ( τ =.2, ξ =0.7) N = for µ=0.2967, ν= for µ= , ν= k+ 0 0, upper bounds ( τ =.2, ξ =.2) N = for µ=0.5, ν= for µ= 0.5, ν= k+ Figure 3.3.5: GMRES residuals for random b with τ =.2, and the upper bounds by Theorem

54 0 0, upper bounds ( τ =, ξ =0.7) N = for µ=0.35, ν= for µ= 0.35, ν= k+ 0 0, upper bounds ( τ =, ξ =.2) N = for µ=0.6, ν= for µ= 0.6, ν= k+ Figure 3.3.6: GMRES residuals for random b with τ =, and the upper bounds by Theorem

55 Thus µ, ν R, and in fact ν > 0 always. When µ > 0, ξ = µ/ν < 0 and τ = /(2 µν) > 0, but when µ < 0, both ξ = ι µ/ν and τ = ι/(2 µν ) are imaginary, where ι = is the imaginary unit. Figures 3.3.4, 3.3.5, plot residual histories for examples of GMRES with each of b s entries being uniformly random in [, ], and their upper bounds (dashed lines) by Theorem The parameters, τ and ξ are set in table Table 3.3.: Parameters Setting τ ξ Figure Figure Figure Each figure has two graphs with different ξ, 0.7 or.2. Each graph has two cases for a real τ and a pure imaginary τ. In each graph, the plot for µ > 0, i.e. τ is real, is above that for µ < 0, i.e. τ is purely imaginary. In other words, this indicates that GMRES converges much faster for µ < 0 than for µ > 0 in each of the plots. There is a simple explanation for this: the eigenvalues of A (see (3.2.9) below) are further away from the origin for a pure imaginary τ than for a real τ for any fixed τ. All plots indicate that our upper bounds are tight, except for the last few steps. Note that the upper bounds for the case µ > 0 are visually indistinguishable from the horizontal line 0 0 when τ, suggesting slow convergence. Figures 3.3.7, 3.3.8, plot residual histories for examples of GMRES with each of b s entries being uniformly random in [, ], and their upper bounds (dashed lines) by Theorem In each figure, the upper bounds are much bigger than the residuals. 42