c 2000 Society for Industrial and Applied Mathematics

Transcription

1 SIAM J. SCI. COMPUT. Vo. 2, No. 5, pp c 2000 Society for Industria and Appied Mathematics A DEFLATED VERSION OF THE CONJUGATE GRADIENT ALGORITHM Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H Abstract. We present a defated version of the conjugate gradient agorithm for soving inear systems. The new agorithm can be usefu in cases when a sma number of eigenvaues of the iteration matrix are very cose to the origin. It can aso be usefu when soving inear systems with mutipe right-hand sides, since the eigenvaue information gathered from soving one inear system can be recyced for soving the next systems and then updated. Key words. conjugate gradient, defation, mutipe right-hand sides, Lanczos agorithm AMS subject cassifications. 65F0, 65F5 PII. S Introduction. A number of recent artices have estabished the benefits of using eigenvaue defation when soving nonsymmetric inear systems with Kryov subspace methods. It has been observed that significant improvements in convergence rates can be achieved from Kryov subspace methods by adding to these subspaces a few approximate eigenvectors associated with the eigenvaues cosest to zero [2, 4, 7, 8, 3, 4]. In practice, approximations to the eigenvectors cosest to zero are obtained from the use of a certain Kryov subspace; then these approximations are dynamicay updated using the new Kryov subspace. Resuts of experiments obtained from these variations indicate that the improvement in convergence over standard Kryov subspaces of the same dimension can sometimes be substantia, especiay when the convergence of the origina scheme is hampered by a sma number of eigenvaues near zero; see, e.g., [2, 8]. In this paper we consider extensions of this idea to the conjugate gradient (CG) agorithm for the symmetric case. Our starting point is an agorithm recenty proposed by Erhe and Guyomarc h [5]. This is an augmented subspace CG method aimed at inear systems with severa right-hand sides. Erhe and Guyomarc h [5] propose an agorithm which adds one specific vector obtained from a subspace reated to a previous right-hand side. We first extend this agorithm to one which handes an arbitrary bock W of vectors. We note that introducing an arbitrary W into the Kryov subspace of CG has aready been considered by Nicoaides in [9]. The agorithm introduced in this paper is mathematicay equivaent to the one in [9]. Nicoaides s agorithm is directy derived from a defated Lanczos procedure and uses the 3-term recurrence version of the conjugate gradient agorithm. The agorithm in this paper expoits the ink between the Lanczos agorithm and the standard CG agorithm. The LDL T factorization of the projected system obtained from the same Received by the editors June, 998; accepted for pubication (in revised form) February 2, 999; pubished eectronicay Apri 28, An extended abstract of this paper appeared in the Proceedings of the Fifth Copper Mountain Conference on Iterative Methods hed in Coorado from March 30 to Apri 3, University of Minnesota, Department of Computer Science and Engineering. The work of this author was supported by NSF grant CCR University of Caifornia at Irvine, Department of Mathematics, 420 Physica Sciences I, Irvine, CA (myeung@math.uci.edu). The work of this author was supported by NSF grant CCR Institut Nationa de Recherche en Informatique et Automatique (INRIA), Rennes, France. 909

2 90 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H defated Lanczos procedure eads to a procedure that is coser to the standard CG agorithm. In the second part of the paper, we appy this technique to the situation when the bock W of added vectors is a set of approximate eigenvectors. This, in turn, is used for soving inear systems with sequentia mutipe right-hand sides. This kind of system was aso discussed in [5] for GMRES. 2. The defated Lanczos agorithm. Consider a symmetric positive definite (SPD) matrix A R n n, and et k rea vectors w, w 2,..., w k be given, aong with a unit vector v that is orthogona to w i for i =, 2,..., k. Define W = [w, w 2,..., w k ]. We assume that [w, w 2,..., w k ] is a set of ineary independent vectors. Since A is SPD, the matrix W T AW is then nonsinguar. The defated Lanczos agorithm buids a sequence {v j } j=,2,... of vectors such that (2.) v j+ span{w, v, v 2,..., v j } and (2.2) v j+ 2 =. To obtain such a sequence, we appy the standard Lanczos procedure [2, p. 74] to the auxiiary matrix (2.3) B := A AW ( W T AW ) W T A with the given initia v. The matrix B is symmetric but not necessariy positive definite. Then we have a sequence {v j } j=,2,... of Lanczos vectors which satisfies (2.4) where T j = BV j = V j T j + σ j+ v j+ e T j, ρ σ 2 σ 2 ρ 2 σ σ j ρ j σ j σ j ρ j and where V j := [v, v 2,..., v j ] and e j is the ast coumn of the j j identity matrix I j. It is guaranteed by the Lanczos procedure that the vectors v j are orthonorma to each other. From (2.4), it foows that σ j+ v j+ = Bv j σ j v j ρ j v j. Using induction and noting that W T B = 0, we have v T j+ W = 0 for j =, 2,..., and hence the sequence {v j } j=,2,... of Lanczos vectors has the properties (2.) and (2.2). Substituting B in the Lanczos agorithm appied to B with the right-hand side of (2.3) gives the foowing agorithm. Agorithm 2.. Defated Lanczos agorithm.. Choose k vectors w, w 2,..., w k. Define W = [w, w 2,..., w k ]. 2. Choose an initia vector v such that v T W = 0 and v 2 =. Set σ v 0 = For j =, 2,..., m, Do:

3 A DEFLATED CG ALGORITHM 9 4. Sove W T AW ˆν j = W T Av j for ˆν 5. z j = Av j AW ˆν j 6. ρ j = v T j z j 7. ˆv j+ = z j σ j v j ρ j v j 8. σ j+ = ˆv j+ 2 ; If σ j+ == 0 Exit. 9. v j+ = ˆv j+ /σ j+ 0. EndDo The Defated-CG agorithm proposed by Nicoaides [9] can be readiy obtained from the above agorithm by deriving a sequence of iterates whose residua vectors are proportiona to the v-vectors. 3. The Defated-CG agorithm. We now turn to the inear system (3.) Ax = b, where A is SPD. Based on the defated Lanczos procedure described in section 2, we wish to derive a projection method, foowing a standard technique used in the derivation of the CG agorithm [2, p. 79] from the standard Lanczos procedure. Here, the w i s and v j s are as defined in the defated Lanczos procedure and we use the notations W and V j introduced in section 2. Assume an initia guess x 0 to (3.) is given such that r 0 := b Ax 0 W. We set v = r 0 / r 0 2. Define K k,j (A, W, r 0 ) span{w, V j }. At the jth step of our projection method, we seek an approximate soution x j with (3.2) x j x 0 + K k,j (A, W, r 0 ) and (3.3) r j = b Ax j K k,j (A, W, r 0 ). Lemma 3.. If x j and r j satisfy (3.2) and (3.3), then (3.4) r j = c j v j+ for some scaar c j. Thus K k,j (A, W, r 0 ) = span{w, r 0, r,..., r j } and the residuas r j are orthogona to each other. Proof. Using (3.2), the approximate soution x j can be written as (3.5) x j = x 0 + W ˆξ j + V j ˆη j for some ˆξ j and ˆη j. Moreover, from (2.3) and (2.4) we have where j := ( W T AW ) W T AV j. Hence AV j = AW j + V j T j + σ j+ v j+ e T j, (3.6) r j = r 0 AW ˆξ j AV j ˆη j = r 0 AW ˆξ j ( AW j + V j T j + σ j+ v j+ e T ) j ˆηj.

4 92 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H Mutipying (3.6) with W T, and using the orthogonaity conditions (2.) and (3.3), immediatey eads to the foowing system of equations for ˆξ j and ˆη j : W T AW ˆξ j + W T AW j ˆη j = 0. Since W T AW is nonsinguar, we get ˆξ j = j ˆη j. Then (3.5) and (3.6) become and x j = x 0 W j ˆη j + V j ˆη j (3.7) r j = r 0 V j T j ˆη j σ j+ v j+ e T j ˆη j. Moreover, since r 0 = r 0 2 v by definition and V T j r j = 0 by (3.3), we have where e is the first coumn of I j. Hence V T j r j = V T j r 0 T j ˆη j = r 0 2 e T j ˆη j = 0, (3.8) ˆη j = r 0 2 T j e. Substituting (3.8) into (3.7), one can see that r j = c j v j+ for some scaar c j. Let T j = L j D j L T j be the LDL T decomposition of the symmetric matrix T j. Define (3.9) P j [p 0, p,..., p j ] = ( W j + V j ) L T j Λ j, where Λ j = diag{c 0, c,..., c j }. Proposition 3.2. The soution x j, the residua r j, and the descent direction p j satisfy the recurrence reations (3.0) x j = x j + α j p j, r j = r j α j Ap j, p j = r j + β j p j W ˆµ j for some α j, β j, ˆµ j. Thus K k,j (A, W, r 0 ) = span{w, p 0, p,..., p j }. Proof. Let The approximate soution is then given by ˆζ j = r 0 2 (L j D j Λ j ) e. x j = x 0 + r 0 2 ( W j + V j ) T j e = x 0 + P j ˆζj. Because of the ower trianguar structure of L j D j Λ j, we have the reation ˆζ j = [ ˆζj α j for some scaar α j. The recurrence reation for x j immediatey foows. ]

5 Now, rewriting (3.9) as A DEFLATED CG ALGORITHM 93 P j Λ j L T j Λ j = W j Λ j + V j Λ j and noting that Λ j L T j Λ j is unit upper bidiagona, we have by comparing the ast coumns of both sides of the above equation p j β j 2 p j 2 = W ˆµ j + c j v j, where β j 2 = c j u j,j /c j 2 and ˆµ j = c j ˆν j. Thus, the recurrence for the p j s is obtained. Proposition 3.3. The vectors p j are A-orthogona to each other, i.e., P T j AP j is diagona. In addition, they are aso A-orthogona to a w i s, i.e., W T AP j = 0. Proof. Indeed, is diagona and Pj T AP j = Pj T ( AW j + AV j ) L T j Λ j ( ) = Pj T Vj T j + σ j+ v j+ e T j L T j = Λ T j L j ( W j + V j ) T ( V j T j + σ j+ v j+ e T j = Λ T j L j T j L T j Λ j = Λ T j D jλ j Λ j W T AP j = W T ( AW j + AV j ) L T j Λ j = W T ( V j T j + σ j+ v j+ e T ) j L T j Λ j = 0. ) L T j Λ j By using the orthogonaity of r j s and the A-orthogonaity of p j s, the coefficients in (3.0) can be expressed via the vectors W, r j and p j. Proposition 3.4. The coefficients in Defated-CG satisfy the reations (3.) rt j r j α j = p T j Ap, j ˆµ j = ( W T AW ) W T Ar j, β j = rt j+ r j+ rj T r. j Proof. Mutipying (3.0) with r T j yieds rt j r j α j = rj T Ap j = rj T r j (β j 2 p j 2 + r j W ˆµ j ) T = rt j r j Ap j p T j Ap. j Simiary, the expressions for ˆµ j and β j are obtained as foows by appying (AW ) T and (Ap j ) T to (3.0), respectivey, ˆµ j = ( W T AW ) W T Ar j,

6 94 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H β j = pt j Ar j p T j Ap = (r j r j ) T r j j α j p T j Ap j = α j r T j r j p T j Ap j = rt j r j rj T r. j Putting reations (3.0) and (3.) together gives the foowing agorithm. Agorithm 3.5. Defated-CG.. Choose k ineary independent vectors w, w 2,..., w k. Define W = [w, w 2,..., w k ]. 2. Choose an initia guess x 0 such that W T r 0 = 0, where r 0 = b Ax Sove W T AW ˆµ 0 = W T Ar 0 for ˆµ and set p 0 = r 0 W ˆµ For j =, 2,..., m, Do: 5. α j = r T j r j /p T j Ap j 6. x j = x j + α j p j 7. r j = r j α j Ap j 8. β j = r T j r j/r T j r j 9. Sove W T AW ˆµ j = W T Ar j for ˆµ 0. p j = β j p j + r j W ˆµ j. EndDo To guarantee that the initia guess x 0 satisfies W T r 0 = 0, we can choose x 0 in the form (3.2) x 0 = x + W ( W T AW ) W T r, where x is arbitrary and r := b Ax. In fact, x 0 with W T r 0 = 0 must have the form (3.2). To see this, we write x 0 = x + W ( W T AW ) W T b for some x. Since W T r 0 = 0, we have W T Ax = 0 and hence x 0 = x +W ( W T AW ) W T r. Formua (3.2) was aso used in [5, 0,, 6]. A preconditioned version of Defated-CG can be derived in a straightforward way. Suppose we are soving the spit-preconditioned system L AL T y = L b, x = L T y. Set M = LL T. Directy appying the Defated-CG agorithm to the system L AL T y = L b for the y-variabe and then redefining the variabes, (3.3) L T W W, L T y j x j, Lr j r j, L T p j p j, yieds the foowing agorithm. Agorithm 3.6. Preconditioned Defated-CG.. Choose k ineary independent vectors w, w 2,..., w k. Define W = [w, w 2,..., w k ]. 2. Choose x 0 such that W T r 0 = 0, where r 0 = b Ax 0. Compute z 0 = M r Sove W T AW ˆµ 0 = W T Az 0 for ˆµ and set p 0 = W ˆµ 0 + z For j =, 2,..., m, Do: 5. α j = r T j z j /p T j Ap j 6. x j = x j + α j p j 7. r j = r j α j Ap j 8. z j = M r j 9. β j = r T j z j/r T j z j

7 A DEFLATED CG ALGORITHM Sove W T AW ˆµ j = W T Az j for ˆµ. p j = β j p j + z j W ˆµ j 2. EndDo When W is a nu matrix, Agorithm 3.6 reduces to the standard preconditioned CG agorithm; see, for instance, [2, p. 247]. In addition to the matrices A and M, five vectors and three matrices of storage are required: p, Ap, r, x, z, W, AW, and W T AW. 4. Theoretica considerations. We observe that Defated-CG is a generaization of AugCG [5] to any subspace W. Since the theory deveoped in [5] did not use the fact that W was a Kryov subspace, the convergence behavior of the Defated-CG agorithm can be anayzed by expoiting the same theory. Define H = I W ( W T AW ) (AW ) T to be the matrix of the A-orthogona projection onto W A and H T = I AW ( W T AW ) W T to be the matrix of the A -orthogona projection onto W. These matrices satisfy the equaity (4.) AH = H T A = H T AH. We first prove that the Defated-CG agorithm does not break down, and it converges. Then we derive a resut on the convergence rate and some other properties. Proposition 4.. Agorithm 3.5 is equivaent to the baanced projection method [6], defined by the soution space condition (4.2) x j+ x j K k,j (A, W, r 0 ), and the Petrov Gaerkin condition (4.3) r 0 W and r j K k,j (A, W, r 0 ). Proof. It is easy to show by induction that p j K k,j (A, W, r 0 ) hence the soution space condition is satisfied in Agorithm 3.5. It satisfies by construction the Petrov Gaerkin condition. Conversey, the BPM defined with (4.2) and (4.3) satisfies a the recurrence reations stated in the Defated-CG agorithm. The foowing resut foows immediatey. Theorem 4.2. Let A be a symmetric positive definite matrix and W be a set of ineary independent vectors. Let x be the exact soution of the inear system Ax = b. The agorithm Defated-CG appied to the inear system Ax = b wi not break down at any step. The approximate soution x j is the unique minimizer of the error norm x j x A over the affine soution space x 0 + K k,j (A, W, r 0 ) and there exists ɛ > 0, independent of x 0, such that for a k x j x A ( ɛ) x j x A. Proof. See Theorems 2.4, 2.6, and 2.7 in [6] and Theorem 2.2 in [5].

8 96 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H We can now directy appy the poynomia formaism buit in [5] to obtain convergence properties. Theorem 4.3. Let κ be the condition number of H T AH. Then (4.4) ( ) j κ x x j A 2 x x 0 A. κ + Proof. See Theorem 3.3 and Coroary 3. in [5]. Theorem 3.3 proves that r j = P j (AH)r 0, where P j is a poynomia of degree j so that, using (4.), we get r j = P j (H T AH)r 0. Then the minimization property eads to the desired resut. Defated-CG can aso be viewed as a preconditioned conjugate gradient (PCG) but with a singuar preconditioner. Define C = HH T. We use the taxonomy defined in [], where two versions of PCG are denoted by Omin(A, C, A) and Odir(A, C, A). Lemma 4.4. The Defated-CG agorithm is equivaent to the version Omin(A, C, A) of PCG appied to A with the preconditioner C = HH T and started with r 0 such that r 0 W. Proof. The reations p 0 = W ˆµ 0 + r 0 and p j = W ˆµ j + β j p j + r j can be rewritten, respectivey, as p 0 = Hr 0 and p j = Hr j + β j p j. However, since r j W, we have r j = H T r j so, p 0 = Cr 0 and p j = Cr j + β j p j. These are exacty the same reations as those of Omin(A, C, A). We must now prove that the coefficients α j and β j are the same. It is sufficient to show that (r j, Cr j ) = (r j, r j ). We have (r j, Cr j ) = (r j, HH T r j ) = (H T r j, H T r j ) = (r j, r j ). Here the preconditioner C is singuar so that convergence is not guaranteed. However, since the initia residua is orthogona to W, the foowing resut can be proved. Theorem 4.5. Defated-CG is equivaent to the version Odir(A, C, A) of PCG appied to A and C and started with r 0 W. Therefore Defated-CG converges. Proof. Since α j = (r j, r j ), we have α j 0 except the case when x j is the exact soution. Hence Defated-CG does not break down and, as shown in [], both versions Omin and Odir are equivaent. Now, using Theorem 3. in [], we infer that Defated-CG converges. To derive the convergence rate, we prove another equivaence. Theorem 4.6. Defated-CG is equivaent to CG version Omin (A,I,A), appied to the inear system H T AH x = H T b. Therefore, the convergence rate is governed by the condition number κ of H T AH and given by (4.4). Proof. Defated-CG starts with x 0 = H x 0 + W y 0 given by formua (3.2). Therefore, r 0 = H T r 0 = H T (b Ax 0 ) = H T b H T AH x 0 H T AW y 0 = H T b H T AH x 0. Agorithm Omin(A,I,A) appied to H T AH x = H T b is as foows:. Choose x 0. Define r 0 = H T b H T AH x 0 and p 0 = r For j =, 2,..., unti convergence Do:

9 A DEFLATED CG ALGORITHM α j = r j r T j / p T j HT AH p j ; 4. x j = x j + α j p j ; 5. r j = r j α j H T AH p j ; 6. βj = r j T r j/ r j r T j ; 7. p j = r j + β j p j ; 8. EndDo Now, define x j = H x j +W y 0 and p j = H p j. Using r j = H T r j and H T AW y 0 = 0, we get Now r j = H T (b Ax j ) = H T b H T AH x j = r j. p T j H T AH p j = (H p j ) T A(H p j ) = p T j Ap j. Putting everything together, we get α j = r T j r j/p T j Ap j and β j = r T j r j/r T j r j. If we rewrite the above agorithm using x j, r j, p j we get exacty agorithm Defated-CG. Therefore, the cassica resut on the convergence rate of CG can be appied; see, e.g., [2, p. 94]. 5. Systems with mutipe dependent right-hand sides. In this section, we present a method which appies the Defated-CG agorithm to the soution of severa symmetric inear systems of the form (5.) Ax = b, s =, 2,..., ν, where A R n n is SPD and where the different right-hand sides b depend on the soutions of their previous systems. This probem has been recenty considered by Erhe and Guyomarc h [5]. The main idea in [5] is to sove the first system by CG and to recyce the Kryov subspace K m (A, x 0 ) created in the first system to acceerate the convergence in soving the subsequent systems. The disadvantage of this approach is that the memory requirements coud be huge when m is arge [3] since it requires keeping a basis of an earier Kryov subspace K m (A, x 0 ). An aternative whose goa is to maintain a simiar convergence rate, is to adopt the idea of eigenvaue defation, as used in the Defated-GMRES agorithm; e.g., see [2, 4, 8]. Defated GMRES injects a few approximate eigenvectors into its Kryov soution subspace. These approximate eigenvectors are usuay seected to be those hampering the convergence of the origina scheme. Imitating the approach of Defated-GMRES, we wi add some approximate eigenvectors, usuay corresponding to eigenvaues nearest zero, to the Kryov soution subspace when we sove each, except the first, system of (5.). The eigenvectors are refined with each new system (5.) being soved. In this way, we may expect that the convergence wi be improved as more systems are soved. The memory requirements for this approach are fairy ow, since ony a sma number of eigenvectors, typicay 4 to 0, are required. We start by deriving the convergence rate from section 4 when W is a set of eigenvectors. We abe a the eigenvaues of A in increasing order: λ λ 2 λ n. In the specia case where the coumn vectors, w, w 2,..., w k, of W are exact eigenvectors of A associated with the smaest eigenvaues λ, λ 2,..., λ k, then ceary κ(h T AH) = λ n /λ k+.

10 98 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H In this specia case, the improvement on the condition number of the equivaent system soved by Defated-CG is expicity known. When the coumn vectors of W are not exact but near eigenvectors associated with λ, λ 2,..., λ k, one can ony expect that κ(h T AH) λ n /λ k Computing approximate eigenvectors. There are severa ways in which approximate eigenvectors can be extracted and improved from the data generated by the successive conjugate gradient runs appied to the systems of (5.). Let W = [w, w 2,..., w k ] be the desired set of eigenvectors to be used for the sth system. Initiay W () =. In what foows vector and scaar quantities generated by Defated-CG appied to the sth system of (5.) are denoted by the superscript s. Ideay, W is the set of eigenvectors associated with the k eigenvectors corresponding to the eigenvaues λ, λ 2,..., λ k, of A. Defated-CG is used to sove each of the systems of (5.) except the first which is soved by the standard CG. After the sth system of (5.) is soved, we update the set W of approximate eigenvectors to be used for the next right-hand side, yieding a new system W (s+). In [2], three projection techniques are described to obtain such approximations. Here we ony describe one of them, suggested by Morgan [8] and referred to as harmonic projection. This approach yieded the best resuts in finding eigenvaues nearest zero. Given ineary independent vectors { z, z 2,..., z }, the method computes the k desired approximate eigenvectors by soving the generaized eigenprobem Gy θf y = 0, where Z := [z, z 2,..., z ], G = (AZ) T AZ and F = Z T AZ. For each new system, the matrix Z is defined as foows: (5.2) where Z = [W, P ], [ ] P = p 0, p,..., p. The generaized eigenvaue probem (5.3) is soved, where (5.4) F = G y i θ i F y i = 0, i =,..., k, ( Z ) T ( AZ, G = AZ ) T AZ and where θ,..., θ k are the k smaest Ritz vaues. Then the new approximate set of eigenvectors to be used for the next system is defined as (5.5) W (s+) = Z Y, One may keep the system of residua vectors r i formuas (5.7) (5.9) become more compex. instead of the search directions p i. But the

11 A DEFLATED CG ALGORITHM 99 where Y [y,..., y k ]. This describes the method mathematicay. From the impementation point of view, it is possibe to avoid the matrix-matrix mutipication AZ in G and F in (5.4). Lemma 5.. Let R = [ r 0, r,..., r ] [ ], + = ˆµ 0, ˆµ,..., ˆµ, and L = , Ũ = β 0 β β. Then the matrix AP can be computed by (5.6) AP ( ) = R + L = W + + P +Ũ L. Proof. The resut foows immediatey from the foowing two reations of Agorithm 3.5: r j = p j Ap j = j β j p j + W ˆµ j, r j j r j+. Next, the reation estabished in the foowing proposition enabes us to sove the harmonic probem (5.3) without requiring additiona products with the matrix A. Theorem 5.2. Define D = diag{ d 0, d,..., d }, ( where d j = p j ) T Ap j, and G = ( d β 0 d 0 ) d d 0 ( + β d 2 ) d d d 2 ( d 2 + β ).

12 920 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H Then the matrices G, F, and AW are given by ( ) AW T ( ) AW W T AW G = + (5.7) ( ) T (W + L ) T AW G (5.8) F = ( (W ) T AW 0 0 D ), L, (5.9) AW (s+) = [AW, AP ]Y, where AP is given by (5.6). Proof. The desired resuts can be obtained by using (5.5), (5.4), (5.6), and the A-othorgonaity of P against W. In order to use this approach we must save d i, i, β i, ˆµ i, and p i of the first steps of the agorithm as we as the matrices W, AW, and (W ) T AW Defated-CG agorithm for dependent mutipe right-hand sides. In summary the defated agorithm for mutipe right-hand sides works as foows. Assume we want to defate with k eigenvectors. In a first run, the standard CG is run with a number of steps which is no ess than k. The data d () i, α () i, β () i for i = 0,,...,, and P () = [p () 0,..., p() ] are saved. Then G() and F () are computed according to (5.7) and (5.8) with s =. The eigenvaue probem (5.3) is soved for k eigenvectors which wi constitute the coumns of Y () and W (2) is computed as W (2) = [P () ]Y () since W () =. In subsequent steps, the defated agorithm is used instead of the standard CG using the set W W. The matrices AW and (W ) T AW are computed using (5.9). We proceed as before except that W (s+) is now defined by W (s+) = [W, P ]Y. The matrices G and F to compute the eigenvectors y i are computed according to the formuas (5.7) (5.9). A preconditioned version of the method described above can be readiy obtained by considering the spit-preconditioned systems L AL T y = L b, x = L T y, s =, 2,..., ν. As in the case of preconditioned Defated-CG, we appy the method to systems L AL T y = L b for the y-variabe and then redefine the origina variabes according to (3.3). Everything in (5.3) remains the same except G which now becomes ( ) AW T ( M AW ) W T AW G = + L (5.0) ( ) T (W + L ) T, AW G where M = LL T. Aso the computation of AP in (5.6) now becomes (5.) M AP = M R + L = ( ) W + + P +Ũ L. Putting these reations together we obtain the foowing agorithm.

13 A DEFLATED CG ALGORITHM 92 Agorithm 5.3. Defated PCG for mutipe right-hand sides.. Seect and k with k. 2. Sove the first system of (5.) with the standard PCG. 3. Compute G () and F () using (5.0) and (5.8). Set W () =. 4. For s = 2, 3,..., ν, Do: 5. Sove (5.3) for k eigenvectors y (s ), y (s ) 2,..., y (s ) [ W (s ), P (s ) ] Y (s ). k. Set W = 6. Choose Sove the s th system of (5.) by preconditioned Defated-CG with W = W. Compute G and F according to (5.0), (5.7), (5.8), and (5.). 8. EndDo In addition to the memory required by the preconditioned Defated-CG agorithm, + vectors ˆµ 0, ˆµ,..., ˆµ L, U, and D. must be stored aong with the three matrices 6. Practica considerations. We now consider the memory and computationa cost requirements of the defated CG agorithm. We assume that k n so that we can negect terms not containing n. In Defated-CG, we must store W and AW in addition to the usua vectors of CG. This means an additiona storage of 2k vectors of ength n. Defated-CG requires computing Hr j at each step. This can be done using common BLAS2 operations in z j = (AW ) T r j and r j W (W T AW ) z j. The cost of these operations is O(kn) so that the CPU overhead is not too high. There remains to consider the cost of computing W. In AugCG, which can be viewed as a particuar case of Defated- CG, W is the set of descent directions from the first system, so that the set W is A-orthogona. The method has at east two advantages: Ony the ast coumn of W needs to be saved instead of a W. The projection H simpifies into ony one orthogonaization (for ony the ast vector w k ). However, if s is greater than 2, then it is not easy to refine W. The ony soution woud be to store a consecutive sets of direction descents W at a cost of a high memory requirement. From a computationa point of view, it is advantageous to have a set of vectors W that is A-orthogona, in which case, the matrix W T AW becomes diagona. However, this is not essentia and the difference in cost invoved is minima. Approximate eigenvectors are computed from the generaized eigenprobem (5.3). We need to store not ony W and AW but aso vectors P, amounting to 2k + vectors of ength n. The computation of W using (5.5) and AW using (5.9) require mainy BLAS3 operations of compexity O(n( + )k) and the computation of W T (AW ) is a BLAS3 operation of compexity O(nk 2 ). If k and are kept sma, the overhead is modest. 7. Numerica experiments. In this section, we present some exampes to iustrate the numerica convergence behavior of Agorithm 5.3 and the anaysis in section 5. A the experiments were performed in MATLAB with machine precision 0 6. The stopping criterion for Exampes 3 is that the reative residua b Ax j 2 / b 2 be ess than 0 7. A the figures except Figure 7.4 pot the true reative residua versus the number of iterations taken. Exampe. The purpose of this exampe is to test the anaysis of section 5. We considered the matrix A = Lap(20, 20), a matrix of size n = 400 generated from a 5- point centered difference discretization of a Lapacian on a mesh (20 points in

14 922 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H true reative residua (a) Iterations Fig. 7.. Exampe. each direction). The right-hand side b of (3.) was chosen to be a random vector with independent and identicay distributed (iid) entries from a norma distribution with mean 0 and variance (N(0, )). The argest eigenvaue of A is and the four smaest ones are , 0.2, 0.2, We first computed the exact eigenvectors w, w 2, w 3 associated with the three smaest eigenvaues , 0.2, 0.2, respectivey, and then appied the standard CG with x 0 = 0, the Defated-CG with x = 0 in (3.2), and W = [w ], [w, w 2 ], [w, w 2, w 3 ], respectivey, to the system (3.). Their convergence behaviors are potted in Figure 7. with soid, dashdot, pus, and dashed curves, respectivey. From Figure 7., we can see that the convergence behavior of Defated-CG with W = [w ] is better than that of CG. Defated-CG with W = [w ] soves a system with the condition number κ = /0.2. On the other hand, the convergence rates of Defated-CG with W = [w ] and W = [w, w 2 ], respectivey, are amost the same since they are soving systems with the same condition numbers κ = /0.2. It can be understood that Defated-CG with W = [w, w 2, w 3 ] has the best behavior since the corresponding condition number κ = /.777 is the smaest. Exampes 2 and 3 demonstrate the efficiency of our agorithm when appied to (5.). We aways choose the initia guess x 0 = 0 in soving the first system and x = 0 in (3.2) for the remaining systems soving. The number of right-hand sides of (5.) is 0 and the b s are independent random vectors with iid entries from N(0, ). Athough we have seected the right-hand sides independenty, we beieve same concusions can be made when they are dependent. Aso, we kept the data in the first = 20 steps, and k = 5 approximate eigenvectors associated with smaest eigenvaues were cacuated via the QZ agorithm in Matab when we soved each system. We compared our agorithm with Agorithm AugCG [5] with m = 30 for the second system (s = 2). A the test matrices were from the Harwe Boeing coection

15 A DEFLATED CG ALGORITHM bcsstk5 0 38bus true reative residua true reative residua (b) Iterations (a) Iterations Fig (a) Convergence curves for matrix BCSSTK5 of Exampe 2. (b) Convergence curves for matrix 38bus of Exampe 3. The convergence behavior of system s =, corresponding to the standard preconditioned CG agorithm, is potted with a dotted ine. The convergence behaviors of systems s = 2,..., s = 0 using Agorithm 5.3 are potted with a dashed ine. The convergence behavior of system s = 2 using Agorithm AugCG is potted with a soid ine. Exampe 2. This exampe is the second matrix named BCSSTK5 from the BC- SSTRUC2 group of the Harwe Boeing coection. The order of the matrix is The system was preconditioned by incompete Choesky factorization ic(). Resuts are shown in Figure 7.2 (a). Exampe 3. The matrix is the fourth one named 38BUS from the PSADMIT group. The order of the matrix is 38. The ic(0) preconditioner was used and the resuts are shown in Figure 7.2 (b). For both Exampes 2 and 3, we observe that the number of iterations decreases significanty after the first few systems soving and quicky tends to its ower bound. This phenomenon is more remarkabe when and k are increased. It is because the approximate eigenvectors we chose quicky approach to their corresponding imits in the first few rounds of refinements. When these approximate eigenvectors have reached their imits, the number of iterations no onger decreases. The convergence speed of the approximate eigenvectors may depend on the distribution of eigenvaues. We computed severa extreme eigenvaues of the matrices L AL T in both exampes and found that the condition number λ n /λ and the number λ k+ /λ in Exampe 2 are both ess than the corresponding ones in Exampe 3. To some degree, the number λ k+ /λ may refect the conditions of the matrices F and G given by (5.4). This observation may hep to expain why the approximate eigenvectors in Exampe 2 have faster convergences than those in Exampe 3. In practice, we may omit ine 5 or vary k and when we find that the approximate eigenvectors are no onger improved when running Agorithm 5.3. However, the question of how to define a criterion to characterize the situation when eigenvectors no onger improve is sti an open probem. The agorithm does not aways behave so we as demonstrated in the ast two exampes. The one beow iustrates this situation.

16 924 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H 0 4 s3rmt3m3 0 4 s3rmt3m true reative residua true reative residua (a) Iterations (b) Iterations Fig (a) Convergence curves for matrix S3RMT3M3 of Exampe 4. (b) Convergence curves with correction of r j for matrix S3RMT3M3 of Exampe 4. Exampe 4. The test matrix is the one named S3RMT3M3 from the CYLSHELL group of Independent Sets and Generators. 3 In this experiment, we chose = k = 0 for Agorithm 5.3 and m = 30 for Agorithm AugCG. Moreover, the incompete Choesky preconditioner ic(2) was used and we set the stopping criterion b Ax j 2 / b 2 < 0 8 and the number of right-hand sides of (5.) to be 5. Everything remained the same as in Exampes 2 and 3 and the convergence behaviors were potted in Figure 7.3 (a). We observe from the experiment that systems 2, 3, and 5 soved by Defated- CG and system 2 soved by AugCG do not converge. What is worse is that their convergence seems to be subject to instabiity. The situation is even worse for arger and k. Theoreticay, the residuas r j in both Defated-CG and AugCG are orthogona to a the coumns of W. In practice, however, this orthogonaity is graduay ost as the agorithms progress. In fact, Figure 7.4 shows a pot of the function ( ) w T othor(j) = min i r j, i =, 2,..., k w i 2 r j 2 for system 2 soved by AugCG and system 5 soved by Defated-CG. Both curves of the function othor(j) show that oss of orthogonaity is so high that it ruins convergences. One remedy to recover orthogonaity is to add the reorthogonaization step (7.) r j := r j W ( W T W ) W T r j right after r j is computed in the agorithms. According to the anaysis in section 6, the computationa cost of the step is O(kn). We ran Agorithm 5.3 and Agorithm AugCG again with this correction incuded and potted the resuts in Figure 7.3 (b). This time, ony systems 3 and 5 soved by Defated-CG and system 2 soved by AugCG do not converge and a of them have decreasing convergence curves. By comparison, we aso potted the function othor(j) after correcting r j with (7.) in Figure 7.4. Note that the curves with correction are much ower than those without correction. The derivation of Defated-CG is basicay parae to that of CG. In Defated-CG, the standard Lanczos procedure is appied to the auxiiary matrix B in (2.3) and the 3

17 A DEFLATED CG ALGORITHM othor(j) (b) Iterations Fig Orthogonaity of r j against W of Exampe 4. Dashed: system 2 soved by AugCG; dashdot: system 5 soved by Defated-CG; soid: system 2 soved by AugCG with correction; dotted: system 5 soved by Defated-CG with correction. matrix T j in (2.4) is spitted into LDL T decomposition. Unike the case of CG, both B and T j are not necessariy positive definite. As a resut, we cannot expect that Defated-CG wi have the same robust behavior as CG does. 8. Concusion. An agorithm was presented which incorporates defation to the conjugate gradient agorithm with arbitrary systems of vectors. The method can be used for soving inear systems with mutipe and dependent right-hand sides. The main advantage of this approach is that the size k of the subspace to be kept can be kept sma without oss of efficiency reative to methods which require saving whoe previous Kryov subspaces. Another advantage is that it is easy to refine the set W as each new system is soved. Theoretica resuts as we as experimentation confirm that convergence which resuts from the defation improves substantiay as the number of systems increases. As is expected, as soon as the set W of approximate eigenvectors is computed accuratey, there is no further improvement for each new system to be soved uness the dimension of W is increases. Acknowedgments. The authors wish to thank the referees and Dr. David Day for their vauabe comments and discussions on this paper. Aso, the first two authors woud ike to acknowedge the support of the Minnesota Supercomputer Institute which provided the computer faciities and an exceent environment to conduct this research. REFERENCES [] S. F. Ashby, T. A. Manteuffe, and P. E. Sayor, A taxonomy for conjugate gradient methods, SIAM J. Numer. Ana., 27 (990), pp [2] A. Chapman and Y. Saad, Defated and augmented Kryov subspace techniques, Numer. Linear Agebra App., 4 (997), pp [3] M. O. Bristeau and J. Erhe, Augmented conjugate gradient. Appication in an iterative process for the soution of scattering probems, Numer. Agorithms, 8 (998), pp

18 926 Y. SAAD, M. YEUNG, J. ERHEL, AND F. GUYOMARC H [4] J. Erhe, K. Burrage, and B. Poh, Restarted GMRES preconditioned by defation, J. Comput. App. Math., 69 (996), pp [5] J. Erhe and F. Guyomarc h, An Augmented Subspace Conjugate Gradient, Research report RR-3278 INRIA, Domaine de Vouceau, Rocquencourt, B.P. 05, 7850 Le Chesnay, France, 997. [6] W. D. Joubert and T. A. Manteuffe, Iterative Methods for Nonsymmetric Linear Systems, Academic Press, New York, 990, pp [7] S. A. Kharchenko and A. Yu. Yeremin, Eigenvaue transation based preconditioners for the GMRES(k) method, Numer. Linear Agebra App., 2 (995), pp [8] R. B. Morgan, A restarted GMRES method augmented with eigenvectors, SIAM J. Matrix Ana. App., 6 (995), pp [9] R. A. Nicoaides, Defation of Conjugate Gradients with Appications to Boundary Vaue Probems, SIAM J. Numer. Ana., 24 (987), pp [0] B. Parett, A new ook at the Lanczos agorithm for soving symmetric systems of inear equations, Linear Agebra App., 29 (980), pp [] Y. Saad, On the Lanczos method for soving symmetric inear systems with severa right-hand sides, Math. Comp., 78 (987), pp [2] Y. Saad, Iterative Methods for Sparse Linear Systems, PWS Pubishing Company, Boston, 996. [3] Y. Saad, Anaysis of augmented Kryov subspace methods, SIAM J. Matrix Ana. App., 8 (997), pp [4] E. De Sturer, Truncation Strategies for Optima Kryov Subspace Methods, Technica Report TR-96-38, Swiss Center for Scientific Computing, Swiss Federa Institute of Technoogy, Zurich, Switzerand, 996. [5] E. De Sturer, Inner-outer methods with defation for inear systems with mutipe right-hand sides, Presentation in Househoder Symposium XIII, 996. [6] H. van der Vorst, An iterative method for soving f(a)x = b using Kryov subspace information obtained for the symmetric positive definite matrix A, J. Comput. App. Math., 8 (987), pp