Convergence of Inner-Iteration GMRES Methods for Least Squares Problems

Transcription

1 ISSN NII Technical Report Convergence of Inner-Iteration GMRES Methods for Least Squares Problems Keiichi Moriuni and Ken Hayami NII-212-5E Aug. 212

2 CONVERGENCE OF INNER-ITERATION GMRES METHODS FOR LEAST SQUARES PROBLEMS KEIICHI MORIKUNI AND KEN HAYAMI Abstract. We develop a general convergence theory for the generalized minimal residual method for least squares problems preconditioned with inner iterations. The inner iterations are performed by stationary iterative methods. We also present theoretical ustifications for using specific inner iterations such as the Jacobi and SOR-type methods. The theory is improved particularly in the randeficient case. We analyse the spectrum of the preconditioned coefficient matrix, and characterize it by the spectral radius of the iteration matrix for the inner iterations. The analysis is supported by numerical experiments. Key words. least squares problems, iterative methods, preconditioner, inner-outer iteration, GMRES method, stationary iterative method, ran-deficient problem AMS subect classifications. 65F8, 65F1, 65F2, 65F5 1. Introduction. Consider solving least squares problems min x R n b Ax 2, (1.1) where A R m n is not necessarily of full ran and b R m is not necessarily in R(A), the range of A. The least squares problem (1.1) is equivalent to the normal equations A T Ax = A T b. (1.2) In addition, applying B R n m, we may transform the problem (1.1) to equivalent problems [6, Theorems 3.1, 3.11]. Theorem 1.1. min b Ax x R n 2 = min b ABz z R m 2 holds for all b Rm if and only if R(AB) =R(A). Theorem 1.2. min b Ax x R n 2 and min Bb BAx x R n 2 are equivalent for all b R m if and only if R(B T BA) =R(A). Thus, the original problem (1.1) may be reduced to least squares problems with a square matrix AB or BA. Based on these transformations, the generalized minimal residual method (GMRES) [12] was extended to deal with least squares problems (1.1) in [6]. The right- and left-preconditioned GMRES for least squares problems were called AB- and BA-GMRES, respectively. Sufficient conditions under which these methods determine a least squares solution without breadown for arbitrary b for overdetermined, underdetermined, and ran-deficient problems, were shown. In [9], these methods were preconditioned with several iterations of stationary iterative methods such as variants of the Jacobi overrelaxation (JOR) and successive overrelaxation (SOR) methods, which may be considered as inner iterations. The This wor was supported by the Grants-in-Aid for Scientific Research (C) of the Ministry of Education, Culture, Sports, Science and Technology, Japan. Department of Informatics, School of Multidisciplinary Sciences, The Graduate University for Advanced Studies, Soendai, Hitotsubashi, Chiyoda-u, Toyo, , Japan (moriuni@nii.ac.p). National Institute of Informatics, and Department of Informatics, School of Multidisciplinary Sciences, The Graduate University for Advanced Studies, Soendai, Hitotsubashi, Chiyoda-u, Toyo, , Japan (hayami@nii.ac.p). 1

3 2 KEIICHI MORIKUNI AND KEN HAYAMI Cimmino-NE and Cimmino-NR methods are mathematically equivalent to JOR applied to AA T u = b with x = A T u and A T Ax = A T b, respectively. The normal-error (NE-)SOR and normal-residual (NR-)SOR methods are mathematically equivalent to SOR applied to AA T u = b with x = A T u and A T Ax = A T b, respectively [9], [11], [1]. Krylov subspace methods preconditioned with inner iterations for solving linear systems of equations were described in [9] and references therein. We assumed that A should be of full-column ran for the convergence theory for BA-GMRES with the Cimmino-NR and NR-SOR inner iterations in [9], but numerical experiments in [9] showed that these methods actually converge also for ran-deficient problems. In this paper, we give theoretical ustifications for the convergence also in the ran-deficient case. The outline of the paper is as follows. In Section 2, we introduce AB- and BA- GMRES, give their new convergence theory, and analyse the spectrum of the preconditioned matrix. In Section 3, we give a framewor of inner-iteration preconditioning for these methods, and main results on sufficient conditions in terms of the inner iterations for the convergence. In Section 4, we analyse the spectrum of the the preconditioned matrix with inner iterations. In Section 5, we give the main conclusions of this paper. Throughout this paper, we use bold letters for column vectors. e denotes the th column of an identity matrix. We denote quantities related to the th inner iteration with a superscript with bracets, e.g., x (), and for outer iterations with a subscript without bracets, e.g., x.(a, b) denotes the inner product a T b between real vectors a and b. 2. GMRES methods for least squares problems. We first give an explanation about AB-GMRES and BA-GMRES. AB-GMRES applies GMRES to min u R m b ABu 2 with x = Bu, whereas BA-GMRES applies GMRES to min x R n Bb BAx 2. Concerning the convergence of AB-GMRES and BA-GMRES, we have the following [6, Corollaries 3.8, 3.19]. Theorem 2.1. If R(B T )=R(A) and R(B) =R(A T ), then AB-GMRES determines a least squares solution of min x R n b Ax 2 for all b Rm and all x R n without breadown. Here we say AB-GMRES breas down at some step if dim AB(K (AB, r )) < dim K (AB, r ) or dim K (AB, r ) <, where K (AB, r ) = span{r,abr,...,(ab) 1 r } is the Krylov subspace of dimension [2]. The breadown causes a division by in the algorithm. Theorem 2.2. If R(B T )=R(A) and R(B) =R(A T ), then BA-GMRES determines a least squares solution of min x R n b Ax 2 for all b Rm and all x R n without breadown. We say BA-GMRES breas down at some step if dim BA(K (BA,Br )) < dim K (BA,Br )ordimk (BA,Br ) <[2]. Note that R(B) =R(A T )givesr(ab) =R(A) (cf. Theorem 1.1), and R(B T )= R(A) givesr(b T BA) =R(A) (cf. Theorem 1.2).

4 CONVERGENCE OF INNER-ITERATION GMRES 3 On the other hand, the convergence of the standard GMRES method for least squares problems min b à x 2 with à x R RN N, is given as follows [5, Theorem N 2.8]. Theorem 2.3. GMRES determines a solution of min b à x 2 for all b x R N R(Ã), x R N if and only if R(Ã) N(Ã) ={}. Here, N (Ã) is the null space of à and x is the initial approximate solution for GMRES. Applying this theorem to the case of AB- and BA-GMRES, we obtain the following. Theorem 2.4. Suppose that R(B) =R(A T ). Then, AB-GMRES determines a solution of min b Ax x R n 2 without breadown for all b R(A) and x R n if and only if R(A) N(B) ={}. Proof. Substitute AB, u, and b into Ã, x, and b, respectively, in Theorem 2.3. R(B) =R(A T )givesr(ab) =R(AA T )=R(A) andn (AB) =R(B T A T ) = R(B T B) = R(B T ) = N (B), where S is the orthogonal complement of a subspace S. Theorem 1.1 completes the proof. We remar that this theorem is restricted to the consistent case b R(A). A similar theorem holds for BA-GMRES. Theorem 2.5. Suppose that R(B T )=R(A). Then, BA-GMRES determines a solution of min b Ax x R n 2 without breadown for all b Rm and x R n if and only if N (A) R(B) ={}. Proof. Substitute BA, x, andbb into Ã, x, and b, respectively, in Theorem 2.3. R(B T )=R(A) givesn (BA) =R(A T B T ) = R(A T A) = R(A T ) = N (A) and R(BA) =R(BB T )=R(B). Hence, for all Bb R(BA) =R(B) is equivalent to for all b R m. Therefore, Theorem 1.2 completes the proof. In contrast to BA-GMRES, the condition b R(AB) =R(A) is required for AB-GMRES since the preconditioned system min b ABu u R m 2 is inconsistent for b R(AB). Since the BA-GMRES algorithm will be treated in Section 3, it is given in the following. For the AB-GMRES algorithm, see [6]. Algorithm 2.6. BA-GMRES method. 1. Let x be the initial approximate solution. 2. r = b Ax, r = Br, β = r 2, v 1 = r /β 3. For =1, 2,... until convergence, Do 4. w = BAv 5. For i =1, 2,...,,Do 6. h i, =(w, v i ), w = w h i, v i 7. EndDo 8. h +1, = w 2, v +1 = w /h +1, 9. EndDo 1. y = arg min βe 1 H y 2, x = x +[v 1, v 2,...,v ] y y R Here, H = {h i, } R (+1) Spectrum of the preconditioned matrix. We describe how BA- GMRES depends on the spectrum of the preconditioned matrix. Assume R(B T )= R(A). Then, Bb R(BA) =R(B) holds, and min x R n Bb BAx 2 is equivalent to BAx = Bb. Let r = ran A, Q 1 R n r such that R(Q 1 )=R(BA), Q 2 R n (n r)

5 4 KEIICHI MORIKUNI AND KEN HAYAMI such that R(Q 2 )=R(BA),andQ =[Q 1,Q 2 ], where the columns of Q are orthonormal. Then, GMRES applied to BAx = Bb is equivalent to GMRES applied to [ ][ ] [ ] A11 A 12 x 1 b 1 x 2 =, where A 11 = Q T 1 (BA)Q 1 R r r, A 12 = Q T 1 (BA)Q 2 R r (n r), x 1 = Q T 1 x, x 2 = Q T 2 x, b 1 = Q T 1 Bb, andb 2 = Q T 2 Bb = since Bb R(BA). As shown in [5], if x R(BA) =R(B), then the R(BA) component of GMRES applied to BAx = Bb, is equivalent to GMRES applied to A 11 x 1 = b 1. On the other hand, in the R(BA) component, x 2 = x2 for all iterates x. Now, note the following. Theorem 2.7. A 11 is nonsingular if and only if R(BA) N(BA). Proof. See [5, Theorem 2.3]. Theorem 2.8. Assume R(BA) N(BA). Then, λ is an eigenvalue of BA if and only if λ is an eigenvalue of A 11. Proof. Let Q =[Q 1,Q 2 ] R n n be as given above. Then, BAu = λu, u Q T (BA)QQ T u = λq T u, u [ ][ ] [ ] [ ] A11 A 12 u 1 u 1 u 1 u 2 = u 2, u = u 2 { A 11 u 1 + A 12 u 2 = λu 1 [ ], u 1 = λu 2 u =, u 2. Hence, if λ is an eigenvalue of BA, then we have u 2 =, A 11 u 1 = λu 1,and u 1 so that λ is an eigenvalue of A 11. On the other hand, if λ isan eigenvalue of A 11, then by setting u 2 =, we can show that λ is an eigenvalue of BA. Assume R(BA) N(BA) ={}, equivalently R(B) N(A) ={}. Then, A 11 is nonsingular, and its eigenvalues are all nonzero and correspond to the nonzero eigenvalues of BA. Therefore, the convergence behavior of GMRES applied to BAx = Bb may be explained by the (nonzero) eigenvalues of A 11 (or BA). 3. BA-GMRES preconditioned by stationary iterative methods as inner iterations. Instead of applying B explicitly as in Algorithm 2.6, consider using inner iterations as follows [9]. Algorithm 3.1. BA-GMRES method with the inner-iteration preconditioning. 1. Let x be the initial approximate solution. 2. r = b Ax 3. Apply l steps of a stationary iterative method to A T Az = A T r to obtain z (l) = B (l) r. 4. β = z (l) 2, v 1 = z (l) /β 5. For =1, 2,... until convergence, Do 6. u = Av 7. Apply l steps of a stationary iterative method to A T Ay = A T u to obtain z (l) = B (l) u. 8. For i =1, 2,...,,Do 9. h i, =(z (l), v i), z (l) = z (l) h i,v i b 2

6 CONVERGENCE OF INNER-ITERATION GMRES 5 1. EndDo 11. h +1, = z (l) 2, v +1 = z (l) /h +1, 12. EndDo 13. y = arg min βe 1 H y 2, x = x +[v 1, v 2,...,v ] y y R Here, B (l) denotes the preconditioning matrix for l inner iterations. Similarly we can obtain AB-GMRES with stationary inner iterations. In lines 3 and 7 in Algorithm 3.1, stationary iterative methods are applied to the normal equations. We now introduce a stationary iterative method for the normal equations A T Az = A T c. Consider the splitting A T A = M N, where M is nonsingular. Then, consider a class of iterative methods of the form z (l) = M 1 Nz (l 1) + M 1 A T c. Let H = M 1 N = I M 1 A T A be the iteration matrix. In practice, there is no need to form A T A, M 1,andN explicitly, as will be seen in the Cimmino-NR and NR-SOR methods [11] in Section 3.1. Here, we define the following, e.g., [8]. Definition 3.2. A matrix C is called semi-convergent if lim C i exists. i Hensel [7], Oldenburger [1], and Tanabe [13] showed the following. Theorem 3.3. The following are equivalent. 1. C is semi-convergent. 2. For any eigenvalue λ of C, either (a) λ < 1 or (b) λ =1and index(i C) =1 holds. Here, index(c) denotes the smallest nonnegative integer i such that R(C i )= R(C i+1 ). Thus, index(c) is equal to the size of the largest Jordan bloc corresponding to the zero eigenvalue of C Convergence theory. We first give an explicit expression for the preconditioned matrix B (l) A for BA-GMRES with l inner iterations. Assume that the initial approximate solution for the inner iteration is z () =. Then, the lth iterate for the inner iteration is l 1 z (l) = Hz (l 1) + M 1 A T c = H M 1 A T c. (3.1) Hence, if we define the preconditioning matrix by l 1 B (l) = H M 1 A T, (3.2) = l 1 we have z (l) = B (l) c. Let C (l) = H M 1. Then, B (l) = C (l) A T. Hence, the = preconditioned matrix is expressed as B (l) A = C (l) A T A. We obtain the following. Lemma 3.4. Let A R m n, A T A = M N, where M is nonsingular, H = M 1 N, and B (l) be given by (3.2). Assume that H is semi-convergent. Then, index(b (l) A) 1 for all l 1. =

7 6 KEIICHI MORIKUNI AND KEN HAYAMI Proof. Let J = S 1 (I H)S be the Jordan canonical form of (I H). Assume that H is semi-convergent. Then, from Theorem 3.3, index(i H) = index(j) 1. Without loss of generality, we denote J as follows: [ ] J J = C n n, J = diag (J1,J 2,...,J s 1 ) C r r, J s = n r, (3.3) J s J i = diag ( J i1,j i2,...,j isi ) C n i n i (i =1, 2,...,s 1), λ i 1 λ i 1 J i = C n i n i, 1 λ i s i =1 s 1 n i = r, (3.4) i=1 n i = n i, (3.5) where r = ran A, n r is the zero matrix of size n r, J has no eigenvalues equal to zero, s is the number of distinct eigenvalues of I H, andλ i is a nonzero eigenvalue of I H. Since J is nonsingular, [ l 1 ( J) l ] B (l) A = S (I J) JS 1 = S I r I r S 1. n r = Since λ i =1 ν(h), where ν(h) is an eigenvalue of H and ν(h) < 1, 1 λ i = ν(h) < 1. The eigenvalue of I r (I r J) l has the form μ i =1 (1 λ i ) l.ifμ i =, then (1 λ i ) l = 1, which contradicts 1 λ i < 1. Hence, I r (I r J) l is nonsingular. Therefore, index(b (l) A) 1 for all l 1. Lemma 3.5. Using the notations and the assumption of Lemma 3.4, R ( B (l)t) = R(A) holds for all l 1. ( Proof. IfC (l) is nonsingular, then R B (l)t) ( = R AC (l)t) = R(A). Hence, we show that C (l) is nonsingular Assume that H is semi-convergent. Then, we have [ l 1 J C (l) = (I SJS 1 ) M 1 1 [I = S r (I r J) ] ] l S 1 M 1. li n r = As in Lemma 3.4, I r (I r ( J) l is nonsingular. Hence, C (l) is nonsingular for all l 1. Therefore, we have R B (l)t) = R(A) for all l 1. Hence, we obtain the main result. Theorem 3.6. Assume that H is semi-convergent. Then, BA-GMRES with the inner-iteration preconditioning of the form (3.1) determines a least squares solution of min b Ax x R n 2 without breadown for all b R m and all x R n. Proof. Assume that H is semi-convergent. Then, from Lemma 3.4, we have index(b ( (l) A) 1, or equivalently R(B (l) A) N(B (l) A) = {}. Moreover, since R B (l)t) ( = R(A) from Lemma 3.5, we have R(B (l) A)=R B (l) B (l)t) = R(B (l) ) ( and N (B (l) A)=R A T B (l)t) ( = R A T A ) = R(A T ) = N (A). Hence, Theorem 2.5 completes the proof.

8 CONVERGENCE OF INNER-ITERATION GMRES 7 We remar that this theorem holds whether A is of full ran or ran-deficient, and whether A is overdetermined or underdetermined, i.e., unconditionally with respect to A. As for the convergence of AB-GMRES with inner iterations, it follows from Theorem 2.4 that a similar convergence theorem holds, which may be shown via lemmas similar to Lemmas 3.4 and 3.5. Now, we consider applying Theorem 3.6 for BA-GMRES preconditioned with specific inner-iteration methods as follows. The inner-iteration preconditioning matrices for Cimmino-NR, NR-SOR, and NR-SSOR [9] are respectively obtained from (3.2) by setting ωd : Cimmino-NR, 1 M = ω (D + ωl) : NR-SOR, ω 1 (2 ω) 1 (D + ωl)d 1 (D + ωl T ) : NR-SSOR, where A T A = L + D + L T, L is a strictly lower triangular matrix, D is a diagonal matrix, and ω is the relaxation parameter. The following are the algorithms for these methods for inner iterations. Algorithm 3.7. Cimmino-NR method. 1. Let z () = and r () = c. 2. For =, 1,...,l,Do 3. d () = D 1 A T r () 4. z (+1) = z () + ωd () 5. r (+1) = r () ωad () 6. EndDo Algorithm 3.8. NR-SOR method. 1. Let z () = and r = c. 2. For =, 1,...,l,Do 3. For =1, 2,...,n,Do 4. d () =(r, a )/ a z (+1) = z () 6. r = r ωd () 7. EndDo + ωd () a Algorithm 3.9. NR-SSOR method. 1. Let z () = and r = c. 2. For =, 1,...,l,Do 3. For =1, 2,...,n,Do 4. d () =(r, a )/ a z (+ 1 2 ) = z () 6. r = r ωd () + ωd () a 7. EndDo 8. For = n, n 1,...,1, Do 9. d (+ 1 2 ) 2 =(r, a )/ a 2 1. z (+1) = z (+ 1 2 ) + ωd (+ 1 2 ) 11. r = r ωd (+ 1 2 ) 12. EndDo 13. EndDo 8. EndDo Here, a is the th column of A. According to Dax [4], the iteration matrix H for Cimmino-NR with <ω<2/ρ(d 1/2 A T AD 1/2 ) NR-SOR with <ω<2, NR-SSOR with <ω<2, is semi-convergent, where ρ(c) is the spectral radius of C. Here, we assume A has no zero columns. Hence, from Theorem 3.6, these methods can serve as the inner iterations for BA-GMRES. Theorem 3.1. Assume that the relaxation parameter for Cimmino-NR satisfies <ω<2/ρ(d 1/2 A T AD 1/2 ). Then, BA-GMRES with the Cimmino-NR inneriteration preconditioning determines a solution of min b Ax x R n 2 without breadown for all b R m and all x R n. Theorem Assume that the relaxation parameter for NR-SOR satisfies <ω<2. Then, BA-GMRES with the NR-SOR inner-iteration preconditioning determines a solution of min b Ax x R n 2 without breadown for all b R m and all a

9 8 KEIICHI MORIKUNI AND KEN HAYAMI x R n. Theorem Assume that the relaxation parameter for NR-SSOR satisfies <ω<2. Then, BA-GMRES with the NR-SSOR inner-iteration preconditioning determines a solution of min b Ax x R n 2 without breadown for all b R m and all x R n. We omit similar convergence theorems and their proofs for AB-GMRES preconditioned with the Cimmino-NE, NE-SOR, and NE-SSOR inner iterations [9]. 4. Spectrum of the preconditioned matrix with inner iterations. Next, we analyse the spectrum of the preconditioned matrix for BA-GMRES with l inner iterations. The preconditioned matrix may be expressed as B (l) A = I H l. ( Let ) ν be an eigenvalue of H. Then, there exists v such that Hv = νv or I H l v =(1 ν l )v. Hence, B (l) A has an eigenvalue μ =1 ν l. Assume that H is semi-convergent. Let r = ran A. Then, from Theorem 3.3, H has r eigenvalues such that ν < 1andn r eigenvalues such that ν =1. Forν =1, we have μ =. For ν < 1, we obtain μ 1 = ν l ρ(h) l < 1. This means that r eigenvalues of B (l) A lie inside the circle of radius ρ(h) l with center at 1, and these eigenvalues approaches 1 as l increases. The remaining n r eigenvalues are zero. We demonstrate the above observation for a test matrix called Maragal 3 [3] of size 1,69 86 with 18,391 nonzero elements (nonzero density 1.27 %), and ran 613. Figure 4.1 shows the spectrum of the preconditioned matrix B (l) A with the NR-SOR inner iterations for l = 1, 2, 4, and 8. The relaxation parameter was set to ω =1. The computations were done using Matlab 211b. As the number of inner iterations l increased, the eigenvalues of B (l) A approached 1. From the discussion in Section 2.1, we see that if H is semi-convergent, then B (l) r 2 depends on the eigenvalues of H not equal to 1, but not on the eigenvalues equal to Conclusions. We considered applying inner-iteration preconditioning to GMRES methods for least squares problems and gave a general convergence theory for the methods. Theoretical ustifications for the convergence were given also for specific inner-iteration methods lie NR-SOR. We have reinforced the previous theory particularly for the ran-deficient case. The spectrum of the preconditioned matrix was analysed and characterized using the spectral radius of the iteration matrix for the inner iterations. Numerical experiments were done to examine the analysis.

10 CONVERGENCE OF INNER-ITERATION GMRES inner iteration.3 2 inner iterations.2.2 Im[μ(B (1) A)] Im[μ(B (2) A)] Re[μ(B (1) A)] Re[μ(B (2) A)].3 4 inner iterations.3 8 inner iterations.2.2 Im[μ(B (4) A)] Im[μ(B (8) A)] Re[μ(B (4) A)] Re[μ(B (8) A)] Fig Spectrum of the preconditioned matrix B (l) A with NR-SOR inner iterations for Maragal 3. Upper left: l =1, right upper: l =2, left lower: l =4, right lower: l =8. REFERENCES [1] Å. Börc and T. Elfving, Accelerated proection methods for computing pseudoinverse solutions of systems of linear equations, BIT, 19 (1979), pp [2] P. N. Brown and H. F. Waler, GMRES on (nearly) singular systems, SIAM J. Matrix Anal. Appl., 18 (1997), pp [3] T. Davis, The University of Florida Sparse Matrix Collection, available online at The University of Florida. [4] A. Dax, The convergece of linear stationary iterative processes for solving singular unstructured systems of linear equations, SIAM Review, 32 (199), pp [5] K. Hayami and M. Sugihara, A geometric view of Krylov subspace methods on singular systems, Numer. Linear Algebra Appl., 18 (211), pp [6] K. Hayami, J.-F. Yin, and T. Ito, GMRES methods for least squares problems, SIAM J. Matrix Anal. Appl., 31 (21), pp [7] V. K. Hensel, Über potenzreihen von matrizen, J. Reine Angew. Math., 155 (1926), pp (in German). [8] C. D. Meyer and R. J. Plemmons, Convergent powers of a matrix with applications to iterative methods for singular linear sysmtes, SIAM J. Numer. Anal., 14 (1977), pp [9] K. Moriuni and K. Hayami, Inner-iteration Krylov subspace methods for least squares problems, NII Technical Report, NII-211-1E (211), pp (revised, submitted). [1] R. Oldenburger, Infinite powers of matrices and characteristic roots, Due Math., 6 (194), pp [11] Y. Saad, Iterative Methods for Sparse Linear Systems, 2nd ed., SIAM, Philadelphia, 23. [12] Y. Saad and M. H. Schultz, GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems, SIAM J. Sci. Stat. Comput., 7 (1986), pp [13] K. Tanabe, Characterization of linear stationary iterative processes for solving a singular system of linear equations, Numer. Math., 22 (1974), pp