A stable variant of Simpler GMRES and GCR

Size: px

Start display at page:

Download "A stable variant of Simpler GMRES and GCR"

Benedict O’Connor’
5 years ago
Views:

1 A stable variant of Simpler GMRES and GCR Miroslav Rozložník joint work with Pavel Jiránek and Martin H. Gutknecht Institute of Computer Science, Czech Academy of Sciences, Prague, Czech Republic miro Applied Linear Algebra Conference in honor of Hans Schneider University of Novi Sad, Serbia, May 24 28, Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 1

2 Minimum residual methods for Ax = b We consider a nonsingular linear system Ax = b, A R N N, b R N. Krylov subspace methods: initial guess x 0, compute {x n} such that x n x 0 + K n r n b Ax 0 r 0 + AK n, K n span(r 0, Ar 0,..., A n 1 r 0 ) is the n-th Krylov subspace generated by A and r 0. Minimum residual approach: minimizing the residual r n = b Ax n = r 0 d n r n = min r 0 d r n AK n. d AK n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 2

3 Minimum residual methods for Ax = b We consider a nonsingular linear system Ax = b, A R N N, b R N. Krylov subspace methods: initial guess x 0, compute {x n} such that x n x 0 + K n r n b Ax 0 r 0 + AK n, K n span(r 0, Ar 0,..., A n 1 r 0 ) is the n-th Krylov subspace generated by A and r 0. Minimum residual approach: minimizing the residual r n = b Ax n = r 0 d n r n = min r 0 d r n AK n. d AK n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 2

4 Minimum residual methods for Ax = b GMRES by Saad and Schultz [1986]: Arnoldi process: AQ n = Q n+1 H n+1,n, x n = x 0 + Q ny n, where y n solves min y ϱ 0 e 1 H n+1,n y, ϱ 0 r 0. Numerical stability of GMRES: Drkošová, Greenbaum, R, and Strakoš [1995], Arioli and Fassino [1996], Greenbaum, R, and Strakoš [1997], Paige, R, and Strakoš [2006]. Other implementations: Simpler GMRES: Walker and Zhou [1994], ORTHODIR, ORTHOMIN( ) GCR: Young and Jea [1980], Vinsome [1976], Eisenstat, Elman, and Schultz [1983]. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 3

5 Minimum residual methods for Ax = b GMRES by Saad and Schultz [1986]: Arnoldi process: AQ n = Q n+1 H n+1,n, x n = x 0 + Q ny n, where y n solves min y ϱ 0 e 1 H n+1,n y, ϱ 0 r 0. Numerical stability of GMRES: Drkošová, Greenbaum, R, and Strakoš [1995], Arioli and Fassino [1996], Greenbaum, R, and Strakoš [1997], Paige, R, and Strakoš [2006]. Other implementations: Simpler GMRES: Walker and Zhou [1994], ORTHODIR, ORTHOMIN( ) GCR: Young and Jea [1980], Vinsome [1976], Eisenstat, Elman, and Schultz [1983]. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 3

6 Sketch of Simpler GMRES and GCR Z n [z 1,..., z n] a (normalized) basis of K n. V n [v 1,..., v n] an orthonormal basis of AK n: AZ n = V nu n, U n is upper triangular. Residual r n r 0 + AK n = r 0 + R(V n), r n R(V n) satisfies r n = (I V nv T n )r 0 = (I v nv T n )r n 1 = r n 1 α nv n, α n v T n r n 1, whereas x n x 0 + K n = x 0 + R(Z n) satisfes x n = x 0 + Z nt n, U nt n = V T n r 0 = [α 1,..., α n] T. Approximate solution can be updated at each step P n [p 1,..., p n] = A 1 V n, Z n = P nu n x n = x n 1 + α np n. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 4

7 Sketch of Simpler GMRES and GCR Z n [z 1,..., z n] a (normalized) basis of K n. V n [v 1,..., v n] an orthonormal basis of AK n: AZ n = V nu n, U n is upper triangular. Residual r n r 0 + AK n = r 0 + R(V n), r n R(V n) satisfies r n = (I V nv T n )r 0 = (I v nv T n )r n 1 = r n 1 α nv n, α n v T n r n 1, whereas x n x 0 + K n = x 0 + R(Z n) satisfes x n = x 0 + Z nt n, U nt n = V T n r 0 = [α 1,..., α n] T. Approximate solution can be updated at each step P n [p 1,..., p n] = A 1 V n, Z n = P nu n x n = x n 1 + α np n. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 4

8 Sketch of Simpler GMRES and GCR Z n [z 1,..., z n] a (normalized) basis of K n. V n [v 1,..., v n] an orthonormal basis of AK n: AZ n = V nu n, U n is upper triangular. Residual r n r 0 + AK n = r 0 + R(V n), r n R(V n) satisfies r n = (I V nv T n )r 0 = (I v nv T n )r n 1 = r n 1 α nv n, α n v T n r n 1, whereas x n x 0 + K n = x 0 + R(Z n) satisfes x n = x 0 + Z nt n, U nt n = V T n r 0 = [α 1,..., α n] T. Approximate solution can be updated at each step P n [p 1,..., p n] = A 1 V n, Z n = P nu n x n = x n 1 + α np n. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 4

9 Sketch of Simpler GMRES and GCR Z n [z 1,..., z n] a (normalized) basis of K n. V n [v 1,..., v n] an orthonormal basis of AK n: AZ n = V nu n, U n is upper triangular. Residual r n r 0 + AK n = r 0 + R(V n), r n R(V n) satisfies r n = (I V nv T n )r 0 = (I v nv T n )r n 1 = r n 1 α nv n, α n v T n r n 1, whereas x n x 0 + K n = x 0 + R(Z n) satisfes x n = x 0 + Z nt n, U nt n = V T n r 0 = [α 1,..., α n] T. Approximate solution can be updated at each step P n [p 1,..., p n] = A 1 V n, Z n = P nu n x n = x n 1 + α np n. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 4

10 Backward and forward errors Assume stable computation of U n (e.g., using modified Gram-Schmidt orthogonalization or Householder reflections), c εκ(a)κ(z n) < 1, and r n 0. Solution of a triangular system ( b Ax n A x n + b c εκ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Updated approximate solutions ( b Ax n A x n + b c εκ(a)κ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 5

11 Backward and forward errors Assume stable computation of U n (e.g., using modified Gram-Schmidt orthogonalization or Householder reflections), c εκ(a)κ(z n) < 1, and r n 0. Solution of a triangular system ( b Ax n A x n + b c εκ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Updated approximate solutions ( b Ax n A x n + b c εκ(a)κ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 5

12 Backward and forward errors Assume stable computation of U n (e.g., using modified Gram-Schmidt orthogonalization or Householder reflections), c εκ(a)κ(z n) < 1, and r n 0. Solution of a triangular system ( b Ax n A x n + b c εκ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Updated approximate solutions ( b Ax n A x n + b c εκ(a)κ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 5

13 Choice of Z n 1. Z n = [r 0 / r 0, V n 1 ] (Simpler GMRES, ORTHODIR) Robust, but numerically less stable (Liesen, R, and Strakoš [2002]): r 0 r n 1 κ([ r 0 r 0, V n 1]) 2 r 0 r n Z n = R n, R n [r 0 / r 0,..., r n 1 / r n 1 ] (ORTHOMIN( ), GCR) Residuals are linearly independent iff the residual norms decrease, but R n is well conditioned: max k=1,...,n 1 ( rk r k 2 r k 1 2 r k 2 ) 1 ( ) 1 2 n 1 κ(rn) (n + 1) 1 r k r k r k=1 k 1 2 r k 2 3. Can we benefit from advantages of both bases without suffering from disadvantages? Adaptive choice of the direction z n: introduce ν [0, 1) and compute z n as r 0 / r 0 if n = 1, z n = r n 1 / r n 1 if n > 1 & r n 1 ν r n 2, v n 1 otherwise. PCR algorithm of Ramage and Wathen [1994]: hybridization of the fast ORTHOMIN and robust ORTHODIR Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 6

14 Choice of Z n 1. Z n = [r 0 / r 0, V n 1 ] (Simpler GMRES, ORTHODIR) Robust, but numerically less stable (Liesen, R, and Strakoš [2002]): r 0 r n 1 κ([ r 0 r 0, V n 1]) 2 r 0 r n Z n = R n, R n [r 0 / r 0,..., r n 1 / r n 1 ] (ORTHOMIN( ), GCR) Residuals are linearly independent iff the residual norms decrease, but R n is well conditioned: max k=1,...,n 1 ( rk r k 2 r k 1 2 r k 2 ) 1 ( ) 1 2 n 1 κ(rn) (n + 1) 1 r k r k r k=1 k 1 2 r k 2 3. Can we benefit from advantages of both bases without suffering from disadvantages? Adaptive choice of the direction z n: introduce ν [0, 1) and compute z n as r 0 / r 0 if n = 1, z n = r n 1 / r n 1 if n > 1 & r n 1 ν r n 2, v n 1 otherwise. PCR algorithm of Ramage and Wathen [1994]: hybridization of the fast ORTHOMIN and robust ORTHODIR Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 6

15 Choice of Z n 1. Z n = [r 0 / r 0, V n 1 ] (Simpler GMRES, ORTHODIR) Robust, but numerically less stable (Liesen, R, and Strakoš [2002]): r 0 r n 1 κ([ r 0 r 0, V n 1]) 2 r 0 r n Z n = R n, R n [r 0 / r 0,..., r n 1 / r n 1 ] (ORTHOMIN( ), GCR) Residuals are linearly independent iff the residual norms decrease, but R n is well conditioned: max k=1,...,n 1 ( rk r k 2 r k 1 2 r k 2 ) 1 ( ) 1 2 n 1 κ(rn) (n + 1) 1 r k r k r k=1 k 1 2 r k 2 3. Can we benefit from advantages of both bases without suffering from disadvantages? Adaptive choice of the direction z n: introduce ν [0, 1) and compute z n as r 0 / r 0 if n = 1, z n = r n 1 / r n 1 if n > 1 & r n 1 ν r n 2, v n 1 otherwise. PCR algorithm of Ramage and Wathen [1994]: hybridization of the fast ORTHOMIN and robust ORTHODIR Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 6

16 Conditioning of the adaptive Z n Assume a near stagnation in the initial phase and a fast convergence in the subsequent steps: Then where r k 1 > ν r k 2 for k = 2,..., q (Simpler GMRES basis), r k 1 ν r k 2 for k = q + 1,..., n (residual basis). { max 1, 1 } 2 γ r q 1 n,q r 0, r 0 r 0 γ 1 n,q κ(z n) 2γ r q 1 n,q r q 1, ( rk r k 2 ) 1 2 n 1 γ n,q max, γn,q k=q,...,n 1 r k 1 2 r k 2 (n q+1) 1 r k r k 2 + r k=q k 1 2 r k 2 Similar estimates for general case (for multiple switches between bases). Moreover, κ(z n) 2 2 ν q ν 1 ν. Quasi-optimal choice ν opt = ( 1 + n 2 1)/n leads to κ(z n) ν=νopt = O(n) Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 7

17 Conditioning of the adaptive Z n Assume a near stagnation in the initial phase and a fast convergence in the subsequent steps: Then where r k 1 > ν r k 2 for k = 2,..., q (Simpler GMRES basis), r k 1 ν r k 2 for k = q + 1,..., n (residual basis). { max 1, 1 } 2 γ r q 1 n,q r 0, r 0 r 0 γ 1 n,q κ(z n) 2γ r q 1 n,q r q 1, ( rk r k 2 ) 1 2 n 1 γ n,q max, γn,q k=q,...,n 1 r k 1 2 r k 2 (n q+1) 1 r k r k 2 + r k=q k 1 2 r k 2 Similar estimates for general case (for multiple switches between bases). Moreover, κ(z n) 2 2 ν q ν 1 ν. Quasi-optimal choice ν opt = ( 1 + n 2 1)/n leads to κ(z n) ν=νopt = O(n) Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 7

18 Conditioning of the adaptive Z n Assume a near stagnation in the initial phase and a fast convergence in the subsequent steps: Then where r k 1 > ν r k 2 for k = 2,..., q (Simpler GMRES basis), r k 1 ν r k 2 for k = q + 1,..., n (residual basis). { max 1, 1 } 2 γ r q 1 n,q r 0, r 0 r 0 γ 1 n,q κ(z n) 2γ r q 1 n,q r q 1, ( rk r k 2 ) 1 2 n 1 γ n,q max, γn,q k=q,...,n 1 r k 1 2 r k 2 (n q+1) 1 r k r k 2 + r k=q k 1 2 r k 2 Similar estimates for general case (for multiple switches between bases). Moreover, κ(z n) 2 2 ν q ν 1 ν. Quasi-optimal choice ν opt = ( 1 + n 2 1)/n leads to κ(z n) ν=νopt = O(n) Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 7

19 FS1836, b = A(1,..., 1) T : Simpler GMRES fails, adaptive wins backward errors ɛκ(basis) ɛκ(a basis) ɛκ(a) 10 0 backward error and condition numbers iteration number n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 8

20 FS1836, b = A(1,..., 1) T : Simpler GMRES fails, adaptive wins backward errors ɛκ(basis) ɛκ(a basis) ɛκ(a) 10 0 backward error and condition numbers iteration number n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 9

21 FS1836, b = u min : GCR fails, adaptive wins backward errors ɛκ(basis) ɛκ(a basis) ɛκ(a) 10 0 backward error and condition numbers iteration number n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 10

22 FS1836, b = u min : GCR fails, adaptive wins backward errors ɛκ(basis) ɛκ(a basis) ɛκ(a) 10 0 backward error and condition numbers iteration number n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 11

23 Dependence of κ(z n ) on ν condition number of the basis Z m SHERMAN2 E05R0000 UTM300 CAVITY03 PDE threshold parameter ν Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 12

24 Conclusions and references Simpler GMRES and ORTHODIR do not break down, but are potentially unstable in the case of the fast convergence. Simpler GMRES with residuals and ORTHOMIN( )/GCR can break down, but already moderate convergence implies good conditioning of the basis. Interpolation between both bases by adaptive selection of the new direction vector can benefit from advantages of both approaches and provides the stable variant of Simpler GMRES or GCR. References: P. Jiránek, R, M. H. Gutknecht, How to make Simpler GMRES and GCR more stable, SIAM J. Matrix Anal. Appl., 30 (2008), pp P. Jiránek, R, Adaptive version of Simpler GMRES, Numerical Algorithms, 53 (2010), pp Thank you for your attention! Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 13

25 Conclusions and references Simpler GMRES and ORTHODIR do not break down, but are potentially unstable in the case of the fast convergence. Simpler GMRES with residuals and ORTHOMIN( )/GCR can break down, but already moderate convergence implies good conditioning of the basis. Interpolation between both bases by adaptive selection of the new direction vector can benefit from advantages of both approaches and provides the stable variant of Simpler GMRES or GCR. References: P. Jiránek, R, M. H. Gutknecht, How to make Simpler GMRES and GCR more stable, SIAM J. Matrix Anal. Appl., 30 (2008), pp P. Jiránek, R, Adaptive version of Simpler GMRES, Numerical Algorithms, 53 (2010), pp Thank you for your attention! Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 13

26 Conclusions and references Simpler GMRES and ORTHODIR do not break down, but are potentially unstable in the case of the fast convergence. Simpler GMRES with residuals and ORTHOMIN( )/GCR can break down, but already moderate convergence implies good conditioning of the basis. Interpolation between both bases by adaptive selection of the new direction vector can benefit from advantages of both approaches and provides the stable variant of Simpler GMRES or GCR. References: P. Jiránek, R, M. H. Gutknecht, How to make Simpler GMRES and GCR more stable, SIAM J. Matrix Anal. Appl., 30 (2008), pp P. Jiránek, R, Adaptive version of Simpler GMRES, Numerical Algorithms, 53 (2010), pp Thank you for your attention! Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 13

27 References I M. Arioli and C. Fassino. Roundoff error analysis of algorithms based on Krylov subspace methods. BIT, 36(2): , J. Drkošová, A. Greenbaum, M. Rozložník, and Z. Strakoš. Numerical stability of GMRES. BIT, 35(3): , S. C. Eisenstat, H. C. Elman, and M. H. Schultz. Variational iterative methods for nonsymmetric systems of linear equations. SIAM J. Numer. Anal., 20(2): , A. Greenbaum, M. Rozložník, and Z. Strakoš. Numerical behaviour of the modified Gram-Schmidt GMRES implementation. BIT, 36(3): , J. Liesen, M. Rozložník, and Z. Strakoš. Least squares residuals and minimal residual methods. SIAM J. Sci. Stat. Comput., 23(5): , C. C. Paige, M. Rozložník, and Z. Strakoš. Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES. SIAM J. Matrix Anal. Appl., 28(1): , A. Ramage and A. J. Wathen. Iterative solution techniques for the Stokes and Navier-Stokes equations. Internat. J. Numer. Methods Fluids, 19(1):67 83, Y. Saad and M. H. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput., 7(3): , P. K. W. Vinsome. Orthomin, an iterative method for solving sparse sets of simultaneous linear equations. In Proceedings of the Fourth Symposium on Reservoir Simulation, pages Society of Petroleum Engineers of the American Institute of Mining, Metallurgical, and Petroleum Engineers, Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 14

28 References II H. F. Walker and L. Zhou. A simpler GMRES. Numer. Linear Algebra Appl., 1(6): , D. M. Young and K. C. Jea. Generalized conjugate-gradient acceleration of nonsymmetrizable iterative methods. Linear Algebra Appl., 34(1): , Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 15

HOW TO MAKE SIMPLER GMRES AND GCR MORE STABLE

HOW TO MAKE SIMPLER GMRES AND GCR MORE STABLE PAVEL JIRÁNEK, MIROSLAV ROZLOŽNÍK, AND MARTIN H. GUTKNECHT Abstract. In this paper we analyze the numerical behavior of several minimum residual methods, which