A stable variant of Simpler GMRES and GCR

A stable variant of Simpler GMRES and GCR Miroslav Rozložník joint work with Pavel Jiránek and Martin H. Gutknecht Institute of Computer Science, Czech Academy of Sciences, Prague, Czech Republic miro@cs.cas.cz, http://www.cs.cas.cz/ miro Applied Linear Algebra Conference in honor of Hans Schneider University of Novi Sad, Serbia, May 24 28, 2010. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 1

Minimum residual methods for Ax = b We consider a nonsingular linear system Ax = b, A R N N, b R N. Krylov subspace methods: initial guess x 0, compute {x n} such that x n x 0 + K n r n b Ax 0 r 0 + AK n, K n span(r 0, Ar 0,..., A n 1 r 0 ) is the n-th Krylov subspace generated by A and r 0. Minimum residual approach: minimizing the residual r n = b Ax n = r 0 d n r n = min r 0 d r n AK n. d AK n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 2

Minimum residual methods for Ax = b GMRES by Saad and Schultz [1986]: Arnoldi process: AQ n = Q n+1 H n+1,n, x n = x 0 + Q ny n, where y n solves min y ϱ 0 e 1 H n+1,n y, ϱ 0 r 0. Numerical stability of GMRES: Drkošová, Greenbaum, R, and Strakoš [1995], Arioli and Fassino [1996], Greenbaum, R, and Strakoš [1997], Paige, R, and Strakoš [2006]. Other implementations: Simpler GMRES: Walker and Zhou [1994], ORTHODIR, ORTHOMIN( ) GCR: Young and Jea [1980], Vinsome [1976], Eisenstat, Elman, and Schultz [1983]. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 3

Sketch of Simpler GMRES and GCR Z n [z 1,..., z n] a (normalized) basis of K n. V n [v 1,..., v n] an orthonormal basis of AK n: AZ n = V nu n, U n is upper triangular. Residual r n r 0 + AK n = r 0 + R(V n), r n R(V n) satisfies r n = (I V nv T n )r 0 = (I v nv T n )r n 1 = r n 1 α nv n, α n v T n r n 1, whereas x n x 0 + K n = x 0 + R(Z n) satisfes x n = x 0 + Z nt n, U nt n = V T n r 0 = [α 1,..., α n] T. Approximate solution can be updated at each step P n [p 1,..., p n] = A 1 V n, Z n = P nu n x n = x n 1 + α np n. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 4

Backward and forward errors Assume stable computation of U n (e.g., using modified Gram-Schmidt orthogonalization or Householder reflections), c εκ(a)κ(z n) < 1, and r n 0. Solution of a triangular system ( b Ax n A x n + b c εκ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Updated approximate solutions ( b Ax n A x n + b c εκ(a)κ(zn) 1 + x ) 0, x n x x n x n c εκ(a)κ(zn) ( 1 + x ) 0. 1 c εκ(a)κ(z n) x n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 5

Choice of Z n 1. Z n = [r 0 / r 0, V n 1 ] (Simpler GMRES, ORTHODIR) Robust, but numerically less stable (Liesen, R, and Strakoš [2002]): r 0 r n 1 κ([ r 0 r 0, V n 1]) 2 r 0 r n 1. 2. Z n = R n, R n [r 0 / r 0,..., r n 1 / r n 1 ] (ORTHOMIN( ), GCR) Residuals are linearly independent iff the residual norms decrease, but R n is well conditioned: max k=1,...,n 1 ( rk 1 2 + r k 2 r k 1 2 r k 2 ) 1 ( ) 1 2 n 1 κ(rn) (n + 1) 1 r k 1 2 + r k 2 2 2 1 + r k=1 k 1 2 r k 2 3. Can we benefit from advantages of both bases without suffering from disadvantages? Adaptive choice of the direction z n: introduce ν [0, 1) and compute z n as r 0 / r 0 if n = 1, z n = r n 1 / r n 1 if n > 1 & r n 1 ν r n 2, v n 1 otherwise. PCR algorithm of Ramage and Wathen [1994]: hybridization of the fast ORTHOMIN and robust ORTHODIR Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 6

Conditioning of the adaptive Z n Assume a near stagnation in the initial phase and a fast convergence in the subsequent steps: Then where r k 1 > ν r k 2 for k = 2,..., q (Simpler GMRES basis), r k 1 ν r k 2 for k = q + 1,..., n (residual basis). { max 1, 1 } 2 γ r q 1 n,q r 0, r 0 r 0 γ 1 n,q κ(z n) 2γ r q 1 n,q r q 1, ( rk 1 2 + r k 2 ) 1 2 n 1 γ n,q max, γn,q k=q,...,n 1 r k 1 2 r k 2 (n q+1) 1 r k 1 2 1 2 + r k 2 + r k=q k 1 2 r k 2 Similar estimates for general case (for multiple switches between bases). Moreover, κ(z n) 2 2 ν q 1 1 + ν 1 ν. Quasi-optimal choice ν opt = ( 1 + n 2 1)/n leads to κ(z n) ν=νopt = O(n). 1 2. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 7

FS1836, b = A(1,..., 1) T : Simpler GMRES fails, adaptive wins backward errors ɛκ(basis) ɛκ(a basis) ɛκ(a) 10 0 backward error and condition numbers 10 5 10 10 10 15 5 10 15 20 25 30 35 40 45 50 55 60 iteration number n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 8

FS1836, b = u min : GCR fails, adaptive wins backward errors ɛκ(basis) ɛκ(a basis) ɛκ(a) 10 0 backward error and condition numbers 10 5 10 10 10 15 5 10 15 20 25 30 35 40 45 50 55 60 iteration number n Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 10

Dependence of κ(z n ) on ν condition number of the basis Z m 10 18 10 16 10 14 10 12 10 10 10 8 10 6 SHERMAN2 E05R0000 UTM300 CAVITY03 PDE225 10 4 10 2 10 0 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 threshold parameter ν Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 12

Conclusions and references Simpler GMRES and ORTHODIR do not break down, but are potentially unstable in the case of the fast convergence. Simpler GMRES with residuals and ORTHOMIN( )/GCR can break down, but already moderate convergence implies good conditioning of the basis. Interpolation between both bases by adaptive selection of the new direction vector can benefit from advantages of both approaches and provides the stable variant of Simpler GMRES or GCR. References: P. Jiránek, R, M. H. Gutknecht, How to make Simpler GMRES and GCR more stable, SIAM J. Matrix Anal. Appl., 30 (2008), pp. 1483 1499. P. Jiránek, R, Adaptive version of Simpler GMRES, Numerical Algorithms, 53 (2010), pp. 93-112. Thank you for your attention! Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 13

References I M. Arioli and C. Fassino. Roundoff error analysis of algorithms based on Krylov subspace methods. BIT, 36(2):189 205, 1996. J. Drkošová, A. Greenbaum, M. Rozložník, and Z. Strakoš. Numerical stability of GMRES. BIT, 35(3):309 330, 1995. S. C. Eisenstat, H. C. Elman, and M. H. Schultz. Variational iterative methods for nonsymmetric systems of linear equations. SIAM J. Numer. Anal., 20(2):345 357, 1983. A. Greenbaum, M. Rozložník, and Z. Strakoš. Numerical behaviour of the modified Gram-Schmidt GMRES implementation. BIT, 36(3):706 719, 1997. J. Liesen, M. Rozložník, and Z. Strakoš. Least squares residuals and minimal residual methods. SIAM J. Sci. Stat. Comput., 23(5):1503 1525, 2002. C. C. Paige, M. Rozložník, and Z. Strakoš. Modified Gram-Schmidt (MGS), least squares, and backward stability of MGS-GMRES. SIAM J. Matrix Anal. Appl., 28(1):264 284, 2006. A. Ramage and A. J. Wathen. Iterative solution techniques for the Stokes and Navier-Stokes equations. Internat. J. Numer. Methods Fluids, 19(1):67 83, 1994. Y. Saad and M. H. Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM J. Sci. Stat. Comput., 7(3):856 869, 1986. P. K. W. Vinsome. Orthomin, an iterative method for solving sparse sets of simultaneous linear equations. In Proceedings of the Fourth Symposium on Reservoir Simulation, pages 149 159. Society of Petroleum Engineers of the American Institute of Mining, Metallurgical, and Petroleum Engineers, 1976. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 14

References II H. F. Walker and L. Zhou. A simpler GMRES. Numer. Linear Algebra Appl., 1(6):571 581, 1994. D. M. Young and K. C. Jea. Generalized conjugate-gradient acceleration of nonsymmetrizable iterative methods. Linear Algebra Appl., 34(1):159 194, 1980. Miro Rozložník (Prague) A stable variant of Simpler GMRES and GCR ALA10 15