Numerical behavior of inexact linear solvers

Numerical behavior of inexact linear solvers Miro Rozložník joint results with Zhong-zhi Bai and Pavel Jiránek Institute of Computer Science, Czech Academy of Sciences, Prague, Czech Republic The fourth international conference on numerical algebra and scientific computing (NASC 2012), Dalian University of Technology, Dalian, China, October 20-24, 2012

Iterative solution of linear systems Iterative methods in exact arithmetic generate approximate solutions to the solution of Ax = b x 0, x 1,..., x n x with residual vectors r 0 = b Ax 0,..., r n = b Ax n 0 Iterative methods in finite precision arithmetic compute approximations x 0, x 1,..., x n and updated residual vectors r 0, r 1,..., r n which are usually close to (but different from) the true residuals b A x n

Two main questions and two main effects How good is the computed approximate solution x n? How many (extra) steps do we need to reach the same accuracy as one can get in the exact method? How well the computed vector r n approximates the (true) residual b A x n? Is there a limitation on the accuracy of the computed approximate solution? Two main effects of rounding errors: DELAY OF CONVERGENCE AND LIMIT ON THE FINAL (MAXIMUM ATTAINABLE) ACCURACY

Delay of convergence and maximum attainable accuracy

The concept of backward stability A backward stable algorithm eventually computes the exact answer to a nearby problem, i.e. the approximate solution x n satisfying (A + A n ) x n = b + b n A n / A O(u), b n / b O(u) The normwise backward error associated with the approximate solution x n satisfies b A x n b + A x O(u) n A forward stable algorithm eventually computes the approximate solution x n with the error that satisfies x n x O(u) A 1 A x Prager, Oettli, 1964; Rigal, Gaches, 1967 see also Higham, 2nd ed. 2002; Stewart, Sun, 1990; Meurant 1999

The level of maximum attainable accuracy We are looking for the difference between the updated r n and true residual b A x n (divided by b + A x n ) b A x n r n b + A x n? r n 0 = lim n b A x n b + A x n? In the optimal case the bound is of O(u); then we have a backward stable solution. The backward stability implies the forward stability. Higham 2002, Higham, Knight 1993, Greenbaum, R, Strakoš, 1997

Stationary iterative methods Ax = b, A = M N A: Mx k+1 = N x k + b B: x k+1 = x k + M 1 (b Ax k ) Inexact solution of systems with M: every computed solution y of My = z is interpreted as an exact solution of a system with perturbed data and relative perturbation bounded by parameter τ such that (M + M)y = z, M τ M, τk(m) 1

Accuracy of the computed approximate solution A: Mx k+1 = N x k + b ˆx k+1 x x τ M 1 ( M + N ) b Aˆx k+1 M + N τ b + A ˆx k+1 A B: x k+1 = x k + M 1 (b Ax k ) ˆx k+1 x x O(u) M 1 A ( 1 + max ) i=0,...,k{ ˆx i } x ( 1 + max i=0,...,k{ ˆx i } x ( 1 + max ) i=0,...,k{ ˆx i } x ( b Aˆx k+1 b + A ˆx k+1 O(u) 1 + max ) i=0,...,k{ ˆx i } x ) Higham, Knight 1993, Bai, R, 2012

Numerical experiments: small model example A = tridiag(1, 4, 1) R 100 100, b = A ones(100, 1), κ(a) = A A 1 = 5.9990 0.4998 2.9983 A = M N, M = D L, N = U

10 0 10 2 τ = 10 2 relative error norms 10 4 10 6 10 8 τ = 10 6 τ = 10 12 10 14 τ = O(u) 10 16 0 5 10 15 20 25 30 35 40 45 50 iteration number k

10 0 10 2 τ = 10 2 normwise backward error 10 4 10 6 10 8 10 12 τ = 10 6 τ = 10 14 τ = O(u) 10 16 0 5 10 15 20 25 30 35 40 45 50 iteration number k

10 0 10 5 relative error norms τ = O(u), τ=, τ=10 6, τ=10 2 10 15 0 5 10 15 20 25 30 35 40 45 50 iteration number k

10 0 10 2 normwise backward errors 10 4 10 6 10 8 10 12 τ = O(u), τ=, τ=10 6, τ=10 2 10 14 10 16 0 5 10 15 20 25 30 35 40 45 50 iteration number k

Two-step splitting iteration methods M 1x k+1/2 = N 1x k + b, A = M 1 N 1 M 2x k+1 = N 2x k+1/2 + b, A = M 2 N 2 Numerous solution schemes: Hermitian/skew-Hermitian (HSS) splitting, modified Hermitian/skew-Hermitian (MHSS) splitting, normal Hermitian/skew-Hermitian (NSS) splitting, preconditioned variant of modified Hermitian/skew-Hermitian (PMHSS) splitting and other splittings,... Bai, Golub, Ng 2003, 2007, 2008; Bai 2009 Bai, Benzi, Chen 2010, 2011; Bai, Benzi, Chen, Wang 2012 ˆx k+1 x x [ τ 1 M 1 2 N 2 M 1 1 ( M 1 + N 1 ) + τ 2 M 1 2 ( M 2 + N ] 2 ) ( 1 + max ) i=0,1/2,...,k+1/2{ ˆx i } x

Two-step splitting iteration methods x k+1/2 = x k + M 1 1 (b Ax k ) x k+1 = x k+1/2 + M 1 2 (b Ax k+1/2 ) x k+1 = x k + (M 1 1 + M 1 2 M 1 2 AM 1 1 )(b Ax k ) = x k + (I + M 1 2 N 1)M 1 1 (b Ax k ) = x k + M 1 2 (I + N 2M 1 1 )(b Ax k ) ˆx k+1 x x ( O(u) M 1 2 (I + N 2M 1 1 ) A 1 + max ) i=0,...,k{ ˆx i } x

Numerical experiments: small model example A = tridiag(2, 4, 1) R 100 100, b = A ones(100, 1), κ(a) = A A 1 = 5.9990 0.4998 2.9983 A = H + S, H = 1 2 (A + AT ), S = 1 2 (A AT ) H = tridiag( 3 2, 4, 3 2 ), S = tridiag( 1 2, 0, 1 2 )

10 0 relative error norms 10 2 10 4 10 6 10 8 10 12 τ 1 = 10 2, τ 2 = τ 1 = 10 6, τ 2 = τ 1 =, τ 2 = τ 1 = O(u), τ 2 = 10 14 τ 1 =O(u),, 10 6, 10 2, τ 2 = 10 16 0 10 20 30 40 50 60 70 80 90 100 iteration number k

10 0 10 2 relative error norms 10 4 10 6 10 8 10 12 τ 1 =, τ 2 =10 2 τ 1 =, τ 2 =10 6 τ 1 =, τ 2 = τ 1 =, τ 2 =O(u) 10 14 τ 1 =, τ 2 =O(u),, 10 6, 10 2 10 16 0 10 20 30 40 50 60 70 80 90 100 iteration number k

Saddle point problems We consider a saddle point problem with the symmetric 2 2 block form ( ) ( ) ( ) A B x f B T =. 0 y 0 A is a square n n nonsingular (symmetric positive definite) matrix, B is a rectangular n m matrix of (full column) rank m. Applications: mixed finite element approximations, weighted least squares, constrained optimization, computational fluid dynamics, electromagnetism etc. [Benzi, Golub and Liesen, 2005], [Elman, Silvester, Wathen, 2005]. For the updated list of applications leading to saddle point problems contact [Benzi].

Iterative solution of saddle point problems Numerous solution schemes: inexact Uzawa algorithms, inexact null-space methods, inner-outer iteration methods, two-stage iteration processes, multilevel or multigrid methods, domain decomposition methods Numerous preconditioning techniques and schemes: block diagonal preconditioners, block triangular preconditioners, constraint preconditioning, Hermitian/skew-Hermitian preconditioning and other splittings, combination preconditioning Numerous iterative solvers: conjugate gradient (CG) method, MINRES, GMRES, flexible GMRES, GCR, BiCG, BiCGSTAB,...

Schur complement reduction method Compute y as a solution of the Schur complement system B T A 1 By = B T A 1 f, compute x as a solution of Ax = f By. Segregated vs. coupled approach: x k and y k approximate solutions to x and y, respectively. Inexact solution of systems with A: every computed solution û of Au = b is interpreted as an exact solution of a perturbed system (A + A)û = b + b, A τ A, b τ b, τκ(a) 1.

Iterative solution of the Schur complement system choose y 0, solve Ax 0 = f By 0 compute α k and p (y) k y k+1 = y k + α k p (y) k solve Ap (x) k = Bp (y) k back-substitution: A: x k+1 = x k + α k p (x) k, B: solve Ax k+1 = f By k+1, C: solve Au k = f Ax k By k+1, x k+1 = x k + u k. r (y) k+1 = r(y) k α k B T p (x) k inner iteration outer iteration

Accuracy in the saddle point system f Ax k By k O(α1)κ(A) 1 τκ(a) ( f + B Y k), B T x k r (y) k O(α2)κ(A) 1 τκ(a) A 1 B ( f + B Y k ), Y k max{ y i i = 0, 1,..., k}. Back-substitution scheme α 1 α 2 A: Generic update x k+1 = x k + α k p (x) τ u k B: Direct substitution τ τ x k+1 = A 1 (f By k+1 ) C: Corrected dir. subst. u τ x k+1 = x k + A 1 (f Ax k By k+1 ) } additional system with A B T A 1 f + B T A 1 By k = B T x k B T A 1 (f Ax k By k )

Numerical experiments: a small model example A = tridiag(1, 4, 1) R 100 100, B = rand(100, 20), f = rand(100, 1), κ(a) = A A 1 = 5.9990 0.4998 2.9983, κ(b) = B B = 7.1695 0.4603 3.3001. [R, Simoncini, 2002]

Generic update: x k+1 = x k + α k p (x) k 10 0 τ = 10 2 10 0 10 2 10 2 residual norm f Ax k By k 10 4 10 6 10 8 10 12 10 14 τ = 10 6 τ = τ = O(u) relative residual norms B T x k / B T x 0, r k (y) / r0 (y) 10 4 10 6 10 8 10 12 10 14 τ = O(u), τ =, τ=10 6, τ=10 2 10 16 10 16 10 18 0 50 100 150 200 250 300 iteration number k 10 18 0 50 100 150 200 250 300 iteration number k

Direct substitution: x k+1 = A 1 (f By k+1 ) 10 0 τ = 10 2 10 0 10 2 10 2 τ = 10 2 residual norm f Ax k By k 10 4 10 6 10 8 10 12 10 14 τ = 10 6 τ = τ = O(u) relative residual norms B T x k / B T x 0, r k (y) / r0 (y) 10 4 10 6 10 8 10 12 10 14 τ = 10 6 τ = τ = O(u) 10 16 10 16 10 18 0 50 100 150 200 250 300 iteration number k 10 18 0 50 100 150 200 250 300 iteration number k

Corrected direct substitution: x k+1 = x k + A 1 (f Ax k By k+1 ) 10 0 10 0 10 2 10 2 τ = 10 2 residual norm f Ax k By k 10 4 10 6 10 8 10 12 10 14 τ = O(u), τ =, τ=10 6, τ=10 2 relative residual norms B T x k / B T x 0, r k (y) / r0 (y) 10 4 10 6 10 8 10 12 10 14 τ = 10 6 τ = τ = O(u) 10 16 10 16 10 18 0 50 100 150 200 250 300 iteration number k 10 18 0 50 100 150 200 250 300 iteration number k

Null-space projection method compute x N(B T ) as a solution of the projected system (I Π)A(I Π)x = (I Π)f, compute y as a solution of the least squares problem Π = B(B T B) 1 B T By f Ax, is the orthogonal projector onto R(B). Schemes with the inexact solution of least squares with B. Every computed approximate solution v of a least squares problem Bv c is interpreted as an exact solution of a perturbed least squares (B + B) v c + c, B τ B, c τ c, τκ(b) 1.

Null-space projection method choose x 0, solve By 0 f Ax 0 compute α k and p (x) k N(B T ) x k+1 = x k + α k p (x) k solve Bp (y) k r (x) k α k Ap (x) k back-substitution: A: y k+1 = y k + p (y) k, B: solve By k+1 f Ax k+1, C: solve Bv k f Ax k+1 By k, y k+1 = y k + v k. r (x) k+1 = r(x) k α k Ap (x) k Bp (y) k inner iteration outer iteration

Accuracy in the saddle point system f Ax k By k r (x) k O(α3)κ(B) 1 τκ(b) ( f + A X k), B T x k O(τ)κ(B) 1 τκ(b) B X k, X k max{ x i i = 0, 1,..., k}. Back-substitution scheme α 3 A: Generic update y k+1 = y k + p (y) u k B: Direct substitution τ y k+1 = B (f Ax k+1 ) C: Corrected dir. subst. u y k+1 = y k + B (f Ax k+1 By k ) } additional least square with B

Generic update: y k+1 = y k + p (y) k 10 0 10 0 τ = 10 2 relative residual norms f Ax k By k / f Ax 0 By 0, r (x) k / r(x) 0 10 2 10 4 10 6 10 8 10 12 10 14 10 16 τ = O(u), τ =, τ=10 6, τ=10 2 residual norm B T x k 10 2 10 4 10 6 10 8 10 12 10 14 10 16 τ = 10 6 τ = τ = O(u) 10 18 0 10 20 30 40 50 60 70 80 90 100 iteration number 10 18 0 10 20 30 40 50 60 70 80 90 100 iteration number

Direct substitution: y k+1 = B (f Ax k+1 ) 10 0 τ = 10 2 10 0 τ = 10 2 relative residual norms f Ax k By k / f Ax 0 By 0, r (x) k / r(x) 0 10 2 10 4 10 6 10 8 10 12 10 14 10 16 τ = 10 6 τ = τ = O(u) residual norm B T x k 10 2 10 4 10 6 10 8 10 12 10 14 10 16 τ = 10 6 τ = τ = O(u) 10 18 0 10 20 30 40 50 60 70 80 90 100 iteration number 10 18 0 10 20 30 40 50 60 70 80 90 100 iteration number

Corrected direct substitution: y k+1 = y k + B (f Ax k+1 By k ) 10 0 10 0 τ = 10 2 relative residual norms f Ax k By k / f Ax 0 By 0, r (x) k / r(x) 0 10 2 10 4 10 6 10 8 10 12 10 14 10 16 τ = O(u), τ =, τ=10 6, τ=10 2 residual norm B T x k 10 2 10 4 10 6 10 8 10 12 10 14 10 16 τ = 10 6 τ = τ = O(u) 10 18 0 10 20 30 40 50 60 70 80 90 100 iteration number 10 18 0 10 20 30 40 50 60 70 80 90 100 iteration number

The maximum attainable accuracy of saddle point solvers The accuracy measured by the residuals of the saddle point problem depends on the choice of the back-substitution scheme [Jiránek, R, 2008]. The schemes with (generic or corrected substitution) updates deliver approximate solutions which satisfy either the first or second block equation to working accuracy. Care must be taken when solving nonsymmetric systems [Jiránek, R, 2008], all bounds of the limiting accuracy depend on the maximum norm of computed iterates, cf. [Greenbaum 1994,1997], [Sleijpen, et al. 1994]. relative residual norms B T A 1 f B T A 1 By / B T A 1 f and r (y) k k / BT A 1 f 10 5 10 0 10 5 10 15 0 50 100 150 200 250 300 350 400 450 iteration number k

Conclusions new value = old value + small correction Stationary iterative methods for Ax = b and their maximum attainable accuracy [Higham and Knight, 1993]: assuming splitting A = M N and inexact solution of systems with M, use x new = x old + M 1 (b Ax old ) rather than x new = M 1 (Nx old + b), [Higham, 2002; Bai, R]. Two-step splitting iteration framework: A = M 1 N 1 = M 2 N 2 assuming inexact solution of systems with M 1 and M 2, reformulation of M 1x 1/2 = N 1x old + b, M 2x new = N 2x 1/2 + b, Hermitian/skew-Hermitian splitting (HSS) iteration [Bai, Golub and Ng 2003; Bai, R]. Saddle point problems and inexact linear solvers: Schur complement and null-space approach [Jiránek, R 2008] Preconditioners for saddle point problems: SIMPLE and SIMPLE(R) type algorithms [Vuik and Saghir, 2002] and constraint preconditioners [R, Simoncini, 2002]. Fixed-precision iterative refinement for improving the computed solution x old to a system Ax = b: solving update equations Az corr = r that have residual r = b Ay old as a right-hand side to obtain x new = x old + z corr, see [Wilkinson, 1963], [Higham, 2002].

Thank you for your attention. http://www.cs.cas.cz/ miro Zhong-Zhi Bai and M. Rozložník, On the behavior of two-step splitting iteration methods, in preparation. P. Jiránek and M. Rozložník. Maximum attainable accuracy of inexact saddle point solvers. SIAM J. Matrix Anal. Appl., 29(4):1297 1321, 2008. P. Jiránek and M. Rozložník. Limiting accuracy of segregated solution methods for nonsymmetric saddle point problems. J. Comput. Appl. Math. 215 (2008), pp. 28-37. M. Rozložník and V. Simoncini, Krylov subspace methods for saddle point problems with indefinite preconditioning, SIAM J. Matrix Anal. Appl., 24 (2002), pp. 368 391.