Preconditioned GMRES Revisited

Preconditioned GMRES Revisited Roland Herzog Kirk Soodhalter UBC (visiting) RICAM Linz Preconditioning Conference 2017 Vancouver August 01, 2017 Preconditioned GMRES Revisited Vancouver 1 / 32

Table of Contents 1 GMRES from First Principles (and a Hilbert Space Perspective) 2 How does it Compare With...? 3 Conclusions Preconditioned GMRES Revisited Vancouver 2 / 32

What is GMRES? Quoting Saad and Schultz (1986): We present an iterative method for solving linear systems, which has the property of minimizing at every step the norm of the residual vector over a Krylov subspace. The algorithm is derived from the Arnoldi process for constructing an l 2 -orthogonal basis of Krylov subspaces. Quoting Wikipedia: In mathematics, the generalized minimal residual method (usually abbreviated GMRES) is an iterative method for the numerical solution of a nonsymmetric system of linear equations. The method approximates the solution by the vector in a Krylov subspace with minimal residual. The Arnoldi iteration is used to nd this vector. Preconditioned GMRES Revisited Vancouver 2 / 32

Some References (Very Incomplete) contributing to the understanding of GMRES in various situations: Starke (1997) Chan, Chow, Saad and Yeung (1998) Chen, Kincaid and Young (1999) Klawonn (1998) Ernst (2000); Eiermann and Ernst (2001) Sarkis and Szyld (2007) Pestana and Wathen (2013) and in Hilbert space (but with A L(X )) in particular: Campbell, Ipsen, Kelley and Meyer (1996) Moret (1997) Calvetti, Lewis and Reichel (2002) Gasparo, Papini and Pasquali (2008) Preconditioned GMRES Revisited Vancouver 3 / 32

Purpose/Plan of This Talk re-derive GMRES with natural algorithmic ingredients draw inspirations from a Hilbert space setting (PDE problems) obtain 'canonical GMRES' locate GMRES variants in the literature in this framework to sort out my own personal lack of knowledge/confusion about these variants Hilbert space analysis matters ( recall talk by W. Zulehner for instance) even though in this talk everything is nite dimensional Preconditioned GMRES Revisited Vancouver 4 / 32

Problem Setting A x = b in R n A R n n (non-singular) A x = b in X A L(X, X ) (non-singular) X Hilbert space inner product (, ) M Preconditioned GMRES Revisited Vancouver 5 / 32

Problem Setting A x = b in R n A R n n (non-singular) R n also a Hilbert space inner product (, ) M (u, v) M = u M v A x = b in X A L(X, X ) (non-singular) X Hilbert space inner product (, ) M Preconditioned GMRES Revisited Vancouver 5 / 32

Problem Setting A x = b in R n A R n n (non-singular) R n also a Hilbert space inner product (, ) M (u, v) M = u M v A x = b in X A L(X, X ) (non-singular) X Hilbert space inner product (, ) M (u, v) M = u, M v X,X ( ) M X ; 1 ( M X ; M 1) M M L(X, X ) Riesz isomorphism Preconditioned GMRES Revisited Vancouver 5 / 32

Problem Setting A x = b in R n A R n n (non-singular) R n also a Hilbert space inner product (, ) M (u, v) M = u M v ( R n ) M ; 1 ( M R n ; M 1) M A x = b in X A L(X, X ) (non-singular) X Hilbert space inner product (, ) M (u, v) M = u, M v X,X ( ) M X ; 1 ( M X ; M 1) M M L(X, X ) Riesz isomorphism Preconditioned GMRES Revisited Vancouver 5 / 32

Problem Setting A x = b in R n A R n n (non-singular) R n also a Hilbert space A x = b in X A L(X, X ) (non-singular) X Hilbert space inner product (, ) M inner product (, ) M (u, v) M = u M v (u, v) M = u, M v X,X ( R n ) M ; 1 ( M R n ; M 1) M The inner product (, ) M in X, the inner product (, ) M 1 in X, the Riesz map M L(X, X ), and its inverse M 1 L(X, X ) ( ) M X ; 1 ( M X ; M 1) M uniquely dene each other. Preconditioned GMRES Revisited Vancouver 5 / 32

Problem Setting (primal) quantities in X iterates, errors (dual) quantities in X rhs, residuals A x = b in R n A R n n (non-singular) R n also a Hilbert space A x = b in X A L(X, X ) (non-singular) X Hilbert space inner product (, ) M inner product (, ) M (u, v) M = u M v (u, v) M = u, M v X,X ( R n ) M ; 1 ( M R n ; M 1) M The inner product (, ) M in X, the inner product (, ) M 1 in X, the Riesz map M L(X, X ), and its inverse M 1 L(X, X ) ( ) M X ; 1 ( M X ; M 1) M uniquely dene each other. Preconditioned GMRES Revisited Vancouver 5 / 32

Example: Stokes (Saddle-Point Problem) [ ] ( A B B 0 u p ) = V = H 1 0 (Ω; R3 ), Q = L 2 0 (Ω) ( ) f V 0 Q A u = a(u, ) V B u = b(u, ) Q B p = b(, p) V µ u : v dx Ω } {{ } a(u,v) p div v dx Ω }{{} b(v,p) div u q dx Ω }{{} b(u,q) = = 0 F v dx Ω } {{ } f, v Preconditioned GMRES Revisited Vancouver 6 / 32

Example: Oseen (Saddle-Point Problem) [ ] ( A+N B B 0 u p ) = V = H 1 0 (Ω; R3 ), Q = L 2 0 (Ω) µ u : v dx Ω } {{ } a(u,v) ( ) f V 0 Q + ( u U) v dx Ω } {{ } n(u,v) A u = a(u, ) V N u = n(u, ) V B u = b(u, ) Q B p = b(, p) V p div v dx Ω }{{} b(v,p) div u q dx Ω }{{} b(u,q) = = 0 F v dx Ω } {{ } f, v Preconditioned GMRES Revisited Vancouver 6 / 32

Ingredients of 'Canonical GMRES' The residual is always going to be r := b A x X. 1 Krylov subspaces Due to A L(X, X ) we cannot form K k (A ; r) := span { r, Ar, A 2 r,..., A k 1 r } Preconditioned GMRES Revisited Vancouver 7 / 32

Ingredients of 'Canonical GMRES' The residual is always going to be r := b A x X. 1 Krylov subspaces Due to A L(X, X ) we have to use K k (AP 1 ; r) := span { r, AP 1 r, (AP 1 ) 2 r,..., (AP 1 ) k 1 r } X where P L(X, X ) non-singular is a 'preconditioner' (required!). 2 Inner product (, ) W in X for Arnoldi Since K k (AP 1 ; r) X holds, orthonormality is dened w.r.t. W 1. Preconditioned GMRES Revisited Vancouver 7 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates an orthonormal basis of the Krylov subspace K k (A ; r 0 ): Input: Matrix A (or matrix-vector products), initial vector r 0 Output: ONB v 1, v 2,... { } with span v 1,..., v k = Kk (A ; r 0 ) r 0 1: Set v 1 := r 0, r 0 1/2 2: for k = 1, 2,... do 3: Set v k+1 := A v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := (v k+1, v j ) // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set h k+1,k := (v k+1, v k+1 ) 1/2 // = (v k+1, v k+1 ) 1/2 9: Set v k+1 := v k+1 h k+1,k 10: end for Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 0 1: Set v 1 := r 0, r 0 1/2 2: for k = 1, 2,... do 3: Set v k+1 := A v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := (v k+1, v j ) // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set h k+1,k := (v k+1, v k+1 ) 1/2 // = (v k+1, v k+1 ) 1/2 9: Set v k+1 := v k+1 h k+1,k 10: end for Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 1: Set z 1 := W 1 0 r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := A v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := (v k+1, v j ) // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set h k+1,k := (v k+1, v k+1 ) 1/2 // = (v k+1, v k+1 ) 1/2 9: Set v k+1 := v k+1 h k+1,k 10: end for Both v j and z j = W 1 v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 1: Set z 1 := W 1 0 r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := AP 1 v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := (v k+1, v j ) // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set h k+1,k := (v k+1, v k+1 ) 1/2 // = (v k+1, v k+1 ) 1/2 9: Set v k+1 := v k+1 h k+1,k 10: end for Both v j and z j = W 1 v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 1: Set z 1 := W 1 0 r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := AP 1 v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := v k+1, z j // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set h k+1,k := (v k+1, v k+1 ) 1/2 // = (v k+1, v k+1 ) 1/2 9: Set v k+1 := v k+1 h k+1,k 10: end for Both v j and z j = W 1 v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 1: Set z 1 := W 1 0 r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := AP 1 v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := v k+1, z j // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set z k+1 := W 1 v k+1 and h k+1,k := v k+1, z k+1 1/2 // = (v k+1, v k+1 ) 1/2 W 1 9: Set v k+1 := v k+1 h k+1,k 10: end for Both v j and z j = W 1 v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 1: Set z 1 := W 1 0 r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := AP 1 v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := v k+1, z j // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set z k+1 := W 1 v k+1 and h k+1,k := v k+1, z k+1 1/2 // = (v k+1, v k+1 ) 1/2 W 1 9: Set v k+1 := v k+1 and z k+1 := z k+1 h k+1,k h k+1,k 10: end for Both v j and z j = W 1 v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 8 / 32

The Arnoldi Process In matrix form the Arnoldi process reads v 2 = AP 1 v 1 h 2,1 v 1 h 1,1 where H is the upper Hessenberg matrix (v 1, AP 1 v 1 ) W 1 (v 1, AP 1 v 2 ) W 1 (v 2, AP 1 v 1 ) W 1 (v 2, AP 1 v 2 ) W 1 H = 0 (v 3, AP 1 v 2 ) W 1 R (k+1) k.. 0.. Preconditioned GMRES Revisited Vancouver 9 / 32

The Arnoldi Process In matrix form the Arnoldi process reads 0 v 3 = AP 1 v 1 v 2 v 1 v 2 H 2,2 h 3,2 where H is the upper Hessenberg matrix (v 1, AP 1 v 1 ) W 1 (v 1, AP 1 v 2 ) W 1 (v 2, AP 1 v 1 ) W 1 (v 2, AP 1 v 2 ) W 1 H = 0 (v 3, AP 1 v 2 ) W 1 R (k+1) k.. 0.. Preconditioned GMRES Revisited Vancouver 9 / 32

The Arnoldi Process In matrix form the Arnoldi process reads 0 v k+1 = AP 1 v 1 v k v 1 v k H k,k h k+1,k where H is the upper Hessenberg matrix (v 1, AP 1 v 1 ) W 1 (v 1, AP 1 v 2 ) W 1 (v 2, AP 1 v 1 ) W 1 (v 2, AP 1 v 2 ) W 1 H = 0 (v 3, AP 1 v 2 ) W 1 R (k+1) k.. 0.. Preconditioned GMRES Revisited Vancouver 9 / 32

The Arnoldi Process In matrix form the Arnoldi process reads 0 v k+1 = AP 1 v 1 v k v 1 v k H k,k h k+1,k or V k+1 H = AP 1 V k where H is the upper Hessenberg matrix (v 1, AP 1 v 1 ) W 1 (v 1, AP 1 v 2 ) W 1 (v 2, AP 1 v 1 ) W 1 (v 2, AP 1 v 2 ) W 1 H = 0 (v 3, AP 1 v 2 ) W 1 R (k+1) k.. 0.. Preconditioned GMRES Revisited Vancouver 9 / 32

Residual Norm Minimization By design, x k x 0 P 1 K k (AP 1 ; r 0 ) and therefore r k r 0 AP 1 K k (AP 1 ; r 0 ) = range AP 1 V k. Hence r k 2 M 1 = r 0 AP 1 V k y 2 y R k M 1 = r 0 V k+1 H y 2 due to AP 1 V M 1 k = V k+1 H = r 0 W 1 v 1 V k+1 H y 2 since v M 1 1 = r 0 / r 0 W 1 = ( V k+1 r0 W 1 e 1 H y ) 2 M 1 The Arnoldi-inner product W 1 is mathematically irrelevant and can be chosen for algorithmic convenience. Here we are led to choose W := M, which from now on we assume. Preconditioned GMRES Revisited Vancouver 10 / 32

Residual Norm Minimization Hence By design, x k x 0 P 1 K k (AP 1 ; r 0 ) and therefore r k r 0 AP 1 K k (AP 1 ; r 0 ) = range AP 1 V k. r k 2 M 1 = r 0 AP 1 V k y 2 M 1 y R k = r 0 V k+1 H y 2 M 1 due to AP 1 V k = V k+1 H = r 0 M 1 v 1 V k+1 H y 2 M 1 since v 1 = r 0 / r 0 M 1 = V k+1 ( r0 M 1 e 1 H y ) 2 M 1 = r 0 M 1 e 1 H y 2 2 by M 1 -orthonormality. The Arnoldi-inner product W 1 is mathematically irrelevant and can be chosen for algorithmic convenience. Here we are led to choose W := M, which from now on we assume. Preconditioned GMRES Revisited Vancouver 10 / 32

Short Recurrences Short recurrences occur when (v 1, AP 1 v 1 ) M 1 (v 1, AP 1 v 2 ) M 1 (v 2, AP 1 v 1 ) M 1 (v 2, AP 1 v 2 ) M 1 H k,k = 0 (v 3, AP 1 v 2 ) M 1 R k k. 0.. is symmetric, i.e., when M 1 AP 1 = P A M 1. (SRC) This means that AP 1 L(X ) is ( X, M 1) -Hilbert space self-adjoint. Preconditioned GMRES Revisited Vancouver 11 / 32

(SRC) and the Normal Equations We are going to refer to GMRES as just described by Recall GMRES ( A, P 1, M 1, b, x 0 ). M 1 AP 1 = P A M 1. (SRC) Given A and M, (SRC) always holds for the choice P 1 := H 1 A M 1, where H = H is non-singular. This is mathematically equivalent to running GMRES on the normal equations (modied problem) A M 1 Ax = A M 1 b with preconditioner H. In other words, GMRES ( A, H 1 A M 1, M 1, b, x 0 ) GMRES ( A M 1 A, H 1, A 1 MA, A M 1 b, x 0 ) Preconditioned GMRES Revisited Vancouver 12 / 32

(SRC) and the Normal Equations We are going to refer to GMRES as just described by Recall GMRES ( A, P 1, M 1, b, x 0 ). M 1 AP 1 = P A M 1. (SRC) Given A and M, (SRC) always holds for the choice P 1 := H 1 A M 1, where H = H is spd. This is mathematically equivalent to running GMRES on the normal equations (modied problem) A M 1 Ax = A M 1 b with preconditioner H. In other words, GMRES ( A, H 1 A M 1, M 1, b, x 0 ) GMRES ( A M 1 A, H 1, A 1 MA, A M 1 b, x 0 ) CG ( A M 1 A, H 1, A M 1 b, x 0 ). Preconditioned GMRES Revisited Vancouver 12 / 32

Convergence Analysis In the absence of any particular properties of (A, P, M) we can use the Jordan decomposition (with U, J C n n ) Then for any polynomial p, M 1/2 AP 1 M 1/2 = U J U 1. p(ap 1 ) = M 1/2 U p(j) U 1 M 1/2 holds and thus p(ap 1 ) r 0 2 M 1 = U p(j) U 1 M 1/2 r 0 2 2 σ 2 max( U p(j) U 1 ) r 0 2 M 1 = λ max ( p(ap 1 ) M 1 p(ap 1 ); M 1) r 0 2 M 1. Alternatively, using the Schur decomposition M 1/2 AP 1 M 1/2 = U R U H (with U, R C n n ) we arrive at p(ap 1 ) r 0 2 M 1 σ 2 max( p(r) ) r0 2 M 1 = λ max ( p(ap 1 ) M 1 p(ap 1 ); M 1) r 0 2 M 1. Preconditioned GMRES Revisited Vancouver 13 / 32

Convergence Analysis In the absence of any particular properties of (A, P, M) we can use the Jordan decomposition (with U, J C n n ) Then for any polynomial p, M 1/2 AP 1 M 1/2 = U J U 1. p(ap 1 ) = M 1/2 U p(j) U 1 M 1/2 holds and thus p(ap 1 ) r 0 2 M 1 = U p(j) U 1 M 1/2 r 0 2 2 σ 2 max( U p(j) U 1 ) r 0 2 M 1 = λ max ( p(ap 1 ) M 1 p(ap 1 ); M 1) r 0 2 M 1. This convergence estimate depends on the complete triplet (A, P, M). Preconditioned GMRES Revisited Vancouver 13 / 32

Normality Condition For AP 1 L(X ), (AP 1 ) L(X ) is its adjoint and (AP 1 ) = M (AP 1 ) M 1 is its Hilbert space adjoint w.r.t. ( X, M 1). Preconditioned GMRES Revisited Vancouver 14 / 32

Normality Condition For AP 1 L(X ), (AP 1 ) L(X ) is its adjoint and (AP 1 ) = M (AP 1 ) M 1 is its Hilbert space adjoint w.r.t. ( X, M 1). We have already encountered the (Hilbert space) self-adjointness condition AP 1 = (AP 1 ), responsible for short recursions: M 1 AP 1 = P A M 1. (SRC) Preconditioned GMRES Revisited Vancouver 14 / 32

Convergence Analysis Under Normality Suppose that (NC) holds. Then with U, D C n n unitary/diagonal, B = M 1/2 AP 1 M 1/2 = U D U H and for any polynomial p, p(ap 1 ) = M 1/2 U p(d) U H M 1/2 holds and thus p(ap 1 ) r 0 2 M 1 = U p(d) U H M 1/2 r 0 2 2 σ 2 max( U p(d) U H ) r 0 2 M 1 = [ max p(λ j ) ] 2 r0 2 j M 1, where λ j are the (complex; real under the (SRC)) eigenvalues of AP 1. Preconditioned GMRES Revisited Vancouver 15 / 32

Intermediate Summary We have re-derived 'canonical' GMRES from 'rst principles' and from a Hilbert space perspective, carefully distinguishing mapping properties and primal and dual quantities. Two choices arise (for the user): 1 preconditioner P L(X, X ) 2 inner product (, ) M 1 in X for residual minimization 3 inner product (, ) W 1 in X for Arnoldi (non-essential) For the latter we x wlog W := M for algorithmic convenience. One application of A, P 1 and M 1 each is required per iteration. (SRC) (NC) convergence est. in terms of spectrum of AP 1 Preconditioned GMRES Revisited Vancouver 16 / 32

Table of Contents 1 GMRES from First Principles (and a Hilbert Space Perspective) 2 How does it Compare With...? 3 Conclusions Preconditioned GMRES Revisited Vancouver 17 / 32

Dual vs. Primal Krylov Subspaces 'Canonical GMRES' is working with the (dual, residual-based) Krylov subspaces K k (AP 1 ; r 0 ) X generated by AP 1 L(X ). It turned out convenient to x wlog W := M. Equivalently, the algorithm can be written using the (primal) Krylov subspaces (see next two slides) P 1 K k (AP 1 ; r 0 ) = K k (P 1 A; P 1 r 0 ) X. 'Equivalently' means same iterates x k, same residuals r k = b Ax k, same residual norms r k M 1. Preconditioned GMRES Revisited Vancouver 17 / 32

Using Primal Krylov Subspaces The Arnoldi process generates a basis {U k } of K k (P 1 A; P 1 r 0 ) X with is orthonormal w.r.t. the inner product W (mathematically irrelevant). In matrix form the Arnoldi process reads P 1 AU k = U k+1 H, where H is the upper Hessenberg matrix (v 1, P 1 Av 1 ) W (v 1, P 1 Av 2 ) W (v 2, P 1 Av 1 ) W (v 2, P 1 Av 2 ) W H = 0 (v 3, P 1 Av 2 ) W R (k+1) k.. 0.. The short recursion condition becomes WP 1 A = (WP 1 A), which appears to be dierent from (SRC)... Preconditioned GMRES Revisited Vancouver 18 / 32

Using Primal Krylov Subspaces We still want to minimize r k 2 M 1 = r 0 AU k y 2 y R k M 1 = P 1 r 0 U k+1 H y 2 P 1 AU P M 1 P k = U k+1 H = P 1 r 0 W u 1 U k+1 H y 2 u P M 1 P 1 = P 1 r 0 / P 1 r 0 W = ( U k+1 P 1 r 0 W e 1 H y ) 2 P M 1 P Preconditioned GMRES Revisited Vancouver 19 / 32

Using Primal Krylov Subspaces We still want to minimize r k 2 M 1 = r 0 AU k y 2 y R k M 1 = P 1 r 0 U k+1 H y 2 P 1 AU P M 1 P k = U k+1 H = P 1 r 0 W u 1 U k+1 H y 2 u P M 1 P 1 = P 1 r 0 / P 1 r 0 W = ( U k+1 P 1 r 0 W e 1 H y ) 2 P M 1 P The Arnoldi-inner product W is mathematically irrelevant and can be chosen for algorithmic convenience. Here we are led to choose W := P M 1 P L(X, X ). Preconditioned GMRES Revisited Vancouver 19 / 32

Using Primal Krylov Subspaces We still want to minimize r k 2 M 1 = r 0 AU k y 2 M 1 y R k = P 1 r 0 U k+1 H y 2 P M 1 P P 1 AU k = U k+1 H = P 1 r 0 W u 1 U k+1 H y 2 P M 1 P u 1 = P 1 r 0 / P 1 r 0 W = U k+1 ( P 1 r 0 W e 1 H y ) 2 P M 1 P = r 0 M 1 e 1 H y 2 2 by W -orthonormality. The Arnoldi-inner product W is mathematically irrelevant and can be chosen for algorithmic convenience. Here we are led to choose W := P M 1 P L(X, X ). But then the implementations with primal/dual Krylov subspaces agree: u j = P 1 v j holds, the Hessenberg matrices are the same, and the short recursion conditions agree. So we continue with the original version, using dual Krylov subspaces. Preconditioned GMRES Revisited Vancouver 19 / 32

What if A L(X )? The residual r := b A x belongs to X now. 1 Krylov subspaces With A L(X ) we can form K k (A ; r) := span { r, Ar, A 2 r,..., A k 1 r } X Preconditioned GMRES Revisited Vancouver 20 / 32

What if A L(X )? The residual r := b A x belongs to X now. 1 Krylov subspaces With A L(X ) we may still want to use a preconditioner K k (AP 1 ; r) := span { r, AP 1 r, (AP 1 ) 2 r,..., (AP 1 ) k 1 r } X but P L(X ) is optional now. 2 Inner product (, ) W in X for Arnoldi Since K k (AP 1 ; r) X holds, orthonormality is dened w.r.t. W. Preconditioned GMRES Revisited Vancouver 20 / 32

The Arnoldi Process for A L(X, X ) The Arnoldi process (here displayed with modied GS) generates a W 1 -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W 1 (or matrix-vector products), initial vector r 0 X Output: W 1 -ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 1: Set z 1 := W 1 0 r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := AP 1 v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := v k+1, z j // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set z k+1 := W 1 v k+1 and h k+1,k := v k+1, z k+1 1/2 // = (v k+1, v k+1 ) 1/2 W 1 9: Set v k+1 := v k+1 and z k+1 := z k+1 h k+1,k h k+1,k 10: end for Both v j and z j = W 1 v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 21 / 32

The Arnoldi Process for A L(X ) The Arnoldi process (here displayed with modied GS) generates a W -orthonormal basis of the Krylov subspace K k (AP 1 ; r 0 ): Input: Matrix A, P 1, W (or matrix-vector products), initial vector r 0 X Output: W - ONB v 1, v 2,... X { } with span v 1,..., v k = Kk (AP 1 ; r 0 ) r 0 1: Set z 1 := W r 0 and v 1 := r 0, z 1 1/2 and z z 1 1 := r 0, z 1 1/2 2: for k = 1, 2,... do 3: Set v k+1 := AP 1 v k // new Krylov vector 4: for j = 1, 2,..., k do 5: Set h j,k := v k+1, z j // orthogonal. coecients 6: Set v k+1 := v k+1 h j,k v j 7: end for 8: Set z k+1 := W v k+1 and h k+1,k := v k+1, z k+1 1/2 // = (v k+1, v k+1 ) 1/2 W 9: Set v k+1 := v k+1 and z k+1 := z k+1 h k+1,k h k+1,k 10: end for Both v j and z j = W v j need to be stored due to MGS. Preconditioned GMRES Revisited Vancouver 21 / 32

Residual Minimization for A L(X, X ) By design, x k x 0 P 1 K k (AP 1 ; r 0 ) and therefore r k r 0 AP 1 K k (AP 1 ; r 0 ) = range AP 1 V k. r k 2 M 1 = r 0 AP 1 V k y 2 y R k M 1 = r 0 V k+1 H y 2 due to AP 1 V M 1 k = V k+1 H = r 0 M 1 v 1 V k+1 H y 2 since v M 1 1 = r 0 / r 0 M 1 = ( Vk+1 r0 M 1 e 1 H y ) 2 M 1 = r 0 M 1 e 1 H y 2 by M 1 -orthonormality. 2 Preconditioned GMRES Revisited Vancouver 22 / 32

Residual Minimization for A L(X ) By design, x k x 0 P 1 K k (AP 1 ; r 0 ) and therefore r k r 0 AP 1 K k (AP 1 ; r 0 ) = range AP 1 V k. r k 2 M = r 0 AP 1 V k y 2 M y R k = r 0 V k+1 H y 2 M due to AP 1 V k = V k+1 H = r 0 M v 1 V k+1 H y 2 M since v 1 = r 0 / r 0 M = Vk+1 ( r0 M e 1 H y ) 2 M = r 0 M e 1 H y 2 2 by M -orthonormality. The Arnoldi-inner product W is mathematically irrelevant and can be chosen for algorithmic convenience. Here we are led again to choose W := M, which from now on we assume. Preconditioned GMRES Revisited Vancouver 22 / 32

Short Rec./Norm. Conditions for A L(X, X ) Self-adjointness AP 1 = (AP 1 ) in ( X ; M 1) is responsible for short recursions: M 1 AP 1 = P A M 1. (SRC) The normality condition AP 1 (AP 1 ) = (AP 1 ) AP 1 reads AP 1 M (AP 1 ) M 1 = M (AP 1 ) M 1 AP 1. (NC) Preconditioned GMRES Revisited Vancouver 23 / 32

Short Rec./Norm. Conditions for A L(X ) Self-adjointness AP 1 = (AP 1 ) in ( ) X ; M is responsible for short recursions: M AP 1 = P A M. (SRC) The normality condition AP 1 (AP 1 ) = (AP 1 ) AP 1 reads AP 1 M 1 (AP 1 ) M = M 1 (AP 1 ) M AP 1. (NC) The convergence analysis carries over, mutatis mutandis. Preconditioned GMRES Revisited Vancouver 23 / 32

Overview of Canonical GMRES A L(X, X ) preconditioner P L(X, X ) inner product M L(X, X ) Krylov subspaces K k (AP 1 ; r 0 ) X Arnoldi orthonormality w.r.t. M 1, apply M 1 A L(X ) preconditioner P L(X ) inner product M L(X, X ) Krylov subspaces K k (AP 1 ; r 0 ) X Arnoldi orthonormality w.r.t. M, apply M residual minimization r M 1 residual minimization r M GMRES ( A, P 1, M 1, b, x 0 ). GMRES ( A, P 1, M, b, x 0 ). Preconditioned GMRES Revisited Vancouver 24 / 32

GMRES in the Literature I The very frequent 'right preconditioned' GMRES is motivated by considering the modied problem AP 1 u = b. The Arnoldi process for K k (AP 1 ; r 0 ) is typically carried out w.r.t. the Euclidean inner product and r 2 = b Ax 2 is minimized. A non-singular, P non-singular [see for instance (Saad, 2003, Ch. 9.3), Matlab, PETSc,... ] Preconditioned GMRES Revisited Vancouver 25 / 32

GMRES in the Literature I The very frequent 'right preconditioned' GMRES is motivated by considering the modied problem AP 1 u = b. The Arnoldi process for K k (AP 1 ; r 0 ) is typically carried out w.r.t. the Euclidean inner product and r 2 = b Ax 2 is minimized. This corresponds to canonical GMRES with A non-singular, P non-singular, W = M = id. U The user does not need to know whether A L(X, X ) or A L(X ) since M = M 1 = id. (This also saves one operation per iteration.) D The user cannot control which norm of the residual is being minimized. This appears to be a restriction, which cannot be compensated for by the preconditioner. Opportunities to satisfy (SRC) and (NC) are limited. [see for instance (Saad, 2003, Ch. 9.3), Matlab, PETSc,... ] Preconditioned GMRES Revisited Vancouver 25 / 32

GMRES in the Literature II A small number of papers consider Ax = b with A L(X, X ) where the preconditioner P := M (spd) is chosen. The Arnoldi process for K k (AP 1 ; r 0 ) is carried out w.r.t. the inner product W 1 = M 1. [see for instance (Starke, 1997, Sect. 3), (Chan et al., 1998, Alg. 2.3), (Ernst, 2000, Ch. 9.3),... ] Preconditioned GMRES Revisited Vancouver 26 / 32

GMRES in the Literature II A small number of papers consider Ax = b with A L(X, X ) where the preconditioner P := M (spd) is chosen. The Arnoldi process for K k (AP 1 ; r 0 ) is carried out w.r.t. the inner product W 1 = M 1. This corresponds to canonical GMRES with A non-singular, P = M, W = M. U P = M saves one operation per iteration. D Coupling the choice of residual norm and preconditioner appears to be a restriction compared to full 'canonical GMRES'. Opportunities to satisfy (SRC) which amounts to A = A and (NC) are limited. [see for instance (Starke, 1997, Sect. 3), (Chan et al., 1998, Alg. 2.3), (Ernst, 2000, Ch. 9.3),... ] Preconditioned GMRES Revisited Vancouver 26 / 32

GMRES in the Literature III Bramble & Pasciak derived a method (BPCG)here adjusted to t our settingfor self-adjoint, indenite problems with A L(X, X ) in saddle-point form [ A1 B A = ] B A 2 with A 1 0 (spd), A 2 0 (spsd). [Bramble and Pasciak (1988),... ] Preconditioned GMRES Revisited Vancouver 27 / 32

GMRES in the Literature III Bramble & Pasciak derived a method (BPCG)here adjusted to t our settingfor self-adjoint, indenite problems with A L(X, X ) in saddle-point form with a particular preconditioner P and inner product M: [ A1 B A = ] ] [Â1 B, P =, M = B A 2 0 id [ ] A1 Â1 0 0 id with A 1 0 (spd), A 2 0 (spsd). The (SRC) holds by construction. Moreover, under appropriate assumptions, the eigenvalues of AP 1 are not only real but positive. U One obtains a CG-like convergence result. D r M 1 = e AM 1 A is non-trivial to interpret. [Bramble and Pasciak (1988),... ] Preconditioned GMRES Revisited Vancouver 27 / 32

GMRES in the Literature IV Klawonn extended the analysis of Bramble & Pasciak and employed canonical GMRES for self-adjoint, indenite problems with A L(X, X ) in saddle-point form [ A1 B A = ] B t 2 A 2 with A 1, A 2 0 (spd). [see for instance Klawonn (1998) and also Simoncini (2004),... ] Preconditioned GMRES Revisited Vancouver 28 / 32

GMRES in the Literature IV Klawonn extended the analysis of Bramble & Pasciak and employed canonical GMRES for self-adjoint, indenite problems with A L(X, X ) in saddle-point form with a particular preconditioner P and inner product M: [ A1 B A = ] ] [Â1 B B t 2, P =, M = A 2 0 Â2 [ ] A 1 Â1 0 0 Â 2 with A 1, A 2 0 (spd). The (SRC) holds (although it does not seem to have been used in the numerical experiments). Moreover, under appropriate assumptions, the eigenvalues of AP 1 are not only real but positive. [see for instance Klawonn (1998) and also Simoncini (2004),... ] Preconditioned GMRES Revisited Vancouver 28 / 32

GMRES in the Literature IV Klawonn extended the analysis of Bramble & Pasciak and employed canonical GMRES for self-adjoint, indenite problems with A L(X, X ) in saddle-point form with a particular preconditioner P and inner product M: [ A1 B A = ] ] [Â1 B B t 2, P =, M = A 2 0 Â2 [ ] A 1 Â1 0 0 Â 2 with A 1, A 2 0 (spd). The (SRC) holds (although it does not seem to have been used in the numerical experiments). Moreover, under appropriate assumptions, the eigenvalues of AP 1 are not only real but positive. U One obtains a CG-like convergence result, robustly in t R. D r M 1 = e AM 1 A is non-trivial to interpret. [see for instance Klawonn (1998) and also Simoncini (2004),... ] Preconditioned GMRES Revisited Vancouver 28 / 32

GMRES in the Literature V The term 'preconditioned MINRES' seems to be reserved for the setting A = A, P spd, M = P. Certainly the (SRC) holds in this setting: M 1 AP 1 = P A M 1 (SRC) but it is also valid in many other situations with or without A = A. [Elman et al. (2014); Günnel et al. (2014)] Preconditioned GMRES Revisited Vancouver 29 / 32

GMRES in the Literature VI The very frequent 'left preconditioned' GMRES is motivated by considering the modied problem P 1 L Ax = P 1 L b. The Arnoldi process for K k (P 1 L A; P 1 L r 0) is typically carried out w.r.t. the Euclidean inner product and P 1 L r 2 = b Ax P is minimized. L P 1 L This cannot be modeled in canonical GMRES except as a modied problem: P 1 L A non-singular, P = id, W = M = id. U The user does not need to know whether A, P belong to L(X, X ) or L(X ) since the choice P = P 1 = id and M = M 1 = id holds. Moreover, this choice reduces the cost per iteration. D The user may not be able to factorize the desired residual metric. Not using P nor M appears to be a restriction, which cannot be compensated for by P L. Opportunities for (SRC) and (NC) are limited. [see for instance (Saad, 2003, Ch. 9.3), Matlab, PETSc,... ] Preconditioned GMRES Revisited Vancouver 30 / 32

GMRES in the Literature VII Pestana & Wathen also consider a 'left preconditioned' GMRES P 1 L Ax = P 1 L b. where A L(X ). Both A and the 'left preconditioner' P L L(X ) are assumed to be ( X, H ) -Hilbert space self-adjoint w.r.t. a reference inner product H L(X, X ): HA = A H and HP L = P L H. This corresponds to canonical GMRES with P 1 L A non-singular, P = id [(Pestana and Wathen, 2013, preprint version)] Preconditioned GMRES Revisited Vancouver 31 / 32

GMRES in the Literature VII Pestana & Wathen also consider a 'left preconditioned' GMRES P 1 L Ax = P 1 L b. where A L(X ). Both A and the 'left preconditioner' P L L(X ) are assumed to be ( ) X, H -Hilbert space self-adjoint w.r.t. a reference inner product H L(X, X ): HA = A H and HP L = PL H. The Arnoldi process for K k (P 1 L A; P 1 L r 0) is carried out w.r.t. the inner product W = HP L and P 1 L r HP L = b Ax P is minimized. This corresponds to canonical GMRES with L H P 1 L A non-singular, P = id, W = M = HP L. [(Pestana and Wathen, 2013, preprint version)] Preconditioned GMRES Revisited Vancouver 31 / 32

GMRES in the Literature VII Pestana & Wathen also consider a 'left preconditioned' GMRES P 1 L Ax = P 1 L b. where A L(X ). Both A and the 'left preconditioner' P L L(X ) are assumed to be ( ) X, H -Hilbert space self-adjoint w.r.t. a reference inner product H L(X, X ): HA = A H and HP L = PL H. The Arnoldi process for K k (P 1 L A; P 1 L r 0) is carried out w.r.t. the inner product W = HP L and P 1 L r HP L = b Ax P is minimized. This corresponds to canonical GMRES with P 1 L A non-singular, P = id, W = M = HP L. U The (SRC) holds by construction. D The requirements appear to constitute a restriction compared to full 'canonical GMRES'. [(Pestana and Wathen, 2013, preprint version)] Preconditioned GMRES Revisited Vancouver 31 / 32 L H

Table of Contents 1 GMRES from First Principles (and a Hilbert Space Perspective) 2 How does it Compare With...? 3 Conclusions Preconditioned GMRES Revisited Vancouver 32 / 32

Concluding Remarks We have re-derived 'canonical GMRES' from a Hilbert space perspective for Ax = b in X (and Ax = b in X ). It appears natural that the user 1 rst chooses the inner product (, ) M 1 in X for residual minimization, 2 then the preconditioner P L(X, X ), so that ideally (SRC) or at least (NC) holds. If (NC) is achieved, then P is responsible for fast convergence (eigenvalues of AP 1 ), while M is responsible for being able to observe a meaningful norm of the residual. Don't use Matlab or PETSc implementations of GMRES unless you are determined to measure (and stop on) the residual norm r 2, since they make the choice M = id for you. Should we say MINRES rather than GMRES when (SRC) hold? Preconditioned GMRES Revisited Vancouver 32 / 32

References I Bramble, J. and Pasciak, J. (1988). A preconditioning technique for indenite systems resulting from mixed approximations of elliptic problems. Mathematics of Computation 50: 117. Calvetti, D., Lewis, B. and Reichel, L. (2002). On the regularizing properties of the GMRES method. Numerische Mathematik 91: 605625, doi: 10.1007/s002110100339. Campbell, S. L., Ipsen, I. C. F., Kelley, C. T. and Meyer, C. D. (1996). GMRES and the minimal polynomial. BIT. Numerical Mathematics 36: 664675, doi: 10.1007/BF01733786. Chan, T. F., Chow, E., Saad, Y. and Yeung, M. C. (1998). Preserving symmetry in preconditioned Krylov subspace methods. SIAM Journal on Scientic Computing 20: 568581, doi: 10.1137/S1064827596311554. Chen, J.-Y., Kincaid, D. R. and Young, D. M. (1999). Generalizations and modications of the GMRES iterative method. Numerical Algorithms 21: 119146, doi: 10.1023/A:1019105328973, numerical methods for partial dierential equations (Marrakech, 1998). Eiermann, M. and Ernst, O. G. (2001). Geometric aspects of the theory of Krylov subspace methods. Acta Numerica 10: 251312, doi: 10.1017/S0962492901000046. Elman, H., Silvester, D. and Wathen, A. (2014). Finite elements and fast iterative solvers: with applications in incompressible uid dynamics. Numerical Mathematics and Scientic Computation. Oxford University Press, 2nd ed. Preconditioned GMRES Revisited Vancouver 33 / 32

References II Ernst, O. (2000). Minimal and orthogonal residual methods and their generalizations for solving linear operator equations. Habilitation Thesis, Faculty of Mathematics and Computer Sciences, TU Bergakademie Freiberg. Gasparo, M. G., Papini, A. and Pasquali, A. (2008). Some properties of GMRES in Hilbert spaces. Numerical Functional Analysis and Optimization. An International Journal 29: 12761285, doi: 10.1080/01630560802580786. Günnel, A., Herzog, R. and Sachs, E. (2014). A note on preconditioners and scalar products in Krylov subspace methods for self-adjoint problems in Hilbert space. Electronic Transactions on Numerical Analysis 41: 1320. Klawonn, A. (1998). Block-triangular preconditioners for saddle point problems with a penalty term. SIAM Journal on Scientic Computing 19: 172184 (electronic), doi: 10.1137/S1064827596303624, special issue on iterative methods (Copper Mountain, CO, 1996). Moret, I. (1997). A note on the superlinear convergence of GMRES. SIAM Journal on Numerical Analysis 34: 513516, doi: 10.1137/S0036142993259792. Pestana, J. and Wathen, A. J. (2013). On the choice of preconditioner for minimum residual methods for non-hermitian matrices. Journal of Computational and Applied Mathematics 249: 5768, doi: 10.1016/j.cam.2013.02.020. Saad, Y. (2003). Iterative Methods for Sparse Linear Systems. Philadelphia: SIAM, 2nd ed. Preconditioned GMRES Revisited Vancouver 34 / 32

References III Saad, Y. and Schultz, M. H. (1986). GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems. Society for Industrial and Applied Mathematics. Journal on Scientic and Statistical Computing 7: 856869, doi: 10.1137/0907058. Sarkis, M. and Szyld, D. B. (2007). Optimal left and right additive Schwarz preconditioning for minimal residual methods with Euclidean and energy norms. Computer Methods in Applied Mechanics and Engineering 196: 16121621, doi: 10.1016/j.cma.2006.03.027. Simoncini, V. (2004). Block triangular preconditioners for symmetric saddle-point problems. Applied Numerical Mathematics. An IMACS Journal 49: 6380, doi: 10.1016/j.apnum.2003.11.012. Starke, G. (1997). Field-of-values analysis of preconditioned iterative methods for nonsymmetric elliptic problems. Numerische Mathematik 78: 103117, doi: 10.1007/s002110050306. Preconditioned GMRES Revisited Vancouver 35 / 32