GMRES: Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems Tsung-Ming Huang Department of Mathematics National Taiwan Normal University December 4, 2011 T.-M. Huang (NTNU) GMRES December 4, 2011 1 / 26
Ref: SISC, 1984, Saad Theorem 1 (Implicit Q theorem) Let AV 1 = V 1 H 1 and AV 2 = V 2 H 2, where H 1, H 2 are Hessenberg and V 1, V 2 are unitary with V 1 e 1 = V 2 e 1 = q 1. Then V 1 = V 2 and H 1 = H 2. Proof A [ ] [ ] v 1 v 2 v n = v1 v 2 v n h 11 h 12 h 1n h 21 h 22 h 2n....... h n,n 1 h nn with v T i v j = δ ij, i, j = 1,..., n T.-M. Huang (NTNU) GMRES December 4, 2011 2 / 26
Arnoldi Algorithm Input: Given v 1 with v 1 2 = 1; Output: Arnoldi factorization: AV k = V k H k + h k+1,k v k+1 e T k. 1: Set k = 0. 2: repeat 3: Compute h ik = (Av k, v i ) for i = 1, 2,..., k; 4: Compute ṽ k+1 = Av k k i=1 h ikv i ; 5: Compute h k+1,k = ṽ k+1 2 ; 6: Compute v k+1 = ṽ k+1 /h k+1,k ; 7: Set k = k + 1; 8: until convergent T.-M. Huang (NTNU) GMRES December 4, 2011 3 / 26
Remark 1 (a) Let V k = [v 1,, v k ] R n k where v j, for j = 1,..., k, is generated by Arnoldi algorithm. Then H k V T k AV k is upper k k Hessenberg. (b) Arnoldi s original method was a Galerkin method for approximate the eigenvalue of A by H k. T.-M. Huang (NTNU) GMRES December 4, 2011 4 / 26
In order to solve Ax = b by the Galerkin method using < K k > < V k >, we seek an approximate solution x k = x 0 + z k with and r 0 = b Ax 0. Definition 2 z k K k =< r 0, Ar 0,, A k 1 r 0 > {x k } is said to be satisfied the Galerkin condition if r k b Ax k is orthogonal to K k for each k. The Galerkin method can be stated as that find such that x k = x 0 + z k with z k V k (1) (b Ax k, v) = 0, v V k, T.-M. Huang (NTNU) GMRES December 4, 2011 5 / 26
which is equivalent to find z k V k y k V k (2) such that Substituting (2) into (3), we get which implies that (r 0 Az k, v) = 0, v V k. (3) V T k (r 0 AV k y k ) = 0, y k = (V T k AV k) 1 r 0 e 1. (4) T.-M. Huang (NTNU) GMRES December 4, 2011 6 / 26
Since V k is computed by the Arnoldi algorithm with v 1 = r 0 / r 0, y k in (4) can be represented as y k = H 1 k r 0 e 1. Substituting it into (2) and (1), we get x k = x 0 + V k H 1 k r 0 e 1. Using the result that AV k = V k H k + h k+1,k v k+1 e T k, r k can be reformulated as r k = b Ax k = r 0 AV k y k = r 0 (V k H k + h k+1,k v k+1 e T k )y k = r 0 V k r 0 e 1 h k+1,k e T k y kv k+1 = (h k+1,k e T k y k)v k+1. T.-M. Huang (NTNU) GMRES December 4, 2011 7 / 26
The generalized minimal residual (GMRES) algorithm The approximate solution of the form x 0 + z k, which minimizes the residual norm over z k K k, can in principle be obtained by following algorithms: The ORTHODIR algorithm of Jea and Young; the generalized conjugate residual method (GCR); GMRES. Let h 1,1 h 1,2 h 1,k h 2,1 h 2,2 h 2,k V k = [v 1,, v k ], Hk =. 0...... R (k+1) k..... hk,k 1 h k,k 0 0 h k+1,k T.-M. Huang (NTNU) GMRES December 4, 2011 8 / 26
By Arnoldi algorithm, we have To solve the least square problem: AV k = V k+1 Hk. (5) min z K k r o Az 2 = min z K k b A(x o + z) 2, (6) where K k =< r o, Ar o,, A k 1 r o >=< v 1,, v k > with v 1 = ro r o 2. T.-M. Huang (NTNU) GMRES December 4, 2011 9 / 26
Set z = V k y, the least square problem (6) is equivalent to min J(y) = min βv 1 AV k y 2, β = r o 2. (7) y Rk y R k Using (5), we have J(y) = V k+1 (βe 1 H ) k y 2 = βe 1 H k y 2. (8) Hence, the solution of the least square (6) is x k = x o + V k y k, where y k minimize the function J(y) defined by (8) over y R k. T.-M. Huang (NTNU) GMRES December 4, 2011 10 / 26
GMRES Algorithm Input: Choose x 0, compute r 0 = b Ax 0 and v 1 = r 0 / r 0 ; Output: Solution of linear system Ax = b. 1: for j = 1, 2,..., k do 2: Compute h ij = (Av j, v i ) for i = 1, 2,..., j; 3: Compute ṽ j+1 = Av j j i=1 h ijv i ; 4: Compute h j+1,j = ṽ j+1 2 ; 5: Compute v j+1 = ṽ j+1 /h j+1,j ; 6: end for 7: Form the solution: x k = x 0 + V k y k, where y k minimizes J(y) in (8). Difficulties: when k is increasing, storage for v j, like k, the number of multiplications is like 1 2 k2 N. T.-M. Huang (NTNU) GMRES December 4, 2011 11 / 26
GMRES(m) Algorithm Input: Choose x 0, compute r 0 = b Ax 0 and v 1 = r 0 / r 0 ; Output: Solution of linear system Ax = b. 1: for j = 1, 2,..., m do 2: Compute h ij = (Av j, v i ) for i = 1, 2,..., j; 3: Compute ṽ j+1 = Av j j i=1 h ijv i ; 4: Compute h j+1,j = ṽ j+1 2 ; 5: Compute v j+1 = ṽ j+1 /h j+1,j ; 6: end for 7: Form the solution: x m = x 0 + V m y m, where y m minimizes βe 1 H m y for y R m. 8: Restart: Compute r m = b Ax m ; 9: if r m is small, then 10: stop, 11: else 12: Compute x 0 = x m and v 1 = r m / r m, GoTo for step. 13: end if T.-M. Huang (NTNU) GMRES December 4, 2011 12 / 26
Practical Implementation: Consider QR factorization of H k Consider the matrix H k. We want to solve the least squares problem: min βe 1 H k y 2. y R k Assume Givens rotations F i, i = 1,..., j such that F j F 1 Hj =F j F 1 0 0 0 0 0 0 = R j R (j+1) j. 0 Definition of Givens T.-M. Huang (NTNU) GMRES December 4, 2011 13 / 26
In order to obtain R j+1 we must start by premultiptying the new column by the previous rotations. H j+1 = + + 0 + 0 0 + 0 0 0 + 0 0 0 0 + F j F 1 Hj+1 = + + + + 0 r 0 h The principal upper (j + 1) j submatrix is nothing but R j, and h := h j+2,j+1 is not affected by the previous rotations. The next rotation F j+1 defined by { cj+1 r/(r 2 + h 2 ) 1/2, s j+1 = h/(r 2 + h 2 ) 1/2. T.-M. Huang (NTNU) GMRES December 4, 2011 14 / 26
Thus, after k steps of the above process, we have achieved Q k Hk = R k where Q k is a (k + 1) (k + 1) unitary matrix and J(y) = βe 1 H k y = Q k (βe 1 H ) k y = g k R k y, (9) where g k Q k βe 1. Since the last row of R k is a zero row, the 1 minimization of (9) is achieved at y k = R k g k, where R k and g k are removed the last row of R k and the last component of g k, respectively. Proposition 1 r k = b Ax k = The (k+1)-st component of g k. T.-M. Huang (NTNU) GMRES December 4, 2011 15 / 26
Proposition 2 The solution x j produced by GMRES at step j is exact which is equivalent to (i) The algorithm breaks down at step j, (ii) ṽ j+1 = 0, (iii) h j+1,j = 0, (iv) The degree of the minimal polynomial of r 0 is j. Corollary 3 For an n n problem GMRES terminates at most n steps. This uncommon type of breakdown is sometimes referred to as a Lucky breakdown is the context of the Lanczos algorithm. T.-M. Huang (NTNU) GMRES December 4, 2011 16 / 26
Proposition 3 Suppose that A is diagonalizable so that A = XDX 1 and let Then ε (m) = where κ(x) = X X 1. min p P m,p(0)=1 max p(λ i). λ i σ(a) r m+1 κ(x)ε (m) r 0, When A is positive real with symmetric part M, it holds that r m [1 α/β] m/2 r 0, where α = (λ min (M)) 2 and β = λ max (A T A). This proves the convergence of GMRES(m) for all m, when A is positive real. T.-M. Huang (NTNU) GMRES December 4, 2011 17 / 26
Theorem 4 Assume λ 1,..., λ ν of A with positive(negative) real parts and the other eigenvalues enclosed in a circle centered at C with C > 0 and have radius R with C > R. Then where ε (m) D = [ ] R m ν max C j=ν+1,,n ν i=1 λ i λ j λ i [ D d ] 2 [ ] R m ν C max i=1,,ν λ i λ j and d = min i. i=1,,ν j=ν+1,,n T.-M. Huang (NTNU) GMRES December 4, 2011 18 / 26
Proof. Consider p(z) = r(z)q(z) where r(z) = (1 z/λ 1 ) (1 z/λ ν ) and q(z) arbitrary polynomial of deg m ν such that q(0) = 1. Since p(0) = 1 and p(λ i ) = 0, for i = 1,..., ν, we have ε (m) It is easily seen that max p(λ j) j=ν+1,,n max r(λ j) max q(λ j). j=ν+1,,n j=ν+1,,n max r(λ j) = j=ν+1,,n max j=ν+1,,n ν i=1 λ i λ j λ i [ ] D ν. d By maximum principle, the maximum of q(z) for z {λ j } N j=ν+1 is on the circle. Taking σ(z) = [(C z)/c] m ν whose maximum on the circle is (R/C) m ν yields the desired result. T.-M. Huang (NTNU) GMRES December 4, 2011 19 / 26
Corollary 5 Under the assumptions of Proposition 3 and Theorem 4, GMRES(m) converges for any initial x 0 if [ ] / [ DC C m > νlog dr κ(x)1/ν Log R. T.-M. Huang (NTNU) GMRES December 4, 2011 20 / 26
Appendix Proof of Implicit Q Theorem Let A[q 1 q 2 q n] = [q 1 q 2 q n] h 11 h 12 h 1n. h 21 h..... 22. 0..................... hn 1,n 0 0 h n,n 1 h nn. (10) T.-M. Huang (NTNU) GMRES December 4, 2011 21 / 26
Then we have Aq 1 = h 11 q 1 + h 21 q 2. (11) Since q 1 q 2, it implies that h 11 = q1aq 1 /q1q 1. From (11), we get that q 2 h 21 q 2 = Aq 1 h 11 q 1. That is q 2 = q 2 / q 2 2 and h 21 = q 2 2. T.-M. Huang (NTNU) GMRES December 4, 2011 22 / 26
Similarly, from (10), Aq 2 = h 12 q 1 + h 22 q 2 + h 32 q 3, where h 12 = q 1Aq 2 and h 22 = q 2Aq 2. Let q 3 = Aq 2 h 12 q 1 + h 22 q 2. Then q 3 = q 3 / q 3 2 and h 32 = q 3, and so on. T.-M. Huang (NTNU) GMRES December 4, 2011 23 / 26
Therefore, [q 1,, q n ] are uniquely determined by q 1. Thus, uniqueness holds. Let K n = [v 1, Av 1,, A n 1 v 1 ] with v 1 2 = 1 is nonsingular. K n = U n R n and U n e 1 = v 1. Then 0 0. 1... AK n = K n C n = [v 1, Av 1,, A n 1 v 1 ]. 0........ (12)....... 0. 0 0 1 T.-M. Huang (NTNU) GMRES December 4, 2011 24 / 26
Since K n is nonsingular, (12) implies that That is A = K n C n K 1 n = (U n R n )C n (Rn 1 Un 1 ). AU n = U n (R n C n R 1 n ), where (R n C n Rn 1 ) is Hessenberg and U n e 1 = v 1. Because < U n >=< K n >, find AV n = V n H n by any method with V n e 1 = v 1, then it holds that V n = U n, i.e., v (i) n = u (i) n for i = 1,, n. Back to Theorem T.-M. Huang (NTNU) GMRES December 4, 2011 25 / 26
Definition 6 (Givens rotation) A plane rotation (also called a Givens rotation) is a matrix of the form [ ] c s G = s c where c 2 + s 2 = 1. Given a 0 and b, set v = a 2 + b 2, c = a /v and s = a a b v, then [ c s s c ] [ a b ] = [ v a a 0 ]. Back to Practice GMRES T.-M. Huang (NTNU) GMRES December 4, 2011 26 / 26