Krylov Subspace Methods for Large/Sparse Eigenvalue Problems

Krylov Subspace Methods for Large/Sparse Eigenvalue Problems Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan April 17, 2012 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 1 / 80

Outline 1 Lanczos decomposition Householder transformation 2 Implicitly restarted Lanczos method 3 Arnoldi method 4 Generalized eigenvalue problem Krylov-Schur restarting T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 2 / 80

Theorem 1 Let V be an eigenspace of A and let V be an orthonormal basis for V. Then there is a unique matrix H such that The matrix H is given by AV = V H. H = V AV. If (λ, x) is an eigenpair of A with x V, then (λ, V x) is an eigenpair of H. Conversely, if (λ, s) is an eigenpair of H, then (λ, V s) is an eigenpair of A. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 3 / 80

Theorem 2 (Optimal residuals) Let [V V ] be unitary. Let R = AV V H and S = V A HV. Then R and S are minimized when in which case (a) (b) H = V AV, R = V AV, S = V AV, (c) V R = 0. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 4 / 80

Definition 3 Let V be orthonormal. Then V AV is a Rayleigh quotient of A. Theorem 4 Let V be orthonormal, A be Hermitian and R = AV V H. If θ 1,..., θ k are the eigenvalues of H, then there are eigenvalues λ j1,..., λ jk of A such that θ i λ ji R 2 and k (θ i λ ji ) 2 2 R F. i=1 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 5 / 80

Suppose the eigenvalue with maximum module is wanted. Power method Compute the dominant eigenpair Disadvantage At each step it considers only the single vector A k u, which throws away the information contained in the previously generated vectors u, Au, A 2 u,..., A k 1 u. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 6 / 80

Definition 5 Let A be of order n and let u 0 be an n vector. Then {u, Au, A 2 u, A 3 u,...} is a Krylov sequence based on A and u. We call the matrix K k (A, u) = [ u Au A 2 u A k 1 u ] the kth Krylov matrix. The space K k (A, u) = R[K k (A, u)] is called the kth Krylov subspace. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 7 / 80

By the definition of K k (A, u), for any vector v K k (A, u) can be written in the form where v = γ 1 u + γ 2 Au + + γ k A k 1 u p(a)u, p(a) = γ 1 I + γ 2 A + γ 3 A 2 + + γ k A k 1. Assume that A = A and Ax i = λ i x i for i = 1,..., n. Write u in the form Since p(a)x i = p(λ i )x i, we have u = α 1 x 1 + α 2 x 2 + + α n x n. p(a)u = α 1 p(λ 1 )x 1 + α 2 p(λ 2 )x 2 + + α n p(λ n )x n. (1) If p(λ i ) is large compared with p(λ j ) for j i, then p(a)u is a good approximation to x i. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 8 / 80

Theorem 6 If x H i u 0 and p(λ i) 0, then tan (p(a)u, x i ) max j i Proof: From (1), we have cos (p(a)u, x i ) = xh i p(a)u p(a)u 2 x i 2 = p(λ j ) p(λ i ) tan (u, x i). α i p(λ i ) n j=1 α jp(λ j ) 2 and j i α jp(λ j ) 2 sin (p(a)u, x i ) = n j=1 α jp(λ j ) 2 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 9 / 80

Hence tan 2 (p(a)u, x i ) = j i max j i = max j i α j p(λ j ) 2 α i p(λ i ) 2 p(λ j ) 2 p(λ i ) 2 j i α j 2 α i 2 p(λ j ) 2 p(λ i ) 2 tan2 (u, x i ). Assume that p(λ i ) = 1, then tan (p(a)u, x i ) Hence max p(λ j) tan (u, x i ) p(a)u K k. j i,p(λ i )=1 tan (x i, K k ) min max p(λ j) tan (u, x i ). deg(p) k 1,p(λ i )=1 j i T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 10 / 80

Assume that λ 1 > λ 2 λ n and that our interest is in the eigenvector x 1. Then tan (x 1, K k ) Question How to compute min deg(p) k 1,p(λ 1 )=1 min deg(p) k 1,p(λ 1 )=1 max p(λ) tan (u, x 1). λ [λ n,λ 2 ] max p(λ)? λ [λ n,λ 2 ] Definition 7 The Chebyshev polynomials are defined by c k (t) = { cos(k cos 1 t), t 1, cosh(k cosh 1 t), t 1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 11 / 80

Theorem 8 (i) c 0 (t) = 1, c 1 (t) = t and c k+1 (t) = 2c k (t) c k 1 (t), k = 1, 2,.... (ii) For t > 1, c k (t) = (1 + t 2 1) k + (1 + t 2 1) k. (iii) For t [ 1, 1], c k (t) 1. Moreover, if t ik = cos then c k (t ik ) = ( 1) k i. (iv) For s > 1, (k i)π, i = 0, 1,..., k, k min max p(t) = 1 deg(p) k,p(s)=1 t [0,1] c k (s), (2) and the minimum is obtained only for p(t) = c k (t)/c k (s). T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 12 / 80

For applying (2), we define λ = λ 2 + (µ 1)(λ 2 λ n ) to transform interval [λ n, λ 2 ] to [0, 1]. Then the value of µ at λ 1 is and µ 1 = 1 + λ 1 λ 2 λ 2 λ n min deg(p) k 1,p(λ 1 )=1 max p(λ) λ [λ n,λ 2 ] = min deg(p) k 1,p(µ 1 )=1 max µ [0,1] p(µ) = 1 c k 1 (µ 1 ) T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 13 / 80

Theorem 9 Let A = A and Ax i = λ i x i, i = 1,, n with λ 1 > λ 2 λ n. Let η = λ 1 λ 2 λ 2 λ n. Then tan [x 1, K k (A, u)] tan (x 1, u) c k 1 (1 + η) tan (x 1, u) = (1 + 2η + η 2 ) k 1 + (1 + 2η + η 2 ). 1 k T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 14 / 80

For k large and if η is small, then the bound becomes tan [x 1, K k (A, u)] tan (x 1, u) (1 + 2η) k 1. Compare it with power method: If λ 1 > λ 2 λ n, then the conv. rate is λ 2 /λ 1 k. For example, let λ 1 = 1, λ 2 = 0.95, λ 3 = 0.95 2,, λ 100 = 0.95 99 be the Ews of A R 100 100. Then η = 0.0530 and the bound on the conv. rate is 1/(1 + 2η) = 0.7544. Thus the square root effect gives a great improvement over the rate of 0.95 for the power method. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 15 / 80

Definition 10 A Householder transformation or elementary reflector is a matrix of where u 2 = 2. H = I uu Note that H is Hermitian and unitary. Theorem 11 Let x be a vector with x 1 0. Let where ρ = x 1 / x 1. Then u = ρ x x 2 + e 1, 1 + ρ x 1 x 2 Hx = ρ x 2 e 1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 16 / 80

Proof: Since [ ρx / x 2 + e 1 ][ρx/ x 2 + e 1 ] = ρρ + ρx 1 / x 2 + ρ x 1 / x 2 + 1 = 2[1 + ρx 1 / x 2 ], it follows that and u u = 2 u 2 = 2 u x = ρ x 2 + x 1. 1 + ρ x 1 x 2 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 17 / 80

Hence, Hx = x (u x)u = x ρ x 2 + x 1 1 + ρ x 1 = [1 ( ρ x 2 + x 1 ) ρ ] x 2 1 + ρ x 1 x 2 = ρ x 2 + x 1 1 + ρ x 1 x 2 e 1 = ρ x 2 e 1. x 2 ρ x x 2 + e 1 1 + ρ x 1 x 2 x ρ x 2 + x 1 1 + ρ x 1 x 2 e 1 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 18 / 80

Definition 12 A complex m n-matrix R = [r ij ] is called an upper (lower) triangular matrix, if r ij = 0 for i > j (i < j). Definition 13 Given A C m n, Q C m m unitary and R C m n upper triangular such that A = QR. Then the product is called a QR-factorization of A. Theorem 14 Any complex m n matrix A can be factorized by the product A = QR, where Q is m m-unitary and R is m n upper triangular. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 19 / 80

Proof: Let A (0) = A = [a (0) 1 a(0) 2 a(0) n ]. Find Q 1 = (I 2w 1 w1 ) such that Q 1 a (0) 1 = ce 1. Then A (1) = Q 1 A (0) = [Q 1 a (0) 1, Q 1a (0) 2,, Q 1a (0) n ] c 1 0 =. a (1) 2 a (1) n. (3) 0 [ ] 1 0 Find Q 2 = 0 I w 2 w2 A (2) = Q 2 A (1) = such that (I 2w 2 w 2 )a(1) 2 = c 2 e 1. Then c 1 0 c 2 0 0.. a (2) 3 a (2) n 0 0. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 20 / 80

We continue this process. Then after l = min(m, n) steps A (l) is an upper triangular matrix satisfying A (l 1) = R = Q l 1 Q 1 A. Then A = QR, where Q = Q 1 Q l 1. Suppose that the columns of K k+1 are linearly independent and let K k+1 = U k+1 R k+1 be the QR factorization of K k+1. Then the columns of U k+1 are results of successively orthogonalizing the columns of K k+1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 21 / 80

Theorem 15 Let u 1 2 = 1 and the columns of K k+1 (A, u 1 ) be linearly indep. Let U k+1 = [u 1 u k+1 ] be the Q-factor of K k+1. Then there is a (k + 1) k unreduced upper Hessenberg matrix Ĥ k ĥ 11 ĥ 1k ĥ 21 ĥ 22 ĥ 2k....... ĥ k,k 1 ĥ kk ĥ k+1,k with ĥ i+1,i 0 (4) such that AU k = U k+1 Ĥ k. (Arnoldi decomp.) (5) Conversely, if U k+1 is orthonormal and satisfies (5), where Ĥk is defined in (4), then U k+1 is the Q-factor of K k+1 (A, u 1 ). T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 22 / 80

Proof: ( ) Let K k = U k R k be the QR factorization and S k = R 1 k. Then [ ] [ ] 0 0 AU k = AK k S k = K k+1 = U S k+1 R k+1 = U k S k+1 Ĥ k, k where Ĥ k = R k+1 [ 0 S k ]. It implies that Ĥk is a (k + 1) k Hessenberg matrix and h i+1,i = r i+1,i+1 s ii = r i+1,i+1 r ii. Thus by the nonsingularity of R k, Ĥk is unreduced. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 23 / 80

( ) If k = 1, then which implies that Au 1 = h 11 u 1 + h 21 u 2 = [ u 1 u 2 ] [ h 11 h 21 K 2 = [ u 1 Au 1 ] = [ u1 u 2 ] [ 1 h 11 0 h 21 Since [ u 1 u 2 ] is orthonormal, [ u 1 u 2 ] is the Q-factor of K 2. Assume U k is the Q-factor of K k, i.e., K k = U k R k. By the definition of the Krylov matrix, we have K k+1 = [ u 1 AK k ] = [ u1 AU k R k ] = [ u1 U k+1 Ĥ k R k ] = U k+1 [ e1 Ĥ k R k ] ] ]. Hence U k+1 is the Q-factor of K k+1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 24 / 80

The uniqueness of Hessenberg reduction Definition 16 Let H be upper Hessenberg of order n. Then H is unreduced if h i+1,i 0 for i = 1,, n 1. Theorem 17 (Implicit Q theorem) Suppose Q = ( q 1 ) ( q n and V = v1 ) v n are unitary matrices with Q AQ = H and V AV = G being upper Hessenberg. Let k denote the smallest positive integer for which h k+1,k = 0, with the convection that k = n if H is unreduced. If v 1 = q 1, then v i = ±q i and h i,i 1 = g i,i 1 for i = 2,, k. Moreover, if k < n, then g k+1,k = 0. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 25 / 80

Definition 18 Let U k+1 C n (k+1) be orthonormal. If there is a (k + 1) k unreduced upper Hessenberg matrix Ĥk such that AU k = U k+1 Ĥ k, (6) then (6) is called an Arnoldi decomposition of order k. If Ĥk is reduced, we say the Arnoldi decomposition is reduced. Partition and set Then (6) is equivalent to [ H Ĥ k = k h k+1,k e T k β k = h k+1,k. ], AU k = U k H k + β k u k+1 e T k. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 26 / 80

Theorem 19 Suppose the Krylov sequence K k+1 (A, u 1 ) does not terminate at k + 1. Then up to scaling of the columns of U k+1, the Arnoldi decomposition of K k+1 is unique. Proof: Since the Krylov sequence K k+1 (A, u 1 ) does not terminate at k + 1, the columns of K k+1 (A, u 1 ) are linearly independent. By Theorem 15, there is an unreduced matrix H k and β k 0 such that AU k = U k H k + β k u k+1 e T k, (7) where U k+1 = [U k u k+1 ] is an orthonormal basis for K k+1 (A, u 1 ). Suppose there is another orthonormal basis Ũk+1 = [Ũk ũ k+1 ] for K k+1 (A, u 1 ), unreduced matrix H k and β k 0 such that AŨk = Ũk H k + β k ũ k+1 e T k. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 27 / 80

Then we claim that Ũ H k u k+1 = 0. For otherwise there is a column ũ j of Ũk such that Hence ũ j = αu k+1 + U k a, α 0. Aũ j = αau k+1 + AU k a which implies that Aũ j contains a component along A k+1 u 1. Since the Krylov sequence K k+1 (A, u 1 ) does not terminate at k + 1, we have K k+2 (A, u 1 ) K k+1 (A, u 1 ). Therefore, Aũ j lies in K k+2 (A, u 1 ) but not in K k+1 (A, u 1 ) which is a contradiction. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 28 / 80

Since U k+1 and Ũk+1 are orthonormal bases for K k+1 (A, u 1 ) and Ũ H k u k+1 = 0, it follows that that is R(U k ) = R(Ũk) and U H k ũk+1 = 0, U k = ŨkQ for some unitary matrix Q. Hence or A(ŨkQ) = (ŨkQ)(Q H Hk Q) + β k ũ k+1 (e T k Q), AU k = U k (Q H Hk Q) + β k ũ k+1 e T k Q. (8) On premultiplying (7) and (8) by U H k, we obtain H k = U H k AU k = Q H Hk Q. Similarly, premultiplying by u H k+1, we obtain β k e T k = uh k+1 AU k = β k (u H k+1ũk+1)e T k Q. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 29 / 80

It follows that the last row of Q is ω k e T k, where ω k = 1. Since the norm of the last column of Q is one, the last column of Q is ω k e k. Since H k is unreduced, it follows from the implicit Q theorem that Q = diag(ω 1,, ω k ), ω j = 1, j = 1,..., k. Thus up to column scaling U k = ŨkQ is the same as Ũk. Subtracting (8) from (7), we find that β k u k+1 = ω k βk ũ k+1 so that up to scaling u k+1 and ũ k+1 are the same. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 30 / 80

Let A be Hermitian and let AU k = U k T k + β k u k+1 e k (9) be an Arnoldi decomposition. Since T k is upper Hessenberg and T k = Uk HAU k is Hermitian, it follows that T k is tridiagonal and can be written in the form T k = α 1 β 1 β 1 α 2 β 2 β 2 α 3 β 3......... β k 2 α k 1 β k 1 β k 1 α k. Equation (9) is called a Lanczos decomposition. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 31 / 80

The first column of (9) is Au 1 = α 1 u 1 + β 1 u 2 or u 2 = Au 1 α 1 u 1 β 1. From the orthonormality of u 1 and u 2, it follows that α 1 = u 1Au 1 and β 1 = Au 1 α 1 u 1 2. More generality, from the j-th column of (9) we get the relation where u j+1 = Au j α j u j β j 1 u j 1 β j α j = u jau j and β j = Au j α j u j β j 1 u j 1 2. This is the Lanczos three-term recurrence. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 32 / 80

Algorithm 1 (Lanczos recurrence) Let u 1 be given. This algorithm generates the Lanczos decomposition AU k = U k T k + β k u k+1 e k where T k is symmetric tridiagonal. 1. u 0 = 0; β 0 = 0; 2. for j = 1 to k 3. u j+1 = Au j 4. α j = u j u j+1 5. v = u j+1 α j u j β j 1 u j 1 6. β j = v 2 7. u j+1 = v/β j 8. end for j T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 33 / 80

Reorthogonalization Let ũ j+1 = Au j α j u j β j 1 u j 1 with α j = u j Au j. Re-orthogonalize ũ j+1 against U j, i.e., j ũ j+1 := ũ j+1 (u i ũ j+1 ) u i Take i=1 j 1 = Au j ( α j + u ) jũ j+1 uj β j 1 u j 1 (u i ũ j+1 ) u i i=1 β j = ũ j+1 2, u j+1 = ũ j+1 /β j. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 34 / 80

Theorem 20 (Stop criterion) Suppose that j steps of the Lanczos algorithm have been performed and that S H j T j S j = diag(θ 1,, θ j ) is the Schur decomposition of the tridiagonal matrix T j, if Y j C n j is defined by Y j [ y 1 y j ] = Uj S j then for i = 1,, j we have where S j = [s pq ]. Ay i θ i y i 2 = β j s ji T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 35 / 80

Proof : Post-multiplying AU j = U j T j + β j u j+1 e j by S j gives AY j = Y j diag(θ 1,, θ j ) + β j u j+1 e j S j, i.e., Ay i = θ i y i + β j u j+1 (e j S j e i ), i = 1,, j. The proof is complete by taking norms. Remark 1 Stop criterion = β j s ji. Do not need to compute Ay i θ i y i 2. In general, β j is not small. It is possible that β j s ji is small. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 36 / 80

Theorem 21 Let A be n n symmetric matrix with eigenvalues λ 1 λ n and corresponding orthonormal eigenvectors z 1,, z n. If θ 1 θ j are the eigenvalues of T j obtained after j steps of the Lanczos iteration, then λ 1 θ 1 λ 1 (λ 1 λ n )(tan φ 1 ) 2 [c j 1 (1 + 2ρ 1 )] 2, where cos φ 1 = u 1 z 1, c j 1 is a Chebychev polynomal of degree j 1 and ρ 1 = λ 1 λ 2 λ 2 λ n. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 37 / 80

Proof: From Courant-Fischer theorem we have θ 1 = max y 0 y T j y y y = max y 0 (U j y) A(U j y) (U j y) (U j y) = max w Aw 0 w K j (u 1,A) w w. Since λ 1 is the maximum of w Aw/w w over all nonzero w, it follows that λ 1 θ 1. To obtain the lower bound for θ 1, note that θ 1 = max p P j 1 u 1 p(a)ap(a)u 1 u 1 p(a)2 u 1, where P j 1 is the set of all j 1 degree polynomials. If u 1 = n i=1 d iz i, then u 1 p(a)ap(a)u 1 u 1 p(a)2 u 1 = n i=1 d2 i p(λ i) 2 λ i n i=1 d2 i p(λ i) 2 λ 1 (λ 1 λ n ) n i=2 d2 i p(λ i) 2 d 2 1 p(λ 1) 2 + n i=2 d2 i p(λ i) 2. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 38 / 80

We can make the lower bound tight by selecting a polynomial p(α) that is large at α = λ 1 in comparison to its value at the remaining eigenvalues. Set ( p(α) = c j 1 1 + 2 α λ ) n, λ 2 λ n where c j 1 (z) is the (j 1)-th Chebychev polynomial generated by c j (z) = 2zc j 1 (z) c j 2 (z), c 0 = 1, c 1 = z. These polynomials are bounded by unity on [-1,1]. It follows that p(λ i ) is bounded by unity for i = 2,, n while p(λ 1 ) = c j 1 (1 + 2ρ 1 ). Thus, θ 1 λ 1 (λ 1 λ n ) (1 d2 1 ) d 2 1 1 c 2 j 1 (1 + 2ρ 1). The desired lower bound is obtained by noting that tan (φ 1 ) 2 = (1 d 2 1 )/d2 1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 39 / 80

Theorem 22 Using the same notation as Theorem 21, where λ n θ j λ n + (λ 1 λ n ) tan 2 ϕ n [c j 1 (1 + 2ρ n )] 2, ρ n = λ n 1 λ n λ 1 λ n 1, cos ϕ n = u 1 z n. Proof : Apply Theorem 21 with A replaced by A. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 40 / 80

Restarted Lanczos method Let be a Lanczos decomposition. AU m = U m T m + β m u m+1 e T m 1 In principle, we can keep expanding the Lanczos decomposition until the Ritz pairs have converged. 2 Unfortunately, it is limited by the amount of memory to storage of U m. 3 Restarted the Lanczos process once m becomes so large that we cannot store U m. Implicitly restarting method Krylov-Schur decomposition T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 41 / 80

Implicitly restarted Lanczos method Choose a new starting vector for the underlying Krylov sequence A natural choice would be a linear combination of Ritz vectors that we are interested in. Filter polynomials Assume A has a complete system of eigenpairs (λ i, x i ) and we are interested in the first k of these eigenpairs. Expand u 1 in the form u 1 = k n γ i x i + γ i x i. i=1 i=k+1 If p is any polynomial, we have p(a)u 1 = k n γ i p(λ i )x i + γ i p(λ i )x i. i=1 i=k+1 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 42 / 80

Choose p so that the values p(λ i ) (i = k + 1,..., n) are small compared to the values p(λ i ) (i = 1,..., k). Then p(a)u 1 is rich in the components of the x i that we want and deficient in the ones that we do not want. p is called a filter polynomial. Suppose we have Ritz values θ 1,..., θ m and θ 1,..., θ m k are not interesting. Then take Implicitly restarted Lanczos: Let p(t) = (t θ 1 ) (t θ m k ). AU m = U m T m + β m u m+1 e m (10) be a Lanczos decomposition with order m. Choose a filter polynomial p of degree m k and use the implicit restarting process to reduce the decomposition to a decomposition AŨk = Ũk T k + β k ũ k+1 e k of order k with starting vector p(a)u 1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 43 / 80

Let θ 1,..., θ m be eigenvalues of T m and suppose that θ 1,..., θ m k correspond to the part of the spectrum we are not interested in. Then take p(t) = (t θ 1 )(t θ 2 ) (t θ m k ). The starting vector p(a)u 1 is equal to p(a)u 1 = (A θ m k I) (A θ 2 I)(A θ 1 I)u 1 = (A θ m k I) [ [(A θ 2 I) [(A θ 1 I)u 1 ]]]. In the first, we construct a Lanczos decomposition with starting vector (A θ 1 I)u 1. From (10), we have where (A θ 1 I)U m = U m (T m θ 1 I) + β m u m+1 e m (11) = U m Q 1 R 1 + β m u m+1 e m, T m θ 1 I = Q 1 R 1 is the QR factorization of T m θ 1 I. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 44 / 80

Postmultiplying by Q 1, we get (A θ 1 I)(U m Q 1 ) = (U m Q 1 )(R 1 Q 1 ) + β m u m+1 (e mq 1 ). It implies that where (T (1) m AU m (1) = U m (1) T m (1) + β m u m+1 b (1)H m+1, U (1) m = U m Q 1, T (1) m = R 1 Q 1 + θ 1 I, b (1)H m+1 = e mq 1. : one step of single shifted QR algorithm) T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 45 / 80

Theorem 23 Let T m be an unreduced tridiagonal. Then T m (1) ] T (1) m = [ ˆT (1) m 0 0 θ 1, has the form where ˆT (1) m is unreduced tridiagonal. Proof: Let T m θ 1 I = Q 1 R 1 be the QR factorization of T m θ 1 I with Q 1 = G(1, 2, ϕ 1 ) G(m 1, m, ϕ m 1 ) where G(i, i + 1, ϕ i ) for i = 1,..., m 1 are Given rotations. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 46 / 80

Since T m is unreduced tridiagonal, i.e., the subdiagonal elements of T m are nonzero, we get and ϕ i 0 for i = 1,..., m 1 (12) (R 1 ) ii 0 for i = 1,..., m 1. (13) Since θ 1 is an eigenvalue of T m, we have that T m θ 1 I is singular and then Using the results of (12), (13) and (14), we get T (1) (R 1 ) mm = 0. (14) m = R 1 Q 1 + θ 1 I = R 1 G(1, 2, ϕ 1 ) G(m 1, m, ϕ m 1 ) + θ 1 I ] = where Ĥ(1) m [ Ĥ(1) m ĥ12 0 θ 1, is unreduced upper Hessenberg. (15) T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 47 / 80

By the definition of T (1) m, we get Q 1 T (1) m Q H 1 = Q 1 (R 1 Q 1 + θ 1 I)Q H 1 = Q 1 R 1 + θ 1 I = T m. It implies that T m (1) obtained. Remark 2 is tridiagonal and then, from (15), the result in (12) is is orthonormal. The vector b (1)H m+1 = e mq 1 has the form U (1) m b (1)H m+1 = [ 0 0 q (1) m 1,m q(1) m,m ] ; i.e., only the last two components of b (1) m+1 are nonzero. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 48 / 80

For on postmultiplying (11) by e 1, we get (A θ 1 I)u 1 = (A θ 1 I)(U m e 1 ) = U (1) m R 1 e 1 = r (1) 11 u(1) 1. Since T m is unreduced, r (1) 11 is nonzero. Therefore, the first column of U m (1) is a multiple of (A θ 1 I)u 1. By the definition of T (1) m, we get Q 1 T (1) m Q H 1 = Q 1 (R 1 Q 1 + θ 1 I)Q H 1 = Q 1 R 1 + θ 1 I = T m. Therefore, θ 1, θ 2,..., θ m are also eigenvalues of T (1) m. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 49 / 80

Similarly, where (A θ 2 I)U m (1) = U m (1) (T m (1) θ 2 I) + β m u m+1 b (1)H m+1 (16) = U m (1) Q 2 R 2 + β m u m+1 b (1)H m+1, is the QR factorization of T (1) m T (1) m θ 2 I = Q 2 R 2 θ 2 I. Postmultiplying by Q 2, we get (A θ 2 I)(U (1) m Q 2 ) = (U (1) m Q 2 )(R 2 Q 2 ) + β m u m+1 (b (1)H m+1 Q 2). It implies that where is orthonormal, AU m (2) = U m (2) T m (2) + β m u m+1 b (2)H m+1, U (2) m U (1) m Q 2 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 50 / 80

T m (2) R 2 Q 2 + θ 2 I = T (2) m 2 0 0 θ 2 0 θ 1 is tridiagonal with unreduced matrix T (2) m 2 and b (2)H m+1 b (1)H m+1 Q 2 = q (1) m 1,m eh m 1Q 2 + q (1) m,me mq 2 = [ 0 0 ]. For on postmultiplying (16) by e 1, we get (A θ 2 I)u (1) 1 = (A θ 2 I)(U (1) m e 1 ) = U (2) m R 2 e 1 = r (2) 11 u(2) 1. Since H m (1) is unreduced, r (2) 11 is nonzero. Therefore, the first column of is a multiple of U (2) m (A θ 2 I)u (1) 1 = 1/r (1) 11 (A θ 2I)(A θ 1 I)u 1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 51 / 80

Repeating this process with θ 3,..., θ m k, the result will be a Krylov decomposition AU (m k) m with the following properties 1 U m (m k) 2 T m (m k) is orthonormal. is tridiagonal. = U m (m k) T m (m k) + β m u m+1 b (m k)h m+1 3 The first k 1 components of b (m k)h m+1 are zero. 4 The first column of U m (m k) is a multiple of (A θ 1 I) (A θ m k I)u 1. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 52 / 80

Corollary 24 Let θ 1,..., θ m be eigenvalues of T m. If the implicitly restarted QR step is performed with shifts θ 1,..., θ m k, then the matrix T m (m k) has the form [ ] T m (m k) T (m k) = kk 0 0 D (m k), where D (m k) is an digonal matrix with Ritz value θ 1,..., θ m k on its diagonal. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 53 / 80

For k = 3 and m = 6, A [ u u u u u u ] = [ u u u u u u ] +u [ 0 0 q q q q ]. 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Therefore, the first k columns of the decomposition can be written in the form AU (m k) k where U (m k) k = U (m k) k T (m k) kk + t k+1,k u (m k) k+1 e k + β mq mk u m+1 e k, consists of the first k columns of U m (m k), T (m k) kk leading principal submatrix of order k of T m (m k) matrix Q = Q 1 Q m k. is the, and q mk is from the T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 54 / 80

Hence if we set then Ũ k = U (m k) k, T k = T (m k) kk, β k = t k+1,k u (m k) k+1 + β m q mk u m+1 2, 1 ũ k+1 = β k (t k+1,ku (m k) k+1 + β m q mk u m+1 ), AŨk = Ũk T k + β k ũ k+1 e k is a Lanczos decomposition whose starting vector is proportional to (A θ 1 I) (A θ m k I)u 1. Avoid any matrix-vector multiplications in forming the new starting vector. Get its Lanczos decomposition of order k for free. For large n the major cost will be in computing UQ. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 55 / 80

Practical Implementation Restarted Lanczos method Input: Given Lanczos decomp. AU m = U m T m + β m u m+1 e m Output: new Lanczos decomp. AU k = U k T k + β k u k+1 e k 1: Compute the eigenvalues θ 1,..., θ m of T m. 2: Determine shifts, said θ 1,..., θ m k, and set b m = e m. 3: for j = 1,..., m k do 4: Compute QR factorization: T m θ j I = Q m R m. 5: Update T m := R m Q m + θ j I, U m := U m Q m, b m := Q mb m. 6: end for 7: Compute v = β k u k+1 + β m b m (k)u m+1. 8: Set U k := U m (:, 1 : k), β k = v 2, u k+1 = v/β k, and T k := T m (1 : k, 1 : k), T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 56 / 80

Question Can we implicitly compute Q m and get new tridiagonal matrix T m? General algorithm 1 Determine the first column c 1 of T m θ j I. 2 Let Q be a Householder transformation such that Q c 1 = σe 1. 3 Set T = Q T m Q. 4 Use Householder transformation Q to reduce T to a new tridiagonal form T Q T Q. 5 Set Q m = Q Q. Question General algorithm = one step of single shift QR algorithm? T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 57 / 80

Answer: (I) Let T m θ j I = [ c 1 C ] = Qm R m = [ q Q m ] [ ρ r 0 R be the [ QR factorization of T m θ j I. Then c 1 = ρq. Partition Q ˆq Q ], then c 1 = σ Qe 1 = σˆq which implies that q and ˆq are proportional to c 1. (II) Since T = Q T Q is tridiagonal, we have Qe 1 = e 1. ] Hence, ( Q Q)e 1 = Qe 1 = ˆq which implies that the first column of Q Q is proportional to q. (III) Since ( Q Q) T m ( Q Q) is tridiagonal and the first column of Q Q is proportional to q, by the implicit Q Theorem, if T is unreduced, then Q = Q 0 Q 1 and T = ( Q Q) T m ( Q Q). T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 58 / 80

Definition 25 (Givens rotation) A plane rotation (also called a Givens rotation) is a matrix of the form [ ] c s G = s c where c 2 + s 2 = 1. Given a 0 and b, set v = a 2 + b 2, c = a /v and s = a a b v, then [ c s s c ] [ a b ] = [ v a a 0 ]. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 59 / 80

Let I i 1 c s G ij = I j i 1 s c (I) Compute the first column t 1 α 1 θ j β 1 0 I n j. of T m θ j I and determine Givens rotation G 12 such that G 12 t 1 = γe 1. (II) Set T m := G 12 T m G 12. T m := G 12 0 0 0 0 0 0 0 0 0 0 0 0 G 12 = + 0 0 0 0 + 0 0 0 0 0 0 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 60 / 80

(III) Construct orthonormal Q such that T m := QT m Q is tridiagonal: + 0 0 0 0 0 0 0 + 0 T m := G 23 + 0 0 0 G 23 = 0 0 0 + 0 0 0 0 0 0 T m := G 34 T m := G 45 0 0 0 + 0 0 0 0 + 0 0 0 0 0 0 0 0 0 + 0 0 0 0 + G 34 = G 45 = 0 0 0 0 0 0 + 0 0 0 0 + 0 0 0 0 0 0 0 0 0 0 0 0 T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 61 / 80

Implicit Restarting for Lanczos method Input: Given Lanczos decomp. AU m = U m T m + β m u m+1 e m Output: new Lanczos decomp. AU k = U k T k + β k u k+1 e k 1: Compute the eigenvalues θ 1,..., θ m of T m. 2: Determine shifts, said θ 1,..., θ m k, and set b m = e m. 3: for j = 1,..., m k do α 1 θ j 4: Compute Givens rotation G 12 such that G 12 β 1 0 = γe 1. 5: Update T m := G 12 T m G 12, U m := U m G 12, b m := G 12 b m. 6: Compute Givens rotations G 23,..., G m 1,m such that T m := G m 1,m G 23 T m G 23 G m 1,m is tridiagonal. 7: Update U m := U m G 23 G m 1,m and b m := G m 1,m G 23 b m. 8: end for 9: Compute v = β k u k+1 + β m b m (k)u m+1. 10: Set U k := U m (:, 1 : k), β k = v 2, u k+1 = v/β k, and T k := T m (1 : k, 1 : k), T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 62 / 80

Problem Mathematically, u j must be orthogonal. In practice, they can lose orthogonality. Solutions Reorthogonalize the vectors at each step. j-th step of Lanczos process Input: Given β j 1 and orthonormal matrix U j = [u 1,..., u j ]. Output: α j, β j and unit vector u j+1 with u j+1 U j = 0 and Au j = α j u j + β j 1 u j 1 + β j u j+1. 1: Compute u j+1 = Au j β j 1 u j 1 and α j = u j u j+1; 2: Update u j+1 := u j+1 α j u j ; 3: for i = 1,..., j do 4: Compute γ i = u i u j+1 and update u j+1 := u j+1 γ i u i ; 5: end for 6: Update α j := α j + γ j and compute β j = u j+1 2 and u j+1 := u j+1 /β j. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 63 / 80

Lanczos algorithm with implicit restarting Input: Given initial unit vector u 1, number k of desired eigenpairs, restarting number m and stopping tolerance ε. Output: desired eigenpairs (θ i, x i ) for i = 1,..., k. 1: Compute Lanczos decomposition with order k: AU k = U k T k + β k u k+1 e k ; 2: repeat 3: Extend the Lanczos decomposition from order k to order m: AU m = U m T m + β m u m+1 e m; 4: Use implicitly restarting scheme to reform a new Lanczos decomposition with order k; 5: Compute the eigenpairs (θ i, s i ), i = 1,..., k, of T k ; 6: until ( β k s i,k < ε for i = 1,..., k) 7: Compute eigenvector x i = U k s i for i = 1,..., k. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 64 / 80

Arnoldi method Recall: Arnoldi decomposition of unsymmetric A: AU k = U k H k + h k+1,k u k+1 e k, (17) where H k is unreduced upper Hessenberg. Write (17) in the form Au k = U k h k + h k+1,k u k+1. Then from the orthogonality of U k+1, we have h k = U H k Au k. Since h k+1,k u k+1 = Au k U k h k and u k+1 2 = 1, we must have h k+1,k = Au k U k h k 2, u k+1 = h 1 k+1,k (Au k U k h k ). T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 65 / 80

Arnoldi process 1: for k = 1, 2,... do 2: h k = Uk HAu k; 3: v = Au k U k h k ; 4: h k+1,k = v 2 ; 5: u k+1 = v/h k+1,k ; [ Ĥk 1 h 6: Ĥ k = k 0 h k+1,k 7: end for ] ; The computation of u k+1 is actually a form of the well-known Gram-Schmidt algorithm. In the presence of inexact arithmetic cancelation in statement 3 can cause it to fail to produce orthogonal vectors. The cure is process called reorthogonalization. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 66 / 80

Reorthogonalized Arnoldi process 1: for k = 1, 2,... do 2: h k = Uk HAu k; 3: v = Au k U k h k ; 4: w = Uk Hv; 5: h k = h k + w; 6: v = v U k w; 7: h k+1,k = v 2 ; 8: u k+1 = v/h k+1,k ; [ Ĥk 1 h 9: Ĥ k = k 0 h k+1,k 10: end for ] ; T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 67 / 80

Let y (k) i and x (k) i Theorem 26 and therefore, be an eigenvector of H k associated with the eigenvalue λ (k) i = U k y (k) i the Ritz approximate eigenvector. (A λ (k) i I)x (k) i = h k+1,k e k y(k) i u k+1. (A λ (k) i I)x (k) i 2 = h k+1,k e k y(k) i. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 68 / 80

Generalized eigenvalue problem Consider the generalized eigenvalue problem where B is nonsingular. Let Ax = λbx, C = B 1 A. Applying Arnoldi process to matrix C, we get or CU k = U k H k + h k+1,k u k+1 e k, AU k = BU k H k + h k+1,k Bu k+1 e k. (18) T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 69 / 80

Write the k-th column of (18) in the form Au k = BU k h k + h k+1,k Bu k+1. (19) Let U k satisfy that Uk U k = I k. Then h k = Uk B 1 Au k and h k+1,k u k+1 = B 1 Au k U k h k t k which implies that h k+1,k = t k, u k+1 = h 1 k+1,k t k. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 70 / 80

Shift-and-invert Lanczos for GEP Consider the generalized eigenvalue problem Ax = λbx, (20) where A is symmetric and B is symmetric positive definite. Shift-and-invert Compute the eigenvalues which are closest to a given shift value σ. Transform (20) into (A σb) 1 Bx = (λ σ) 1 x. (21) T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 71 / 80

The basic recursion for applying Lanczos method to (21) is (A σb) 1 BU j = U j T j + β j u j+1 e j, (22) where the basis U j is B-orthogonal and T j is a real symmetric tridiagonal matrix defined by α 1 β 1. T j = β 1 α.. 2...... βj 1 or equivalent to β j 1 (A σb) 1 Bu j = α j u j + β j 1 u j 1 + β j u j+1. α j T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 72 / 80

By the condition U j BU j = I j, it holds that where α j = u jb(a σb) 1 Bu j, β 2 j = t jbt j, t j (A σb) 1 Bu j α j u j β j 1 u j 1 = β j u j+1. An eigenpair (θ i, s i ) of T j is used to get an approximate eigenpair (λ i, x i ) of (A, B) by λ i = σ + 1 θ i, x i = U j s i. The corresponding residual is r i = AU j s i λ i BU j s i = (A σb)u j s i θi 1 BU j s i = θi 1 [BU j (A σb)u j T j ] s i = θ 1 i (A σb) [ (A σb) 1 BU j U j T j ] si = θ 1 i β j (e j s i )(A σb)u j+1 which implies r i is small whenever β j (e j s i)/θ i is small. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 73 / 80

Shift-and-invert Lanczos method for symmetric GEP 1: Given starting vector t, compute q = Bt and β 0 = q t. 2: for j = 1, 2,..., do 3: Compute w j = q/β j 1 and u j = t/β j 1. 4: Solve linear system (A σb)t = w j. 5: Set t := t β j 1 u j 1 ; compute α j = w j t and reset t := t α ju j. 6: B-reorthogonalize t to u 1,..., u j if necessary. 7: Compute q = Bt and β j = q t. 8: Compute approximate eigenvalues T j = S j Θ j S j. 9: Test for convergence. 10: end for 11: Compute approximate eigenvectors X = U j S j. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 74 / 80

Krylov-Schur restarting method (Locking and Purging) Theorem 27 (Schur decomposition) Let A C n n. Then there is a unitary matrix V such that A = V T V H, where T is upper triangular. The process of Krylov-Schur restarting: Compute the Schur decomposition of the Rayleigh quotient Move the desired eigenvalues to the beginning Throw away the rest of the decomposition T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 75 / 80

Use Lanczos process to generate the Lanczos decomposition of order m k + p (A σb) 1 BU k+p = U k+p T k+p + β k+p u k+p+1 e k+p. (23) Let T k+p = V k+p D k+p Vk+p [ [ ] V V k V p diag (Dk, D p ) k Vp ] (24) be a Schur decomposition of T k+p. The diagonal elements of D k and D p contain the k wanted and p unwanted Ritz values, respectively. T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 76 / 80

Substituting (24) into (23), it holds that (A σb) 1 B(U k+p V k+p ) = (U k+p V k+p )(V k+p T k+pv k+p ) + β k+p u k+p+1 (e k+p V k+p), which implies that (A σb) 1 BŨk = ŨkD k + u k+p+1 t k = [ Ũ k ũ k+1 ] [ D k t k ] is a Krylov decomposition of order k where Ũk U k+p V k, ũ k+1 = u k+p+1 and [ t k, ] t p βk+p e k+p V k+p. The new vectors ũ k+2,..., ũ k+p+1 are computed sequentially starting from ũ j+2 with (A σb) 1 BŨk+1 = [ ] Ũ k+1 ũ k+2 D k t k t k α k+1 0 βk+1, T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 77 / 80

or equivalently for (A σb) 1 Bũ k+1 = Ũkt k + α k+1 ũ k+1 + β k+1 ũ k+2, α k+1 = ũ k+1 B(A σb) 1 Bũ k+1. Consequently a new Krylov decomposition of order k + p can be generated by (A σb) 1 BŨk+p = Ũk+p T k+p + β k+p ũ k+p+1 e k+p, (25) where T k+p = D k t k t k α k+1 βk+1 β k+1 α k+2......... βk+p 1 β k+p 1 α k+p. (26) T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 78 / 80

Shift-and-invert Krylov-Schur method for solving Ax = λbx Build an initial Lanczos decomposition of order k + p: (A σb) 1 BU k+p = U k+p T k+p + β k+p u k+p+1 e k+p. repeat Compute Schur decomposition T k+p = [ V k V p ] diag ( Dk, D p ) [ Vk V p ]. Set U k := U k+p V k, u k+1 := u k+p+1 and t k := β k+pe k+p V k to get (A σb) 1 BU k = U k D k + u k+1 t k. for j = k + 1,..., k + p do Solve linear system (A σb)q = Bu j. if j = k + 1 then Set q := q U k t k. else Set q := q β j 1 u j 1. end if Compute α j = u j Bq and reset q := q α j u j. B-reorthogonalize q to u 1,..., u j 1 if necessary. Compute β j = q Bq and u j+1 = q/β j. end for Decompose T k+p = S k+p Θ k+p Sk+p to get the approximate eigenvalues. Test for convergence. until all the k wanted eigenpairs are convergent T.-M. Huang (Taiwan Normal University) Krylov Subspace Methods for Large/Sparse Eigenvalue Problems April 17, 2012 79 / 80