Large-scale eigenvalue problems

Transcription

1 ELE 538B: Mathematics of High-Dimensional Data Large-scale eigenvalue problems Yuxin Chen Princeton University, Fall 208

2 Outline Power method Lanczos algorithm Eigenvalue problems 4-2

3 Eigendecomposition Consider a symmetric matrix A R n n, where n is large How to compute the eigenvalues and eigenvectors of A efficiently? hopefully accomplished via a few matrix-vector products Eigenvalue problems 4-3

4 Power method

5 Power iteration q t = Aq t, t =, 2, Aq t 2 }{{} re-normalization each iteration consists of a matrix-vector product equivalently, q t = A t q 0 2 A t q 0 Eigenvalue problems 4-5

6 Example Consider A = = q t = [ 2 ] and q 0 = A t q 0 = [ A t A t q 0 = q 0 2 [ ] [ 2 t, then ] 2 t 2 2t + 2 2t + ] [ ] 0 }{{} leading eigenvector of A as t Eigenvalue problems 4-6

7 Power method Algorithm 4. Power method : initialize q 0 random unit vector 2: for t =, 2, do 3: q t = Aq t 2 Aq t 4: ˆλ(t) = q t Aq t (power iteration) q t : estimate of the leading eigenvector of A (t) ˆλ : estimate of the leading eigenvalue of A Eigenvalue problems 4-7

8 Convergence of power method A R n n : eigenvalues λ > λ 2 λ n ; eigenvectors u,, u n Theorem 4. (Convergence of power method) If λ > λ 2 λ n and set ν = q 0 u, then ˆλ(t) λ (λ λ n ) ν2 ν 2 ( ) 2t λ2 λ Eigenvalue problems 4-8

9 Write q 0 = n i= ν iu i, then Proof of Theorem 4. A t q 0 = n λ t iu i u i q 0 = i= = A t q 0 2 = n λ t iν i u i i= n λ t 2 iν i u i = n i= Since q t = A t q 0 2 A t q 0 and A is symmetric, we get ˆλ (t) = q t Aq t = = = A t q 0 2 q0 A 2t+ q 0 2 ( n n q i= λ2t i ν2 0 i n i= λ2t i ν2 i n i= i= λ 2t+ i νi 2 i= λ 2t+ i u i u i λ 2t i ν2 i Eigenvalue problems 4-9 ) q 0

10 As a consequence, ˆλ(t) λ = n i= λ2t i ν2 i as claimed = Proof of Theorem 4. (cont.) n i= λ2t i ν2 i n λ λ n λ 2t ν2 i=2 λ λ n λ 2t λ 2t ν2 2 n i= n i=2 λ 2t+ i νi 2 λ 2t n λ λ 2t i= i (λ λ i )νi 2 i ν 2 i λ 2t i ν 2 i (since λ λ i λ λ n ) n i=2 ν 2 i = λ λ n λ 2t λ 2t ν2 2 ( ν) 2 (since i ν 2 i = (as q 0 2 = )) Eigenvalue problems 4-0

11 Block power method Computing the top-r eigen-subspace: Algorithm 4.2 Power method : initialize Q 0 R n r random orthonormal matrix 2: for t =, 2, do 3: Z t = AQ t 4: compute QR decomposition Z t = Q t R t, where Q t R n r is orthonormal and R t R r r is upper-triangular use QR decomposition to reorthogonalize power iterates Eigenvalue problems 4-

12 Lanczos algorithm

13 Key idea : reduction to tridiagonal form Intermediate step } {{ } A find orthonormal Q }{{} T =Q AQ (tridiagonal) eigendecomposition of a tridiagonal matrix can be performed efficiently (via a number of specialized algorithms), due to its special structure Eigenvalue problems 4-3

14 Key idea 2: tridiagonalization and Krylov subspaces One way to tridiagonalize A is to compute an orthonormal basis of certain subspaces, defined as follows Krylov subspaces generated by A R n n and b R n are defined as K t := span { b, Ab,, A t b }, t =,, n Krylov matrices K t := [ b, Ab,, A t b ], t =,, n Eigenvalue problems 4-4

15 Key idea 2: tridiagonalization and Krylov subspaces Lemma 4.2 If Q t := [q,, q t ] forms an orthonormal basis of K t, t n. Then T t := Q t AQ t is tridiagonal, t n tridiagonalization can be carried out by successively computing the orthonormal basis of Krylov subspaces {K t } t=,2, Eigenvalue problems 4-5

16 Proof of Lemma 4.2 For any i > j +, (T t ) i,j = q i, Aq j Since Q j is orthonormal basis of span{b, Ab,, A j b}, we have q j span{b, Ab,, A j b} = Aq j span{ab,, A j b} span{q,, q j+ } Since i > j +, one has q i {q,, q j+ } and hence (T t ) i,j = q i, Aq j = 0 Similarly, (T t ) i,j = 0 if j > i +. This completes the proof Eigenvalue problems 4-6

17 A simple formula: 3-term recurrence Denote T = Q n AQ n = α β β α βn or AQ n = Q n T β n α n Exploiting the tridiagonal structure yields A[q,, q t ] = [q,, q t+ ] }{{}}{{} Q t Q t+ α β β α βt β t α t β t = Aq t = β t q t + α t q t + β t q t+ Eigenvalue problems 4-7

18 Lanczos iterations Aq t = β t q t + α t q t + β t q t+ 3-term recurrence says Aq t span{q t, q t, q t+ } this means α t = qt Aq t, since {q t, q t, q t+ } are }{{} projection of Aq t onto span(q t) orthonormal Since q t+ needs to have unit norm, one has q t+ normalize(aq t β t q t α t q t ) (direction of residual) β t = Aq t β t q t α t q t 2 (size of residual) Eigenvalue problems 4-8

19 Lanczos algorithm Algorithm 4.3 Lanczos algorithm : initialize β 0 = 0, q 0 = 0, q random unit vector 2: for t =, 2, do 3: α t = q t Aq t 4: β t = Aq t β t q t α t q t 2 5: q t+ = β t (Aq t β t q t α t q t ) each iteration only requires a matrix-vector product systematic construction of the orthonormal bases for successive Krylov subspaces Eigenvalue problems 4-9

20 Convergence of the Lanczos algorithm A R n n : eigenvalues λ λ n, eigenvectors u,, u n α β. β T t = α : eigenvalues θ θ t βt β t α t Theorem 4.3 (Kaniel-Paige convergence theory) Let ν = q u, ρ = λ λ 2 λ 2 λ n, and C t (x) be the Chebyshev polynomial of degree t. Then λ θ λ (λ λ n ) ν2 ν 2 ( Ct ( + 2ρ) ) 2 Eigenvalue problems 4-20

21 Convergence of the Lanczos algorithm Corollary 4.4 Let R = + 2ρ + 2 ρ 2 + ρ with ρ = λ λ 2 λ 2 λ n. We have λ θ 4( ν2 ) (λ λ n ) } ν 2 {{ } prefactor R 2(t ) }{{} convergence rate this follows immediately from the following fact ( R Ct ( 2 t + R (t )) 2 + 2ρ) = }{{ 4 } properties of Chebyshev polynomials R2(t ) 4 Eigenvalue problems 4-2

22 Power method vs. Lanczos algorithm Consider a case where λ 2 = λ n. Recall that ρ = λ λ 2 λ 2 λ n = λ λ 2 2λ 2 power method: convergence rate ( ) 2t λ2 = λ ( + 2ρ) 2t Lanczos algorithm: convergence rate ( + 2ρ + 2 ρ 2 + ρ) 2t if ρ, then + 2ρ + 2 ρ 2 + ρ + 4ρ 2( + 2ρ) if ρ, then + 2ρ + 2 ρ 2 + ρ + 2 ρ > + 2ρ outperforms the power method Eigenvalue problems 4-22

23 Proof of Theorem 4.3 It sufficies to prove the 2nd inequality. Recalling that T t = Q t AQ t, we have v T t v θ = max v:v 0 v v = max (Q t v) A(Q t v) w Aw v:v 0 (Q t v) = max (Q t v) w K t:w 0 w w For any w K t := {q, Aq,, A t q }, one can write it as P(A)q for some polynomial P( ) of degree t. This means θ = (P(A)q ) A(P(A)q ) max P( ) P t (P(A)q ) (P(A)q ) where P t is set of polynomials of degree t. If q = n i= ν iu i, then (P(A)q ) n A(P(A)q ) i= (P(A)q ) = ν2 i P2 (λ i )λ i (P(A)q ) n (check) i= ν2 i P2 (λ i ) n i=2 = λ ν2 i (λ λ i )P 2 (λ i ) ν 2P2 (λ ) + n i=2 ν2 i P2 (λ i ) n i=2 λ (λ λ n ) ν2 i P2 (λ i ) ν 2P2 (λ ) + n i=2 ν2 i P2 (λ i ) Eigenvalue problems 4-23

24 Proof of Theorem 4.3 (cont.) C0(x) C(x) C2(x) C3(x) C4(x) C Pick a polynomial P(x) that is large at x = λ. One choice is ( ) 2x λ2 λ n P(x) = C t λ 2 λ n where C t ( ) is the (t )-th Chebyshev polynomial generated by C t (x) = 2xC t (x) C t 2 (x), C 0 (x) =, C (x) = x These polynomials are bounded by on [, ], but grow rapidly outside Eigenvalue problems 4-24

25 Proof of Theorem 4.3 (cont.) Using boundedness of Chebyshev polynomial in [, ], we have n n i=2 (λ λ n ) ν2 i P2 (λ i ) ν 2P2 (λ ) + n i=2 ν2 i P2 (λ i ) (λ i=2 λ n ) ν2 i ν 2P2 (λ ) = (λ λ n ) ν2 ν 2 P2 (λ ) where the last identity follows since i ν2 i = (given q 2 = ). This yields as claimed θ λ (λ λ n ) ν2 ν 2 C 2 t ( + 2ρ) Eigenvalue problems 4-25

26 Warning: numerical instability The vanilla Lanczos algorithm (which is efficient with exact arithmetic) is very sensitive to round-off issues orthogonality of {q,, q t } might be lost quickly eigenvalues might be duplicated Many variations have been proposed to prevent loss of orthogonality, and to remove spurious eigenvalues Eigenvalue problems 4-26

27 Reference [] Numerical linear algebra, L. Trefethen, D. Bau, SIAM, 997. [2] An iteration method for the solution of the eigenvalue problem of linear differential and integral operators, C. Lanczos, 950. [3] EE38V lecture notes, S. Sanghavi, C. Caramanis, UT Austin. [4] Matrix computations, G. Golub, C. Van Loan, JHU Press, 202. [5] Estimates for some computational techniques in linear algebra, S. Kaniel, Mathematics of Computation, 966. [6] The computation of eigenvalues and eigenvectors of very large sparse matrices, C. Paige, 97. Eigenvalue problems 4-27