Orthogonal Eigenvectors and Gram-Schmidt

Transcription

1 Orthogonal Eigenvectors and Gram-Schmidt Inderjit S. Dhillon The University of Texas at Austin Beresford N. Parlett The University of California at Berkeley Joint GAMM-SIAM Conference on Applied Linear Algebra University of Düsseldorf, Germany July 25, 2006

2 One of FOUR main papers on this work Algorithm MR 3 or MRRR Acronym for Multiple Relatively Robust Representations Accurate but turgid title Jim Demmel has a more catchy title... Guiding Principle: No Gram-Schmidt

3 Excerpt from the Introduction

4 Excerpt from the Introduction What this talk will not do

5 Excerpt from the Introduction What this talk will not do Roundoff Error Analysis

6 Excerpt from the Introduction What this talk will not do Roundoff Error Analysis Theorems, Proofs

7 Excerpt from the Introduction What this talk will not do Roundoff Error Analysis Theorems, Proofs Performance Numbers

8 Excerpt from the Introduction What this talk will not do Roundoff Error Analysis Theorems, Proofs Performance Numbers Stick closely to the paper

9 Diagonal of the Inverse Let J be a tridiagonal that is irreducible and invertible Not necessarily symmetric

10 Diagonal of the Inverse Let J be a tridiagonal that is irreducible and invertible Not necessarily symmetric Perform triangular factorization down and up (no pivoting) J = L + D + U + = U D L D + are forward pivots while D are backward pivots

11 Diagonal of the Inverse Let J be a tridiagonal that is irreducible and invertible Not necessarily symmetric Perform triangular factorization down and up (no pivoting) J = L + D + U + = U D L D + are forward pivots while D are backward pivots Is there a relation between D + and D?

12 Diagonal of the Inverse Let J be a tridiagonal that is irreducible and invertible Not necessarily symmetric Perform triangular factorization down and up (no pivoting) J = L + D + U + = U D L D + are forward pivots while D are backward pivots Is there a relation between D + and D? Beautiful identity: D + + D = diag (J) + diag (J 1 ) 1

13 FOUR New Ideas

14 FOUR New Ideas Replace the tridiagonal with a bidiagonal

15 FOUR New Ideas Replace the tridiagonal with a bidiagonal Shift close to clusters, but with differential transforms

16 FOUR New Ideas Replace the tridiagonal with a bidiagonal Shift close to clusters, but with differential transforms Twist, again with differential transforms

17 FOUR New Ideas Replace the tridiagonal with a bidiagonal Shift close to clusters, but with differential transforms Twist, again with differential transforms Analyze with a Representation Tree

18 Difficulties All eigenvalues of T are easily computed in O(n 2 ) time Given ˆλ, inverse iteration computes the eigenvector: (T ˆλI)x i+1 = x i, i = 0,1, 2,... Costs O(n) per iteration Typically, 1-3 iterations are enough BUT, inverse iteration only guarantees T ˆv ˆλˆv = O(ε T )

19 ¾ Fundamental Limitations Gap Theorem : sin (v, ˆv) T ˆv ˆλˆv Gap(ˆλ) Gap(ˆλ) can be small : 1 ε 1 ε 1 1 ε 2 ε 2 1 ε 3 ε 3 1 When eigenvalues are close, independently computed eigenvectors WILL NOT be mutually orthogonal

20 Example from Quantum Chemistry Symmetric positive definite eigenproblem, n = 966 Occurs in Møller-Plesset theory in the modeling of biphenyl Eigenvalue Distribution: Eigenvalue Eigenvalue Index LAPACK clusters numbered 1,...,939 and smaller ones

21 Example from Quantum Chemistry Plot of Absgap(i) = log 10 (min(λ i+1 λ i, λ i λ i 1 )/ T ) versus i : 0 2 Absolute Eigenvalue Gaps Eigenvalue Index Plot of Relgap(i) = log 10 (min(λ i+1 λ i, λ i λ i 1 )/ λ i ) versus i : 0 2 Relative Eigenvalue Gaps Eigenvalue Index

22 First Steps Factor T = LDL T Compute eigenvalues of LDL T by dqds or bisection For each eigenvalue λ, compute eigenvector by inverse iteration

23 Computing Eigenvector #1 ˆλ 1 = , ˆλ 2 = , ˆλ 3 = Factor T 1 = LDL T ˆλ 1 I = L + D + L T + = U D U T (up & down) Compute γ(i) = D + (i) + D (i) T 1 (i,i) log10( gamma ) i Solve T 1 z 1 = γ r e r, where γ r = min k γ k z i

24 Computing Eigenvector #2 ˆλ 1 = , ˆλ 2 = , ˆλ 3 = Factor T 2 = LDL T ˆλ 2 I = L + D + L T + = U D U T (up & down) Compute γ(i) = D + (i) + D (i) T 2 (i,i) log10( gamma ) i Solve T 2 z 2 = γ r e r, where γ r = min k γ k z i

25 Accuracy of the vectors z T 1 z 2 < 2ε Residual norms are < ε T In general, min k γ k ε ˆλ Dot Product between two vectors ε ˆλ Gap If two eigenvalues share d leading digits, then dot product 10 d ε

26 Smaller Relative Gaps ˆλ 280 to ˆλ 303 are : Some eigenvalues agree in 5 digits

27 Smaller Relative Gaps ˆλ 280 to ˆλ 303 are : Some eigenvalues agree in 5 digits Compute new shifted representations: LDL T I = L 1 D 1 L T

28 Smaller Relative Gaps ˆλ 280 to ˆλ 303 are : Some eigenvalues agree in 5 digits Compute new shifted representations: LDL T I = L 1 D 1 L T 1 L 1 D 1 L T I = L 2 D 2 L T

29 The computed vectors ˆλ = log10( gamma ) i z i

30 The computed vectors ˆλ = log10( gamma ) i z i

31 The computed vectors ˆλ = log10( gamma ) i z i Maximum Dot Product < 3ε.

32 Key Idea 1 Replace the tridiagonal with a bidiagonal Bidiagonal Factorization T LDL T

33 Key Idea 1 Replace the tridiagonal with a bidiagonal Bidiagonal Factorization Relative condition number T LDL T relcond(λ) = vt L D L T v v T LDL T v

34 Key Idea 1 Replace the tridiagonal with a bidiagonal Bidiagonal Factorization Relative condition number T LDL T relcond(λ) = vt L D L T v v T LDL T v Seminal 1991 Demmel-Kahan paper (SIAG/LA Prize) Small relative changes in the entries of a bidiagonal cause small relative changes in all its singular values

35 Key Idea 1 Replace the tridiagonal with a bidiagonal Bidiagonal Factorization Relative condition number T LDL T relcond(λ) = vt L D L T v v T LDL T v Seminal 1991 Demmel-Kahan paper (SIAG/LA Prize) Small relative changes in the entries of a bidiagonal cause small relative changes in all its singular values L and D almost always define the small eigenvalues of LDL T to high relative accuracy Even in the face of element growth!

36 Key Idea 2 Shift close to clusters Connection between residuals and orthogonality. Let r = Ax xˆλ 0, s = Ay yˆµ 0,

37 Key Idea 2 Shift close to clusters Connection between residuals and orthogonality. Let r = Ax xˆλ 0, s = Ay yˆµ 0, Then x T y(ˆµ ˆλ) = y T r x T s

38 Key Idea 2 Shift close to clusters Connection between residuals and orthogonality. Let r = Ax xˆλ 0, s = Ay yˆµ 0, Then x T y(ˆµ ˆλ) = y T r x T s Suppose, by some miracle, r ε 1 ˆλ and s ε 2 ˆµ, then

39 Key Idea 2 Shift close to clusters Connection between residuals and orthogonality. Let r = Ax xˆλ 0, s = Ay yˆµ 0, Then x T y(ˆµ ˆλ) = y T r x T s Suppose, by some miracle, r ε 1 ˆλ and s ε 2 ˆµ, then x T y max(ε 1, ε 2 ) ˆλ ˆµ ˆλ + ˆµ Thus, orthogonality depends on the relative separation

40 Key Idea 2 Shift close to clusters Connection between residuals and orthogonality. Let r = Ax xˆλ 0, s = Ay yˆµ 0, Then x T y(ˆµ ˆλ) = y T r x T s Suppose, by some miracle, r ε 1 ˆλ and s ε 2 ˆµ, then x T y max(ε 1, ε 2 ) ˆλ ˆµ ˆλ + ˆµ Thus, orthogonality depends on the relative separation Relative Gaps can be made larger by shifting LDL T ξi = L D L T Different representations for different clusters! Essential to use differential transforms

41 Differential Transforms LDL T ˆλI = L + D + L T + Simple qd : D + (1) := d 1 ˆλ for i = 1,n 1 L + (i) := (d i l i )/D + (i) D + (i + 1) := d i l 2 i + d i+1 L + (i)d i l i ˆλ end for Differential qd : s 1 := ˆλ for i = 1,n 1 D + (i) := s i + d i L + (i) := (d i l i )/D + (i) s i+1 := L + (i)l i s i ˆλ end for D + (n) := s n + d n

42 ¾ Key Idea 3 Twist, again with differential transformations Godunov et al. [1985], Fernando [1995] Compute the appropriate Twisted Factorization : LDL T ˆλI = N r D r N T r, where N r = x x x.. x x x.. x x x x x Solve for z, N r D r N T r z = γ r e r ( N T r z = e r ) : z(i) = 1, i = r, L + (i) z(i + 1), i = r 1,...,1, U (i 1) z(i 1), i = r + 1,...,n.

43 Key Idea 4 Representation Tree Eigenvalues: ε, 1 + ε, ε, 2

44 Key Idea 4 Representation Tree Eigenvalues: ε, 1 + ε, ε, 2 Extra representation needed at σ = 1: L p D p L T p I = L 0 D 0 L T 0

45 Key Idea 4 Representation Tree Eigenvalues: ε, 1 + ε, ε, 2 Extra representation needed at σ = 1: L p D p L T p I = L 0 D 0 L T 0 Following Representation Tree captures the steps of the algorithm: ({N (1) 1, 1},{1}) ({L p,d p },{1,2,3,4}) ε 1 2 ε ({L 0,D 0 },{2,3}) ({N (3) 4, 4},{4}) 2 ε ({N (4) 2, 2},{2}) ({N (4) 3, 3},{3})

46 Caveats BLAS Complexity is O(nk) to compute k eigenpairs However, all operations are BLAS 1 Closest competitor D&C O(n 3 ), but BLAS 3 Very tight eigenvalue clusters C. Vömel s torture tests

47 Wilkinson s Matrix W + 21 λ 20 and λ 21 are identical to working precision

48 Wilkinson s Matrix W + 21 λ 20 and λ 21 are identical to working precision Form new representation: L p D p L T p ˆλ 21 I = L 0 D 0 L T 0

49 Wilkinson s Matrix W + 21 λ 20 and λ 21 are identical to working precision Form new representation: L p D p L T p ˆλ 21 I = L 0 D 0 L T 0 Roundoff to the rescue: λ 20 (L 0 D 0 L T 0 ) & λ 21 (L 0 D 0 L T 0 ) no digits in common! &

50 Wilkinson s Matrix W + 21 λ 20 and λ 21 are identical to working precision Form new representation: L p D p L T p ˆλ 21 I = L 0 D 0 L T 0 Roundoff to the rescue: λ 20 (L 0 D 0 L T 0 ) & λ 21 (L 0 D 0 L T 0 ) no digits in common! & Computed Eigenvectors ˆv 20 and ˆv 21 (inner product is ):

51 Caveats BLAS Complexity is O(nk) to compute k eigenpairs However, all operations are BLAS 1 Closest competitor D&C O(n 3 ), but BLAS 3 Very tight eigenvalue clusters C. Vömel s torture tests 5 copies of W with glue of ε Required a tweak Perturb base representation