Solving Symmetric Semi-definite (ill-conditioned) Generalized Eigenvalue Problems

Transcription

1 Solving Symmetric Semi-definite (ill-conditioned) Generalized Eigenvalue Problems Zhaojun Bai University of California, Davis Berkeley, August 19, 2016

2 Symmetric definite generalized eigenvalue problem Symmetric definite generalized eigenvalue problem Ax = λbx where A T = A and B T = B > 0 Eigen-decomposition AX = BXΛ where Λ = diag(λ 1, λ 2,..., λ n ) X = (x 1, x 2,..., x n ) X T BX = I. Assume λ 1 λ 2 λ n

3 LAPACK solvers LAPACK routines xsygv, xsygvd, xsygvx are based on the following algorithm (Wilkinson 65): 1. compute the Cholesky factorization B = GG T 2. compute C = G 1 AG T 3. compute symmetric eigen-decomposition Q T CQ = Λ 4. set X = G T Q xsygv[d,x] could be numerically unstable if B is ill-conditioned: and λ i λ i p(n)( B 1 2 A 2 + cond(b) λ i ) ɛ θ( x i, x i ) p(n) B 1 2 A 2 (cond(b)) 1/2 + cond(b) λ i specgap i User s choice between the inversion of ill-conditioned Cholesky decomposition and the QZ algorithm that destroys symmetry ɛ

4 Algorithms to address the ill-conditioning 1. Fix-Heiberger 72 (Parlett 80): explicit reduction 2. Chang-Chung Chang 74: SQZ method (QZ by Moler and Stewart 73) 3. Bunse-Gerstner 84: MDR method 4. Chandrasekaran 00: proper pivoting scheme 5. Davies-Higham-Tisseur 01: Cholesky+Jacobi 6. Working notes by Kahan 11 and Moler 14

5 This talk Three approaches: 1. A LAPACK-style implementation of Fix-Heiberger algorithm 2. An algebraic reformulation 3. Locally accelerated block preconditioned steepest descent (LABPSD)

6 This talk Three approaches: 1. A LAPACK-style implementation of Fix-Heiberger algorithm Status: beta-release 2. An algebraic reformulation Status: completed basic theory and proof-of-concept 3. Locally accelerated block preconditioned steepest descent (LABPSD) Status: published two manuscripts, software to be released

7 A LAPACK-style implementation of Fix-Heiberger algorithm (with C. Jiang)

8 A LAPACK-style solver xsygvic: computes ε-stable eigenpairs when B T = B 0 wrt a prescribed threshold ε. Implementation is based on Fix-Heiberger s algorithm, and organized in three phases. Given the threshold ε, xsygvic determines: 1. A λb is regular and has k (0 k n) ε-stable eigenvalues or 2. A λb is singular. The new routine xsygvic has the following calling sequence: xsygvic( ITYPE, JOBZ, UPLO, N, A, LDA, B, LDB, ETOL, & K, W, WORK, LDWORK, WORK2, LWORK2, IWORK, INFO )

9 xsygvic Phase I 1. Compute the eigenvalue decomposition of B (xsyev): B (0) = Q T 1 BQ 1 = D = [ n1 n2 n 1 D (0) n 2 E (0) ], where diagonal entries of D: d 11 d d nn, and elements of E (0) are smaller than εd (0) Set E (0) = 0, and update A and B (0) : A (1) = R T 1 Q T 1 AQ 1 R 1 = [ n 1 n 2 n 1 A (1) 11 A (1) 12 n 2 A (1)T 12 A (1) 22 ] and B (1) = R T 1 B (0) R 1 = n 1 [ n1 n2 I n 2 0 ], where R 1 = diag((d (0) ) 1/2, I)

10 xsygvic Phase I 3. Early exit B is ε-well-conditioned. A λb is regular and has n ε-stable eigenpairs (Λ, X): A (1) U = UΛ (xsyev). X = Q1R 1U

11 xsygvic Phase I: performance profile Test matrices A = Q A D A Q T A and B = Q BD B Q T B where QA, Q B are random orthogonal matrices; DA is diagonal with 1 < D A(i, i) < 1, i = 1,..., n; DB is diagonal with 0 < ε < D B(i, i) < 1, i = 1,..., n; 12-core on an Intel Ivy Bridge processor (Edison@NERSC) Runtime of xsygv/xsygvic with B>0 DSYGV DSYGVIC Percentage of sections in xsygvic Runtime (seconds) Percentage Matrix order n Matrix order n xsyev of B xsyev of A Others

12 xsygvic Phase II 1. Compute the eigendecomposition of (2,2)-block A (1) 22 of A(1) (xsyev): A (2) 22 = Q(2)T 22 A (1) 22 Q(2) 22 = [ n 3 n 4 n 3 D (2) n 4 E (2) ] where eigenvalues are ordered such that d (2) 11 d(2) 22 d(2) n 2n 2, and elements of E (2) are smaller than ε d (2) Set E (2) = 0, and update A (1) and B (1) : where Q 2 = diag(i, Q (2) 22 ). A (2) = Q T 2 A (1) Q 2, B (2) = Q T 2 B (1) Q 2 3. Early exit When A (1) 22 is a ε-well-conditioned matrix. A λb is regular and has n 1 ε-stable eigenpairs (Λ, X): A (2) U = B (2) UΛ (Schur complement and xsyev) X = Q1R 1Q 2U.

13 xsygvic Phase II (backup) where A (2) U = B (2) UΛ (1) A (2) = [ n 1 n 2 n 1 A (2) 11 A (2) 12 n 2 A (2)T 12 D (2) ] and B (2) = n 1 [ n1 n2 I n 2 0 ]. Let U = [ n1 n 1 U 1 n 2 U 2 The eigenvalue problem (1) becomes ( ) F (2) U 1 = A (2) 11 A(2) 12 (D(2) ) 1 A (2)T 12 U 1 = U 1 Λ ] (xsyev) U 2 = (D (2) ) 1 (A (2) 12 )T U 1

14 xsygvic Phase II: performance profile Accuracy: 1. If B 0 has n 2 zero eigenvalues: xsygv stops, the Cholesky factorization of B could not be completed. xsygvic successfully computes n n2 ε-stable eigenpairs. 2. If B has n 2 small eigenvalues about δ, both xsygv and xsygvic work, but produce different quality numerically. 1 n = 1000, n2 = 100, δ = and ε = Res1 Res2 DSYGV 3.5e-8 1.7e-11 DSYGVIC 9.5e e-12 n = 1000, n2 = 100, δ = and ε = Res1 Res2 DSYGV 3.6e-6 1.8e-10 DSYGVIC 1.3e e-14 1 Res1 = A X B X Λ F /(n A F X F ) and Res2 = X T B X I F /( B F X 2 F ).

15 xsygvic Phase II: performance profile Timing: Test matrices A = Q A D A Q T A and B = Q BD B Q T B where QA, Q B are random orthogonal matrices; DA is diagonal with 1 < D A(i, i) < 1, i = 1,..., n; DB is diagonal with 0 < D B(i, i) < 1, i = 1,..., n and n 2/n D B(i, i) < ε. 12-core on an Intel Ivy Bridge processor (Edison@NERSC) Runtime of xsygv/xsygvic xsygv xsygvic Percentage of sections in xsygvic Runtime (seconds) Percentage Matrix order n Matrix order n xsyev(b) xsyev(f (2) ) Others

16 xsygvic Phase II: performance profile Why is the overhead ratio of xsygvic lower? Performance of xsygv varies depending on the percentage of zero eigenvalues of B. For example, for n = 4000 on a 12-core processor execution: Runtime of DSYGV for n=4000 DSYEV Others Runtime (seconds) % 20% 33% 50% Percentage of "zero" eigenvalues of B

17 xsygvic Phase III 1. A (2) and B (2) can be written as 3 by 3 blocks: A (2) = n 1 n 3 n 4 n 1 A (2) 11 A (2) 12 A (2) 13 n 3 n 4 A (2)T 12 D (2) A (2)T 13 0 and B (2) = n 1 n 1 n 3 n 4 I n 3 0 n 4 0 where n 3 + n 4 = n Reveal the rank of A (2) 13 by QR decomposition with pivoting: A (2) 13 P (3) 13 = Q(3) 13 R(3) 13 where R (3) 13 = [ n 4 n 4 A (3) 14 n 5 0 ]

18 xsygvic Phase III 3. Final exit When n 1 > n 4 and A (2) 13 is full rank,2 then A λb is regular and has n 1 n 4 ε-stable eigenpairs (Λ, X): A (3) U = B (3) UΛ X = Q1R 1Q 2Q 3U. 2 All the other cases either lead A λb to be singular or regular but no finite eigenvalues.

19 xsygvic Phase III (backup) Update where A (3) U = B (3) UΛ (2) A (3) = Q T 3 A (2) Q 3 and B (3) = Q T 3 B (2) Q 3 Q 3 = Write A (3) and B (3) as 4 4 blocks: n 1 n 3 n 4 n 1 Q (3) 13 n 3 I n 4 P (3) 13 n 4 n 5 n 3 n 4 n 4 A (3) 11 A (3) 12 A (3) 13 A (3) 14 A (3) n = 5 (A (3) 12 )T A (3) 22 A (3) 23 0 n 3 (A (3) 13 )T (A (3) 23 )T D (2) 0 n 4 (A (3) 14 )T n 4 n 5 n 3 n 4 n 4 I, B (3) n = 5 I n 3 0, n 4 0 where n 1 = n 4 + n 5 and n 2 = n 3 + n 4.

20 xsygvic Phase III (backup) Let U = n 5 n 4 U 1 n 5 U 2 n 3 U 3 n 4 U 4 then the eigenvalue problem (2) becomes: U 1 = 0 ( ) A (3) 22 A(3) 23 (D(3) ) 1 A (3)T 23 U 2 = U 2 Λ (xsyev) U 3 = (D (2) ) 1 A (3)T 23 U 2 U 4 = (A (3) 14 ) 1 ( A (3) 12 U 2 + A (3) 13 U 3 )

21 xsygvic Phase III: performance profile Test case (Fix-Heiberger 72) 1. Consider 8 8 matrices: where and A = Q T HQ and B = Q T SQ, H = S = diag[1, 1, 1, 1, δ, δ, δ, δ] As δ 0, λ = 3, 4 are the only stable eigenvalues of A λb.

22 xsygvic Phase III: performance profile 2. The computed eigenvalues when δ = : λ i eig(a,b, chol ) DSYGV DSYGVIC(ε = ) e e e e e e e e e e e e e e e e e e+16

23 An algebraic reformulation (with H. Xie)

24 Symmetric semi-definite pencil Symmetric semi-definite pencil: A λb, with A T = A and B T = B 0

25 Symmetric semi-definite pencil Canonical form. There exists a nonsingular matrix W R n n such that W T AW = 2n 1 Λ 1 2n 1 r n 2 s r Λ 2 n 2 Λ 3 s 0 and W T BW = 2n 1 Ω 1 r 2n 1 r n 2 s n 2 0 s 0 I where Λ 1 = I n1 K, Λ 2 = diag(λ i ), Λ 3 = diag(±1), Ω 1 = I n1 T and K = [ ] [ 1 0, T = 0 0 ]

26 Algebraic reformulation Theorem [Xie and B. 16]. Suppose that the symmetric semi-definite pencil A λb is regular and simultaneously diagonalizable with a congruence transformation. Given a symmetric positive definite matrix H R k k and µ R, let us define à = A + µ(az)h(az) T, B = B + (AZ)H(AZ) T, where Z R n k spans the nullspace of B. Then 3 (1) The pencil à λ B is symmetric definite, (2) λ(ã, B) = λ f (A, B) λ(µh + (Z T AZ) 1, H) By appropriately chosen H and µ, one can compute the k smallest (finite) eigenvalues of A λb directly, say by LOBPCG. 3 Notations: λ(a, B) denotes the set of eigenvalues of a pencil A λb. λ f (A, B) denotes the set of all finite eigenvalues of A λb.

27 Algebraic reformulation A test case from structure dynamics LOBPCG: Residual norms st eigenvalue 2nd eigenvalue 3rd eigenvalue 4th eigenvalue 5th eigenvalue Iterations

28 Algebraic reformulation A test case from structure dynamics Algebraic reformulation + LOBPCG without preconditioning Residual norms st eigenvalue 2nd eigenvalue rd eigenvalue 4th eigenvalue 5th eigenvalue Iterations

29 Algebraic reformulation A test case from structure dynamics Algebraic reformulation + LOBPCG with preconditioning Residual norms st eigenvalue 2nd eigenvalue 3rd eigenvalue 4th eigenvalue 5th eigenvalue Iterations

30 A locally accelerated BPSD (LABPSD)

31 Ill-conditioned GSEP GSEP: Hu i = λ i Su i 1. Matrices H and S are ill-conditioned e.g., cond(h), cond(s) = O(10 10 ) 2. Share a near-nullspace span{v } e.g., HV = SV = O(10 4 ) 3. No obvious spectrum gap between eigenvalues of interest and the rest e.g., lowest 8

32 PSDid PSDed: Preconditioned Steepest Descent with implicit deflation Assume the first i 1 eigenpairs (λ 1, u 1 ),..., (λ i 1, u i 1 ) computed, and denote U i 1 = [u 1, u 2,..., u i 1 ]. PSDid computes the i-th eigenpair (λ i, u i ) 0 initialize (λ i;0, u i;0 ) 1 for j = 0, 1,... until convergence 2 compute r i;j = Hu i;j λ i;j Su i;j 3 precondition p i;j = K i;j r i;j 4 (γ i, w i ) = RR(H, S, Z), where Z = [U i 1 u i;j p i;j ] 5 λ i;j+1 = γ i, u i;j+1 = Zw i..., [Faddeev/Faddeeva 63],..., [Longsine/McCormick 80] for K i;j = I,...

33 PSDid assumptions 1. initialize u i;0 such that Ui 1 H Su i;0 = 0 and u i;0 S = 1 2. λ i;0 = ρ(u i;0 ), Rayleigh quotient 3. the preconditioners K i;j are effective positive definite, namely, Ki;j d (Ui 1) c T SK i;j SUi 1 c > 0, where Ui 1 c = [u i, u i+1,..., u n ].

34 PSDid properties 1. Z is of full column rank 2. Ui 1 H Su i;j+1 = 0 and u i;j+1 S = 1 3. λ i λ i;j+1 < λ i;j 4. λ i;j λ i;j+1 g 2 + φ 2 g = step size > 0, λ i:j+1 λ i:j λ i-1 λ i step size λ i+1 5. p i;j = K i;j r i;j is an ideal search direction if p i;j satisfies U T S(u i;j + p i;j ) = (,...,, ξ i, 0,..., 0) T and ξ i 0. (3) It implies that λ i;j+1 = λ i.

35 PSDid convergence If λ i < λ i;0 < λ i+1 and sup j cond(ki;j d ) = q <, then the sequence {λ i;j } j is strictly decreasing and bounded from below by λ i, i.e., and as j, λ i;0 > λ i;1 > > λ i;j > λ i;j+1 > λ i 1. λ i;j λ i 2. u i;j converges to u i directionally: r i;j S 1 = Hu i;j λ i;j Su i;j S 1 0

36 PSDid convergence rate Let ɛ i;j = λ i;j λ i, then [ + τ ] 2 θ i;j ɛ i;j ɛ i;j+1 1 τ( ɛ i;j θ i;j ɛ i;j + δ i;j ɛ i;j ) provided that the i-th approximate eigenvalue λ i;j is localized, i.e. where = Γ γ Γ +γ and τ = 2 Γ +γ τ( θ i;j ɛ i;j + δ i;j ɛ i;j ) < 1, δ i;j = S 1 2 K i;j S 1 2 and θ i:j = S 1 2 K i;j MK i;j S 1 2 Γ and γ are largest and smallest pos. eigenvalues of K i;j M and M = P H i 1 (H λ is)p i 1 and P i 1 = I U i 1 U H i 1 S

37 PSDid convergence rate, cont d Remarks: 1. If K i;j = I, the convergence of SD proven in [Faddeev/Faddeeva 63,..., Longside/McCormick 80,...] 2. If i = 1 and K 1;j = K > 0, it is Samokish s theorem (1958), which is first and still sharpest quantitative analysis [Ovtchinnikov 06]. 3. Asymptotically, ɛ i;j+1 [ + O(ɛ 1/2 i;j ) ] 2 ɛi;j 4. Optimal K i;j : = 0 quadratic conv. 5. Semi-optimal K i;j : + τ θ i;j ɛ i;j 0 superlinear conv. 6. (Semi-)optimality depends on the eigenvalue distribution of K i;j M

38 Locally accelerated preconditioner Consider the preconditioner K i;j = ( H β i;j S ) 1 with β i;j = λ i;j c r i;j S 1 If 0 < i;j < min{ i, 0.1} and c > 3 i;j. Then 1. K i;j is effective positive definite 2. λ i;j is localized 3. + τ θ i;j ɛ i;j 0 Therefore, Ki;j is asymptotically optimal

39 PSDid LABPSD = Locally Accelerated BPSD 0 Initialize U m+l;0 = [ u 1;0 u 2;0... u m+l;0 ] 1 (Γ, W ) = RR(H, S, U m+l;0 ) 2 update Λ m+l;0 = Γ and U m+l;0 = U m+l;0 W 3 for j = 0, 1,..., do 4 compute R = HU m;j SU m;j Λ m;j [ r 1;j r 2;j... r m;j ] 5 if Res[Λ m;j, U m;j ] = max 1 i m Res[λ i;j, u i;j ] τ eig, break 6 for i = 1, 2,..., m if λ i;j is localized, then solve (H λ i;j S)p i;j = r i;j for p i;j 7 (Γ m+l, W m+l ) = RR(H, S, Z), where Z = [U m+l;j P j ] 8 update Λ m+l;j+1 = Γ m+l and U m+l;j+1 = ZW m+l 9 end 10 return {(λ i;j, u i;j )} m i=1 Note: A global preconditioner (H σs) 1 can be used to accelerate the localization and convergence of step 6.

40 Numerical example 1: Harmonic1D PUFE discretization for harmonic oscillator in 1D n = 112 for 6-digit accuracy of 4 smallest eigenvalues λ 1, λ 2, λ 3, λ 4 H and S are ill-conditioned cond(h) = and cond(s) = H and S share a near-nullspace span{v } HV = SV = O(10 5 ) and dim(v ) = 17 All computed λ 1, λ 2, λ 3, λ 4 have 6-digit accuracy DPSD Res LABPSD Res 1 Res Res 2 Res 3 relative residual 10 5 Res 3 Res 4 relative residual Res eigen iteration eigen iteration

41 Numerical example 2: CeAl-PUFE Metallic, triclinic CeAl, particularly challenging [Cai, B., Pask, Sukumar 13] n = 5336 from PUFE discretization of the Kohn-Sham equation H and S are ill-conditioned cond(h) = and cond(s) = H and S share a near-nullspace span{v } HV = SV = O(10 4 ) and dim(v ) = DPSD Res LABPSD Res 1 Res Res 2 Res 3 relative residual 10 5 Res 3 Res 4 relative residual Res eigen iteration eigen iteration