Cuppen s Divide and Conquer Algorithm

Transcription

1 Chapter 4 Cuppen s Divide and Conquer Algorithm In this chapter we deal with an algorithm that is designed for the efficient solution of the symmetric tridiagonal eigenvalue problem a b (4) x λx, b a bn We noticed from able 3 that the reduction of a full symmetric matrix to a similar tridiagonal matrix requires about 8 3 n3 while the tridiagonal QR algorithm needs an estimated 6n 3 floating operations (flops) to converge Because of the importance of this subproblem a considerable effort has been put into finding faster algorithms than the QR algorithms to solve the tridiagonal eigenvalue problem In the mid-98 s Dongarra and Sorensen [4] promoted an algorithm originally proposed by Cuppen [] his algorithm was based on a divide and conquer strategy However, it took ten more years until a stable variant was found by Gu and Eisenstat [5, 6] oday, a stable implementation of this latter algorithm is available in LAPACK [] 4 he divide and conquer idea Divide and conquer is an old strategy in military to defeat an enemy going back at least to Caesar In computer science, divide and conquer (D&C) is an important algorithm design paradigm It works by recursively breaking down a problem into two or more subproblems of the same (or related) type, until these become simple enough to be solved directly he solutions to the subproblems are then combined to give a solution to the original problem ranslated to our problem the strategy becomes b n Partition the tridiagonal eigenvalue problem into two (or more) smaller tridiagonal eigenvalue problems Solve the two smaller problems 3 Combine the solutions of the smaller problems to get the desired solution of the overall problem Evidently, this strategy can be applied recursively 77 a n

2 78 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM 4 Partitioning the tridiagonal matrix Partitioning the irreducible tridiagonal matrix is done in the following way We write (4) a b b a bm a b b m a m b m b m a m+ b m+ b m+ a m+ bn b a bm [ b n a n b m a m b m a m+ b m b m+ b m+ a m+ bn b n a n ] [ ] +ρuu ±em with u and ρ ±b m, e + ±b m b m b m ±b m where e m is a vector of length m n and e is a vector of length n m Notice that the most straightforward way to partition the problem without modifying the diagonal elements leads to a rank-two modification With the approach of (4) we have the original as a sum of two smaller tridiagonal systems plus a rank-one modification 43 Solving the small systems We solve the half-sized eigenvalue problems, (43) i Q i Λ i Q i, Q i Q i I, i, hese two spectral decompositions can be computed by any algorithm, in particular also by this divide and conquer algorithm by which the i would be further split It is clear that by this partitioning an large number of small problems can be generated that can be potentially solved in parallel For a parallel algorithm, however, the further phases of the algorithm must be parallelizable as well Plugging (43) into (4) gives [ ]([ )[ ] [ ] Q (44) ]+ρuu Q Λ +ρvv Q Q Λ with [ Q (45) v Q ] [ ±Q u e m Q e ] [ ±last row of Q first row of Q ]

3 44 DEFLAION 79 Now we have arrived at the eigenvalue problem (46) (D +ρvv )x λx, D Λ Λ diag(λ,,λ n ) hat is, we have to compute the spectral decomposition of a matrix that is a diagonal plus a rank-one update Let (47) D +ρvv QΛQ be this spectral decomposition hen, the spectral decomposition of the tridiagonal is [ ] [ ] Q (48) QΛQ Q Q Q Forming the product (Q Q )Q will turn out to be the most expensive step of the algorithm It costs n 3 +O(n ) floating point operations 44 Deflation here are certain solutions of (47) that can be given immediately, by just looking carefully at the equation If there are zero entries in v then we have ( (49) vi v e i ) (D +ρvv )e i d i e i hus, if an entry of v vanishes we can read the eigenvalue from the diagonal of D at once and the corresponding eigenvector is a coordinate vector If identical entries occur in the diagonal of D, say d i d j, with i < j, then we can find a plane rotation G(i,j,φ) (see (34)) such that it introduces a zero into the j-th position of v, Notice, that (for any ϕ), G v G(i,j,ϕ) v vi +v j i j G(i,j,ϕ) DG(i,j,ϕ) D, d i d j So, if there are multiple eigenvalues in D we can reduce all but one of them by introducing zeros in v and then proceed as previously in (49) When working with floating point numbers we deflate if (4) v i < Cε or d i d j < Cε, ( D +ρvv ) where C is a small constant Deflation changes the eigenvalue problem for D+ρvv into the eigenvalue problem for [ D +ρv (4) v ] O G (D+ρvv )G+E, E < Cε D O D + ρ v 4, where D has no multiple diagonal entries and v has no zero entries So, we have to compute the spectral decomposition of the matrix in (4) which is similar to a slight perturbation of the original matrix G is the product of Givens rotations

4 8 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM 44 Numerical examples Let us first consider uu hen a little Matlab experiment shows that Q Q with Q Here it is not possible to deflate Let us now look at an example with more symmetry, + + +uu

5 45 HE EIGENVALUE PROBLEM FOR D +ρvv 8 Now, Matlab gives Q Q with Q In this example we have three double eigenvalues Because the corresponding components of v (v i and v i+ ) are equal we define hen, G G(,4,π/4)G(,5,π/4)G(3,6,π/4) G Q Q G G Q Q G+G v(g v) D +G v(g v) herefore, (in this example) e 4, e 5, and e 6 are eigenvectors of D +G v(g v) D +G vv G corresponding to the eigenvalues d 4, d 5, and d 6, respectively he eigenvectors of corresponding to these three eigenvalues are the last three columns of Q G he eigenvalue problem for D+ρvv We know that ρ Otherwise there is nothing to be done Furthermore, after deflation, we know that all elements of v are nonzero and that the diagonal elements of D are all

6 8 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM distinct, in fact, d i d j > Cε We order the diagonal elements of D such that d < d < < d n Notice that this procedure permutes the elements of v as well Let (λ,x) be an eigenpair of (4) (D +ρvv )x λx hen, (43) (D λi)x ρvv x λ cannot be equal to one of the d i If λ d k then the k-th element on the left of (43) vanishes But then either v k or v x he first cannot be true for our assumption about v If on the other hand v x then (D d k I)x hus x e k and v e k v k, which cannot be true herefore D λi is nonsingular and (44) x ρ(λi D) v(v x) his equation shows that x is proportional to (λi D) v If we require x then (45) x (λi D) v (λi D) v Multiplying (44) by v from the left we get (46) v x ρv (λi D) v(v x) Since v x, λ is an eigenvalue of (4) if and only if Figure 4: Graph of + λ + λ λ λ λ λ (47) f(λ) : ρv (λi D) v ρ n k v k λ d k

7 45 HE EIGENVALUE PROBLEM FOR D +ρvv 83 his equation is called secular equation he secular equation has poles at the eigenvalues of D and zeros at the eigenvalues of D +ρvv Notice that f (λ) ρ n k v k (λ d k ) hus, the derivative of f is positive if ρ > wherever it has a finite value If ρ < the derivative of f is negative (almost) everywhere A typical graph of f with ρ > is depicted in Fig 4 (If ρ is negative the image can be flipped left to right) he secular equation implies the interlacing property of the eigenvalues of D and of D +ρvv, (48) d < λ < d < λ < < d n < λ n, ρ > or (49) λ < d < λ < d < < λ n < d n, ρ < So, we have to compute one eigenvalue in each of the intervals (d i,d i+ ), i < n, and a further eigenvalue in (d n, ) or (,d ) he corresponding eigenvector is then given by (45) Evidently, these tasks are easy to parallelize Equations (47) and (45) can also been obtained from the relations [ ] ρ v [ ] [ ] [ ρ ] ρv v λi D ρv I λi D ρvv I [ v (λi D) ] [ ] [ ρ v (λi D) v I λi D (λi D) v I hese are simply block LDL factorizations of the first matrix he first is the well-known one where the factorization is started with the (,) block he second is a backward factorization that is started with the (, ) block Because the determinants of the tridiagonal matrices are all unity, we have ] (4) ρ det(λi D ρvv ) ρ ( ρv (λi D) v)det(λi D) Denoting the eigenvalues of D +ρvv again by λ < λ < < λ n this implies (4) (λ λ j ) ( ρv (λi D) v) (λ d j ) j j ( ) n v n k ρ (λ d j ) λ d k k (λ d j ) ρ n j vk j k j k (λ d j ) Setting λ d k gives (4) (d k λ j ) ρvk j (d k d j ) j j i

8 84 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM or (43) v k ρ ρ (d k λ j ) j (d k d j ) j j i (d k λ j ) k j (d k d j ) k j ρ (d k λ j ) k j (d k d j ) k j (λ j d k ) jk jk+ > (d j d k ) (λ j d k )( ) n k+ jk jk+ herefore, the quantity on the right side is positive, so k (d k λ j ) n (λ j d k ) (44) v k j jk (d k d j ) (d j d k ) ρ k j jk+ (d j d k )( ) n k (Similar arguments hold if ρ < ) hus, we have the solution of the following inverse eigenvalue problem: Given D diag(d,,d n ) and values λ,,λ n that satisfy (48) Find a vector v [v,,v n ] with positive components v k such that the matrix D+vv has the prescribed eigenvalues λ,,λ n he solution is given by (44) he positivity of the v k makes the solution unique 46 Solving the secular equation In this section we follow closely the exposition of Demmel[3] We consider the computation of the zero of f(λ) in the interval (d i,d i+ ) We assume that ρ We may simply apply Newton s iteration to solve f(λ) However, if we look carefully at Fig 4 then we notice that the tangent at certain points in (d i,d i+ ) crosses the real axis outside this interval his happens in particular if the weights v i or v i+ are small herefore that zero finder has to be adapted in such a way that it captures the poles at the interval points It is relatively straightforward to try the ansatz (45) h(λ) c d i λ + c d i+ λ +c 3 Notice that, given the coefficients c, c, and c 3, the equation h(λ) can easily be solved by means of the equivalent quadratic equation (46) c (d i+ λ)+c (d i λ)+c 3 (d i λ)(d i+ λ) his equation has two zeros Precisly one of them is inside (d i,d i+ ) he coefficients c, c, and c 3 are computed in the following way Let us assume that we have available an approximation λ j to the zero in (d i,d i+ ) We request that h(λ j ) f(λ j ) and h (λ j ) f (λ j ) he exact procedure is as follows We write i v n k vk (47) f(λ) + + +ψ (λ)+ψ (λ) d k λ d k λ k ki+ }{{}}{{} ψ (λ) ψ (λ)

9 47 A FIRS ALGORIHM 85 ψ (λ) is a sum of positive terms and ψ (λ) is a sum of negative terms Both ψ (λ) and ψ (λ) can be computed accurately, whereas adding them would likely provoke cancellation and loss of relative accuracy We now choose c and ĉ such that (48) h (λ) : ĉ + c d i λ satisfies h (λ j ) ψ (λ j ) and h (λ j ) ψ (λ j ) his means that the graphs of h and of ψ are tangent at λ λ j his is similar to Newton s method However in Newton s method a straight line is fitted to the given function he coefficients in (48) are given by c ψ (λ j )(d i λ j ) >, ĉ ψ (λ j ) ψ (λ j )(d i λ j ) i k v k d k d i (d k λ j ) Similarly, the two constants c and ĉ are determined such that (49) h (λ) : ĉ + with the coefficients Finally, we set c d i+ λ satisfies h (λ j ) ψ (λ j ) and h (λ j ) ψ (λ j ) c ψ (λ j )(d i+ λ j ) >, n ĉ ψ (λ j ) ψ (λ j )(d i+ λ j ) vk d k d i+ (d k λ) ki+ (43) h(λ) +h (λ)+h (λ) (+ĉ +ĉ ) + }{{} d i λ + c d i+ λ c 3 his zerofinder is converging quadratically to the desired zero [7] Usually to 3 steps are sufficient to get the zero to machine precision herefore finding a zero only requires O(n) flops hus, finding all zeros costs O(n ) floating point operations c 47 A first algorithm We are now ready to give the divide and conquer algorithm, see Algorithm 4 All steps except step require O(n ) operations to complete he step costs n 3 flops hus, the full divide and conquer algorithm, requires (43) ( n ) 3 (n) n 3 + (n/) n (n/4) n 3 + n3 ( n ) (n/8) 4 3 n3 his serial complexity of the algorithm very often overestimates the computational costs of the algorithm due to significant deflation that is observed surprisingly often

10 86 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM Algorithm 4 he tridiagonal divide and conquer algorithm : Let C n n be a real symmetric tridiagonal matrix his algorithm computes the spectral decomposition of QΛQ, where the diagonal Λ is the matrix of eigenvalues and Q is orthogonal : if is then 3: return (Λ ;Q ) 4: else [ ] O 5: Partition +ρuu O according to (4) 6: Call this algorithm with as input and Q, Λ as output 7: Call this algorithm with as input and Q, Λ as output 8: Form D +ρvv from Λ,Λ,Q,Q according to (44) (46) 9: Find the eigenvalues [ Λ] and the eigenvectors Q of D +ρvv Q O : Form Q Q which are the eigenvectors of : return (Λ; Q) : if O Q 47 A numerical example Let A be a 4 4 matrix (43) A D +vv β +β 5 + β β [ β β ] In this example (that is similar to one in [8]) we want to point at a problem that the divide and conquer algorithm possesses as it is given in Algorithm 4, namely the loss of orthogonality among eigenvectors Before we do some Matlab tests let us look more closely at D and v in (43) his example becomes difficult to solve if β gets very small In Figures 4 to 45 we see graphs of the function f β (λ) that appears in the secular equation for β, β, and β he critical zeros move towards from both sides he weights v v 3 β are however not so small that they should be deflated he following Matlab code shows the problem We execute the commands for β k for k,,,4,8 v [ beta beta ] ; % rank- modification d [, -beta, +beta, 5] ; % diagonal matrix L eig(diag(d) + v*v ) % eigenvalues of the modified matrix e ones(4,); q (d*e -e*l )\(v*e ); % unnormalized eigenvectors cf (55) Q sqrt(diag(q *q)); q q/(e*q ); % normalized eigenvectors norm(q *q-eye(4)) % check for orthogonality We do not bother how we compute the eigenvalues We simply use Matlab s built-in function eig We get the results of able 4

11 47 A FIRS ALGORIHM Figure 4: Secular equation corresponding to (43) for β Figure 43: Secular equation corresponding to (43) for β We observe loss of orthogonality among the eigenvectors as the eigenvalues get closer and closer his may not be surprising as we compute the eigenvectors by formula (45) x (λi D) v (λi D) v If λ λ and λ λ 3 which are almost equal, λ λ 3 then intuitively one expects almost the same eigenvectors We have in fact Q Q I Orthogonality is lost only with respect to the vectors corresponding to the eigenvalues close to

12 88 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM Figure 44: Secular equation corresponding to (43) for β for λ Figure 45: Secular equation corresponding to (43) for β for 9 λ Already Dongarra and Sorensen [4] analyzed this problem In their formulation they normalize the vector v of D +ρvv to have norm unity, v hey stated Lemma 4 Let ( )[ ] (433) q λ v d λ, v d λ,, v n ρ / d n λ f (λ) hen for any λ,µ {d,,d n } we have (434) q λ q µ f(λ) f(µ) λ µ [f (λ)f (µ)] / Proof Observe that hen the proof is straightforward λ µ (d j λ)(d j µ) d j λ d j µ

13 48 HE ALGORIHM OF GU AND EISENSA 89 β λ λ λ 3 λ 4 Q Q I able 4: Loss of orthogonality among the eigenvectors computed by (45) Formula (434) indicates how problems may arise In exact arithmetic, if λ and µ are eigenvalues then f(λ) f(µ) However, in floating point arithmetic this values may be small but nonzero, eg, O(ε) If λ µ is very small as well then we may have trouble! So, a remedy for the problem was for a long time to compute the eigenvalues in doubled precision, so that f(λ) O(ε ) his would counteract a potential O(ε) of λ µ his solution was quite unsatisfactory because doubled precision is in general very slow since it is implemented in software It took a decade until a proper solution was found 48 he algorithm of Gu and Eisenstat Computing eigenvector according to the formula v λ d (435) x α(λi D) v α v n λ d n, α (λi D) v, is bound to fail if λ is very close to a pole d k and the difference λ d k has an error of size O(ε d k ) instead of only O(ε d k λ ) o resolve this problem Gu and Eisenstat [5] found a trick that is at the same time ingenious and simple hey observed that the v k in (44) are very accurately determined by the data d i and λ i herefore, once the eigenvalues are computed accurately a vector ˆv could be computed such that the λ i are accurate eigenvalues of D+ˆvˆv If ˆv approximates well the original v then the new eigenvectors will be the exact eigenvectors of a slightly modified eigenvalue problem, which is all we can hope for he zeros of the secular equation can be computed accurately by the method presented in section 46 However, a shift of variables is necessary In the interval (d i,d i+ ) the origin of the real axis is moved to d i if λ i is closer to d i than to d i+, ie, if f((d i +d i+ )/) > Otherwise, the origin is shifted to d i+ his shift of the origin avoids the computation of the smallest difference d i λ (or d i+ λ) in (435), thus avoiding cancellation in this most sensitive quantity Equation (46) can be rewritten as (436) (c i+ +c i +c 3 i i+ ) (c }{{} +c +c 3 ( i + i+ )) η + c }{{} 3 η }{{}, b a c where i d i λ j, i+ d i+ λ j, and λ j+ λ j +η is the next approximate zero With equations (48) (43) the coefficients in (436) get (437) a c +c +c 3 ( i + i+ ) (+Ψ +Ψ )( i + i+ ) (Ψ +Ψ ) i i+, b c i+ +c i +c 3 i i+ i i+ (+Ψ +Ψ ), c c 3 +Ψ +Ψ i Ψ i+ Ψ

14 9 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM If we are looking for a zero that is closer to d i than to d i+ then we move the origin to λ j, ie, we have eg i λ j he solution of (436) that lies inside the interval is [7] (438) η a a 4bc c, if a, b a+, a 4bc if a > he following algorithm shows how step 9 of the tridiagonal divide and conquer algorithm 4 must be implemented Algorithm 4 A stable eigensolver for D +vv : hisalgorithmstablycomputesthespectraldecompositionofd+vv QΛQ where D diag(d,d n ), v [v,,v n ] R n, Λ diag(λ,λ n ), and Q [q,,q n ] : d i+ d n + v 3: In each interval (d i,d i+ ) compute the zero λ i of the secular equation f(λ) 4: Use the formula (44) to compute the vector ˆv such that the λ i are the exact eigenvalues of D +ˆvˆv 5: In each interval (d i,d i+ ) compute the eigenvectors of D + ˆvˆv according to (45), 6: return (Λ;Q) q i (λ ii D) ˆv (λ i I D) ˆv 48 A numerical example [continued] We continue the discussion of the example on page 86 where the eigenvalue problem of (439) A D +vv β +β + β ] β [ β β 5 he Matlab code that we showed did not give orthogonal eigenvectors We show in the following script that the formulae (44) really solve the problem dlam zeros(n,); for k:n, [dlam(k), dvec(:,k)] zerodandc(d,v,k); V ones(n,); for k:n, V(k) prod(abs(dvec(k,:)))/prod(d(k) - d(:k-))/prod(d(k+:n) - d(k)); V(k) sqrt(v(k)); Q (dvec)\(v*e ); diagq sqrt(diag(q *Q)); Q Q/(e*diagq );

15 48 HE ALGORIHM OF GU AND EISENSA 9 for k:n, if dlam(k)>, dlam(k) dlam(k) + d(k); else dlam(k) d(k+) + dlam(k); norm(q *Q-eye(n)) norm((diag(d) + v*v )*Q - Q*diag(dlam )) A zero finder returns for each interval the quantity λ i d i and the vector [d λ i,,d n λ i ] to high precision hese vector elements have been computed as (d k d i ) (λ i d i ) he zerofinder of Li [7] has been employed here At the of this section we list the zerofinder written in Matlab that was used here he formulae (437) and (438) have been used to solve the quadratic equation (436) Notice that only one of the while loops is traversed, deping on if the zero is closer to the pole on the left or to the right of the interval he v k of formula (44) are computed next Q contains the eigenvectors β Algorithm Q Q I AQ QΛ I II I II I II I II able 4: Loss of orthogonality among the eigenvectors computed by the straightforward algorithm (I) and the Gu-Eisenstat approach (II) Again we ran the code for β k for k,,,4,8 he numbers in able 4 confirm that the new formulae are much more accurate than the straight forward ones he norms of the errors obtained for the Gu-Eisenstat algorithm always are in the order of machine precision, ie, 6 function [lambda,dl] zerodandc(d,v,i) % ZERODANDC - Computes eigenvalue lambda in the i-th interval % (d(i), d(i+)) with Li s middle way zero finder % dl is the n-vector [d(n) - lambda] n length(d); di d(i); v v^; if i < n, di d(i+); lambda (di + di)/; else di d(n) + norm(v)^; lambda di; eta ; psi sum((v(:i)^)/(d(:i) - lambda));

16 9 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM psi sum((v(i+:n)^)/(d(i+:n) - lambda)); if + psi + psi >, % zero is on the left half of the interval d d - di; lambda lambda - di; di di - di; di ; while abs(eta) > *eps psi sum(v(:i)/(d(:i) - lambda)); psis sum(v(:i)/((d(:i) - lambda))^); psi sum((v(i+:n))/(d(i+:n) - lambda)); psis sum(v(i+:n)/((d(i+:n) - lambda))^); % Solve for zero Di -lambda; Di di - lambda; a (Di + Di)*( + psi + psi) - Di*Di*(psis + psis); b Di*Di*( + psi + psi); c ( + psi + psi) - Di*psis - Di*psis; if a >, eta (*b)/(a + sqrt(a^ - 4*b*c)); else eta (a - sqrt(a^ - 4*b*c))/(*c); lambda lambda + eta; else % zero is on the right half of the interval d d - di; lambda lambda - di; di di - di; di ; while abs(eta) > *eps psi sum(v(:i)/(d(:i) - lambda)); psis sum(v(:i)/((d(:i) - lambda))^); psi sum((v(i+:n))/(d(i+:n) - lambda)); psis sum(v(i+:n)/((d(i+:n) - lambda))^); % Solve for zero Di di - lambda; Di - lambda; a (Di + Di)*( + psi + psi) - Di*Di*(psis + psis); b Di*Di*( + psi + psi); c ( + psi + psi) - Di*psis - Di*psis; if a >, eta (*b)/(a + sqrt(a^ - 4*b*c)); else eta (a - sqrt(a^ - 4*b*c))/(*c); lambda lambda + eta; dl d - lambda; return

17 BIBLIOGRAPHY 93 Bibliography [] E Anderson, Z Bai, C Bischof, J Demmel, J Dongarra, J D Croz, A Greenbaum, S Hammarling, A McKenney, S Ostrouchov, and D Sorensen, LAPACK Users Guide - Release, SIAM, Philadelphia, PA, 994 (Software and guide are available from Netlib at URL [] J J M Cuppen, A divide and conquer method for the symmetric tridiagonal eigenproblem, Numer Math, 36 (98), pp [3] J W Demmel, Applied Numerical Linear Algebra, SIAM, Philadelphia, PA, 997 [4] J J Dongarra and D C Sorensen, A fully parallel algorithm for the symmetric eigenvalue problem, SIAM J Sci Stat Comput, 8 (987), pp s39 s54 [5] M Gu and S C Eisenstat, A stable and efficient algorithm for the rank-one modification of the symmetric eigenproblem, SIAM J Matrix Anal Appl, 5 (994), pp [6], A divide-and-conquer algorithm for the symmetric tridiagonal eigenproblem, SIAM J Matrix Anal Appl, 6 (995), pp 7 9 [7] R-C Li, Solving secular equations stably and efficiently, echnical Report U-CS- 94-6, University of ennessee, Knoxville, N, Nov 994 LAPACK Working Note No 89 [8] D C Sorensen and P P ang, On the orthogonality of eigenvectors computed by divide-and-conquer techniques, SIAM J Numer Anal, 8 (99), pp

18 94 CHAPER 4 CUPPEN S DIVIDE AND CONQUER ALGORIHM