A Model-Trust-Region Framework for Symmetric Generalized Eigenvalue Problems

Transcription

1 A Model-Trust-Region Framework for Symmetric Generalized Eigenvalue Problems C. G. Baker P.-A. Absil K. A. Gallivan Technical Report FSU-SCS Submitted June 7, 2005 Abstract A general inner-outer iteration for computing extreme eigenpairs of symmetric/positive-definite matrix pencils is proposed. The principle of the method is to produce a sequence of p-dimensional bases {X k } that converge to a minimizer of a generalized Rayleigh quotient. The role of the inner iteration is to produce an update vector by (approximately) minimizing a quadratic model of the Rayleigh quotient within a neighbourhood of X k where the model is trusted. The role of the outer iteration is to make the best out of the proposed update vector combined with previously-obtained information; it consists of a Rayleigh-Ritz process that minimizes the exact Rayleigh quotient in a low-dimensional subspace. This general scheme leaves a lot of leeway for choosing the algorithmic details of the inner and outer iterations. The global and local convergence of the scheme are analytically studied under weak assumptions on these algorithmic choices. Moreover, numerous experiments are carried out to explore the influence of the algorithmic choices on the performance of the scheme. In particular, the question of balancing the computational effort between the inner and outer iterations is investigated. School of Computational Science, Florida State University, Tallahassee, FL , USA ( cbaker, absil, gallivan}). This work was supported by the USA National Science Foundation under Grants ACI and CCR , and by the School of Computational Science of Florida State University through a postdoctoral fellowship. 1

2 1 Introduction The generalized symmetric eigenvalue problem occurs in many applications in the engineering and scientific disciplines. Given a symmetric matrix A and a symmetric positive definite matrix B, the generalized symmetric eigenvalue problem seeks to find nonzero vectors x and scalars λ satisfying Ax = λbx. Here λ is an eigenvalue and x is an eigenvector associated with λ. Let λ 1 λ 2... λ n be the eigenvalues of (A,B), where n is the order of the two matrices. It is known that the eigenvalues and their associated eigenvectors satisfy optimality conditions with respect to the generalized Rayleigh quotient. That is, the function f : R n p matrices, R, where R n p is the set of all rank p f(x) = trace ( (X T BX) 1 (X T AX) ) (1) is minimized by any matrix whose column space corresponds to the eigenspace associated with the leftmost eigenvalues of (A, B). The minimum at this point is the sum of the p leftmost eigenvalues. It is easily verified that the function varies only with the column space of X. It follows that f can be restricted to the set of all p-dimensional subspaces of R n. This set is termed the Grassmann manifold. Absil et al.[abg04c, ABG05] recently described the Riemannian Trust- Region (RTR) algorithm. The RTR algorithm is a method for finding the extreme points of a function defined on a Riemannian manifold. Under mild regularity conditions, the method is proven to always converge to critical points. Furthermore, saddle points are unstable, so that the algorithm converges to a local minimum in practice. If the local minimum is nondegenerate, then the convergence is superlinear. Assume that λ p < λ p+1, so that the leftmost eigenspace is well-defined. Then there is a unique local minimizer of the generalized Rayleigh quotient (1) defined on the Grassmann manifold. Furthermore, the convergence conditions for the RTR algorithm are satisfied by the generalized Rayleigh quotient. Therefore, applying the RTR algorithm yields a superlinear method which converges to the leftmost eigenspace for virtually all initial conditions. The authors show [ABG04a] that the RTR algorithm, directly applied to this problem, is competitive with many of the specialized methods that have been developed for extreme eigenspace computation. 2

3 The method has been shown [ABG04a] to be related to existing eigenvalue solvers, especially the Trace Minimization [SW82, ST00] and Jacobi- Davidson methods [SV96]. As such, techniques used by these methods, as well as other methods (such as [Kny01]), can be used to augment RTR. One technique is the idea of subspace acceleration, where a subspace is constructed and the next iterate is chosen using a Ritz-Rayleigh procedure. We two approaches for constructing these subspaces that come from Jacobi-Davidson and. Another approach is that of adaptive methods. One method that will be explored in this paper is a Tracemin/RTR hybrid, which initially uses Tracemin before switching to RTR. Another method combines a like subspace acceleration with RTR-like update vectors. 2 The general algorithm Here points on the Grassmann manifold M are represented by orthonormal bases, so that X T BX = I is assumed throughout. The tangent space to M at colsp(x) is represented by T X := {η R n p : X T Bη = 0}, i.e., the set of R n p matrices whose columns are B-orthogonal to the column space of X. Let ˆf denote the cost function (1) restricted to T X : ˆf X (η) = f(x + η) ( ((X = trace + η) T B(X + η) ) 1 ( (X + η) T A(X + η) )) (2) The standard RTR [ABG04a] attempts to minimize at each iterate X a quadratic model m X of ˆf X, m X (η) = ˆf(X) ( + trace η T grad ˆf(X) ) trace ( η T H X [η] ), η T X (3) within the current trust-region radius (trace(η T η) ). In (3), the firstorder term is determined by grad ˆf(X) = 2P BX,BX AX. The operator H X is some approximation to the Hessian of ˆf at X, which need be available only as an operator on the tangent bundle. Possible choices for the model Hessian H X are discussed in Section 5. The performance of the quadratic model determines whether the trustregion radius is enlarged or reduced. When not performing subspace acceleration, the model performance determines whether the proposed iterate is accepted or not. The performance of the quadratic model is measured 3

4 according to the following ratio: ρ(η) = f(x) f(x + η) m(0) m(η) A general framework follows. We use gsb to denote a Gram-Schmidt orthonormalization using the B inner product. Algorithm 1 (Subspace Accelerated RTR) Data: Symmetric matrices A and B, B positive definite Parameters: maximum rank of acceleration subspace m, initial trust-region radius 0, model performance mark ρ Input: initial iterate X 0, X T 0 BX 0 = I p. Output: sequence of iterates {X k }. for k = 0,1,2,... Part I: Model-based Minimization Compute η k to approximately minimize m Xk within the trust-region {η : trace(η T η) k }, using tcg (Section 4) Adjust trust-region radius: if ρ(η k ) > 3/4 and η k = k+1 = 2 k else if ρ(η k ) < 1/4 k+1 = k /4 else k+1 = k end Part II: Generate next iterate if performing subspace acceleration Update the acceleration subspaces using η k (Section 3) Generate X k+1 using a Ritz-Rayleigh procedure else if ρ(η k ) > ρ X k+1 = gsb(x k + η k ) else X k+1 = X k end end Check gradf(x k+1 ) end for. 4

5 The issue remains of the method for solving the model trust-region minimization at each iteration, as well as the technique used to update the acceleration subspace. We will discuss the latter in Section 3 and the former in Section 4. 3 Subspace Acceleration Strategies In this section, we propose practical techniques for carrying out the subspace acceleration procedure involved in Part II of Algorithm 1. We will discuss two methods from the literature for constructing the acceleration subspace, one from Jacobi-Davidson and another inspired by and the sequential subspace method of Hager[Hag01]. The first subspace acceleration strategy comes directly from Jacobi- Davidson. It is referred to here as Expanding Subspaces (ES), and can be described as follows. Take a basis U for a subspace U, which we desire to update using some η. Compute the new subspace U + = colsp([u,η]). If the subspace has already reached the maximum allowable size (m), then reset it to U + = colsp([x,η]), where X is the current iterate. The second subspace acceleration strategy is a simple extension of the technique used in. Knyazev [Kny01] proposes a non-linear conjugate gradient approach which takes advantage of the nature of the generalized Rayleigh quotient to perform exact searches using a Ritz-Rayleigh procedure. At each step, the next iterate X k+1 is chosen using a Ritz-Rayleigh procedure with respect to the subspace given by colsp([x k,h k,p k ]), where H k is the preconditioned residual (gradient) and P k is an analogue of the CG search direction. The components of the primitive Ritz vectors define the coefficients that combine X k, H k and P k to form X k+1. The new preconditioned residual H k+1 is determined by X k+1, and the P k+1 is computed via a CG-like recurrence. This method is stated explicitly in Algorithm 2. Algorithm 2 () Data: Symmetric matrices A and B, B positive definite, preconditioner T Input: initial iterate X 0, X T 0 BX 0 = I p. Output: sequence of iterates {X k }. Initialize: P 0 = 0 for k = 0,1,2,... Build search space for j = 1,...,p Compute Rayleigh quotients µ j = (x T j Ax j)/(x T j Bx j), where x j = X k e j 5

6 Compute residuals r j = Ax j µ j Bx j end for Check r j for convergence Apply preconditioner: H k = [h 1,...,h p ] = [Tr 1,...,Tr p ] Set V k = [X k,h k,p k ] Ritz-Rayleigh procedure Form  = V T k AV k and ˆB = V T k BV k Compute p smallest eigenpairs (u k, ˆλ k ) of (Â, ˆB) X k+1 = V k U k, where U k = [u 1,...,u p ] P k+1 = [H k,p k ] U k (p + 1 : 3 p,:) end for. Note that although the first Ritz-Rayleigh process is carried out with p 0 = 0 on a two-dimensional subspace, subsequent steps occur on threedimensional subspaces. Therefore, there is no restarting like in the Expanding Subspaces method. To our knowledge, this technique has only been applied using three blocks of vectors: X k, H k, and P k. We propose a simple extension that makes use of previous search directions as well, so that the Ritz-Rayleigh process operates on a larger, m-dimensional subspace (m = bp, were b is the number of blocks). Without discussing here the nature of the vectors H k (see Section 4), we describe one step of a subspace acceleration algorithm, which we refer to as Sequential Subspaces (SS). Algorithm 3 (Sequential Subspace step) Data: Symmetric matrices A and B, B positive definite Input: Acceleration subspace basis V k = [X k,h k,p k b+3,...,p k ] of dimension m = b p. Output: iterate X k+1, X T k+1 BX k+1 = I p, and new search direction P k+1 Ritz-Rayleigh procedure Form  = V T k AV k and ˆB = V T k BV k Compute p smallest eigenpairs (u k, ˆλ k ) of (Â, ˆB) Compute next iterate X k+1 = V k U k, where U k = [u 1,...,u p ] Compute next search direction 6

7 P k+1 = [H k,p k b+3,...,p k ] U k (p + 1 : b p,:) 4 Model Minimization Strategies In classical RTR, the model minimization can be solved using a Steihaug- Toint truncated CG (tcg) [ABG04c]. This version of CG checks the curvature of the model (i.e., the definiteness of the Hessian) for each search direction. Upon encountering a direction of negative curvature, it is followed to the edge of the trust region. This is one of the major differences between RTR and JD-like methods, as this curvature mechanism is explicitly attempting to minimize the quadratic model, instead of simply searching for its critical point (which is not a minimizer when the model is not positive definite.) The second truncation mechanism in tcg is the trust region constraint. The underlying idea is that the quadratic model (3) is an accurate approximation of the cost function (2) only in the vicinity of η = 0; far away from the origin, the model becomes irrelevant. Because tcg builds a solution iteratively from a sequence of search directions, it is simple to test the length of the intermediate solutions and to note when (and along which direction) the trust region is breached. On this occasion, the solution is only allowed to move (in the current direction) to the edge of the trust region. Finally, there is the matter of a stopping criterion. Because tcg uses conjugate search directions, the number of iterations should be limited to the dimension of the search space. This search space is the space orthogonal to the current iterate, with dimension d = p(n p). In order to avoid a waste of computational effort without losing superlinear convergence, the following residual-based stopping criterion is used: r i r 0 min(κ,( r 0 /σ) θ ), (4) where r i is the residual of the model at inner step i, and κ and θ are terms targeting the rate of convergence (see [ABG04b]). This condition varies from the condition in [ABG04b] with the introduction of σ. For problems whose scaling is such that the method converges before the norm of the gradient becomes smaller than 1, then κ < 1 < r 0 θ, so that superlinear convergence is never attempted. The σ term seeks to adjust the scaling, with the suggestion that σ is chosen as σ = gradf(x 0 ). A p = 1, unpreconditioned version of truncated CG algorithm follows. We use, to denote the standard inner product. 7

8 Algorithm 4 (Truncated CG) Set η 0 = 0, r 0 = P,Bx k Bx k Ax k, δ 0 = r 0 ; for j = 0, 1, 2,... until a stopping criterion is satisfied Check curvature of current search direction if δ j, H xk [δ j ] 0 Compute τ such that η = η j + τδ j satisfies η = ; return η; Set α j = r j,r j / δ j, H xk [δ j ] ; Generate next inner iterate Set η j+1 = η j + α j δ j ; Check trust-region if η j+1 Compute τ 0 such that η = η j + τδ j satisfies η = ; return η; Use CG recurrences to update residual and search direction Set r j+1 = r j + α j H xk [δ j ]; Set β j+1 = r j+1,r j+1 / r j,r j ; Set δ j+1 = r j+1 + β j+1 δ j ; end for. Another parametrization regards the stopping criteria. A limit on the number of iterations may be imposed ( returns the preconditioned residual, implicitly allowing one/zero inner iteration, depending on how iterations are number). The residual-based stopping criterion from classical RTR may be used (as described above), or the residual-based stopping criterion suggested by Notay (equations (27) and (28) from [Not02]) for use in JDCG can be employed. The question becomes, what is the trade-off between the amount of work on the model minimization and the performance of the algorithm. 5 Choice of the model Having described the technique used to minimize the model, the remaining issue is the choice of the model. We have chosen to approximate the Rayleigh quotient via a quadratic function, agreeing through the first order. The issue becomes, what are the benefits of different choices for the model Hessian? As shown in [ABGS05], different choices for the quadratic model may be 8

9 appropriate, depending on the circumstances. The two choices discussed here are an inexact Hessian from the Trace Minimization method [SW82, ST00] and the exact Hessian from the classical RTR described in the original presentation [ABG04b]. The Tracemin Hessian is H X [η] = 2(P BX,BX AP BX,BX )η, (5) where P BX,BX = I BX(X T BBX) 1 X T B is a projector onto the tangent plane at X. The Hessian from the classical RTR is the exact Hessian of the function: H X [η] = Hess X [η] = 2P BX,BX (Aη Bη(X T AX)) (6) Using the exact Hessian yields a quadratic model that closely represents the Rayleigh quotient within a neighborhood of the current iterate. This is what gives the RTR superlinear convergence near a solution, as the superlinear term in the residual-based stopping criterion becomes the significant stopping criterion near a solution. In fact, without using a good approximation of the Hessian, it is difficult to achieve fast convergence. However, far from the solution, the quality of the second order terms become less significant, so that an approximate Hessian may be used, such as the Hessian from Tracemin. Furthermore, this Hessian brings with it specific advantages. In choosing the Tracemin Hessian as in (5), it is not necessary to check for directions of negative curvature because the Hessian is (in exact arithmetic) positive definite. Also, any reduction in the quadratic model is known to produce a reduction in the generalized Rayleigh quotient, so that we may ignore the trust-region constraint [ABGS05]. The former requires that the matrix A is positive definite, always assumed when discussing the Tracemin method. These ideas are discussed in more detail in [ABGS05]. 6 Numerical Experiments The choice of a subspace acceleration strategy, along with the options for performing the model minimization, create from SA-RTR a family of algorithms for finding the leftmost eigenvectors and eigenvalues of a symmetric matrix pencil. Some of these family members are tested here. Tests are run on the following systems: A dense matrix A, of order n = 100, and B = I, constructed so that the eigenvalues are linearly spaced, with no gap between the targeted 9

10 and untargeted parts of the spectrum. The spectrum of this matrix is shown in Figure 1. A dense matrix A, of order n = 100, and B = I, constructed so that the eigenvalues are linearly spaced, except for a significant gap between the the targeted and untargeted parts of the spectrum. The spectrum of this matrix is shown in Figure 1. A matrix A representing a Laplacian operator, generated via the MAT- LAB command A = delsq(numgrid( N,30)), of order n = 784. This test is run with no preconditioner, using an incomplete Cholesky generated via cholinc(a, 0 ), and using an exact Cholesky generated via chol(a). All three choices of A are positive definite, so that the eigenvalues of (A,B) are positive as well (this is necessary for a discussion of Trace Minimization). 6.1 Subspace size The first set (Figure 2) of experiments illustrates the benefit of adding more search directions to the method. Here, RTR is run with SS subspace acceleration, with subspace rank 4p, where the model minimization performs 0 steps (returns the preconditioned gradient). This is denoted SS- RTR(4p,0). Here, we compute only a single eigenpair, so p = 1. In this case, adding a single vector to the acceleration subspace basis yields a benefit. Note that the number of vectors in the acceleration subspaces affects storage cost, as well as the cost of the Ritz-Rayleigh process (which solves a dense eigenvalue problem.) However, it does not affect the number of multiplications against A. In the second set (Figure 3), we again compare against SA- RTR(4p,0), this time on the gap-controlled matrices. Note that for the first test, the matrix with a gap, the performance difference between the two algorithms is negligible. For the matrix with no gap, however, the extra search direction yields a noticeable effect. 6.2 Number of inner iterations Figure 4 shows vs. SS-RTR(3p), where the model minimization was allowed either 0, 2 or 4 steps to produce a tangent vector. Note that for the first case (0 steps), SS-RTR(3p,0) performs identically to. 10

11 120 Spectrum for NoGap test, p= Spectrum for Gap test, p= Figure 1: Spectrum of controlled gap problems, without and with gap, respectively. 11

12 Laplacian, n=784, no precon, p=1 Laplacian, n=784, IC(0), p=1 SS RTR(4p,0) SS RTR(4p,0) Laplacian, n=784, EC, p=1 SS RTR(4p,0) Figure 2: SA-RTR(4p,0) vs., for the Laplacian problem, with no preconditioner, incomplete Cholesky, and exact Cholesky, respectively. 12

13 Dense,n=100,gap after 1 SS RTR(4p,0) Dense,n=100,no gap SS RTR(4p,0) Figure 3: SA-RTR(4p,0) vs., for the dense matrices, with a gap and without a gap in the eigenvalues, respectively. 13

14 As more steps are allowed, the performance of SS-RTR overtakes that of. However, this trend does not hold for the problem where a factorization of A was available for use in the model minimization preconditioner. The set in Figure 5 shows the effect of more iterations in the model minimization on the gap-chosen matrices. Similar to Figure 3, the parameters of the algorithm are noticeable only for the matrix with no gap (the harder problem). 6.3 Effect of subspace acceleration In Figure 6, we compare the classical RTR with no subspace acceleration against RTR with 3-dimensional subspace acceleration via Expanding Subspaces (ES) and Sequential Subspaces (SS). The data used is the Laplacian problem, with three different preconditioning methods. Note that in all three case, the subspace accelerated methods outperform the un-accelerated method. Note also that for such a low-dimensional acceleration subspace, there is no significant performance difference between the two acceleration schemes. Figure 7 shows the performance of this set of methods applied to the dense, gap-controlled problems. Similar to the Laplacian problem, there is not a large performance difference between classical RTR and the subspace accelerated RTR methods. 6.4 Residual-based stopping criteria In Figure 8, we compare SS-RTR with two different stopping criteria: the residual-norm-based method proposed in the classical RTR against the residualnorm-based method proposed by Notay in [Not02]. is included as a reference. Again, the test data used here is the Laplacian, with three different preconditioning schemes. Note that in these experiments, the RTR stopping criterion performs at least as well as the stopping criterion suggested by Notay, performing significantly better for the exact Cholesky preconditioned test. Figure 9 tests these methods on the gap-controlled problems. Again, the stopping criterion proposed by Notay serves the method well for the more difficult problem, where there is only a small gap between eigenvalues. However, for the easier problem, where a large gap separates the two parts of the spectrum, the Notay stopping criterion spends too much time on inner iterations. 14

15 Laplacian, n=784, no precon, p=1 SS RTR(3p,0) SS RTR(3p,2) SS RTR(3p,4) Laplacian, n=784, IC(0), p=1 SS RTR(3p,0) SS RTR(3p,2) SS RTR(3p,4) Laplacian, n=784, EC, p=1 SS RTR(3p,0) SS RTR(3p,2) SS RTR(3p,4) Figure 4: SA-RTR(3p) vs., for the Laplacian problem, with varying number of inner steps (0,2,4) allowed for the model minimization, for no preconditioner, incomplete Cholesky, and exact Cholesky, respectively. 15

16 Dense,n=100,gap after 1 SS RTR(3p,0) SS RTR(3p,2) SS RTR(3p,4) Dense,n=100,no gap SS RTR(3p,0) SS RTR(3p,2) SS RTR(3p,4) Figure 5: SA-RTR(3p) vs., for the dense problems, with varying number of inner steps (0,2,4) allowed for the model minimization, with a gap and without a gap, respectively. 16

17 Laplacian, n=784, no precon, p=1 Laplacian, n=784, IC(0), p=1 RTR(nosa) ES RTR(3p,RTR stop) SS RTR(3p,RTR stop) RTR(nosa) ES RTR(3p,RTR stop) SS RTR(3p,RTR stop) Laplacian, n=784, EC, p=1 RTR(nosa) ES RTR(3p,RTR stop) SS RTR(3p,RTR stop) Figure 6: RTR(nosa) vs. ES-RTR(3p) vs. SS-RTR(3p), for the Laplacian problem, for no preconditioner, incomplete Cholesky, and exact Cholesky, respectively. 17

18 Dense,n=100,gap after 1 RTR(nosa) ES RTR(3p,RTR stop) SS RTR(3p,RTR stop) Dense,n=100,no gap RTR(nosa) ES RTR(3p,RTR stop) SS RTR(3p,RTR stop) Figure 7: RTR(nosa) vs. ES-RTR(3p) vs. SS-RTR(3p), for dense problems, with a gap and without a gap, respectively. 18

19 Laplacian, n=784, no precon, p=1 Laplacian, n=784, IC(0), p=1 SS RTR(3p,RTR stop) SS RTR(3p,Notay stop) SS RTR(3p,RTR stop) SS RTR(3p,Notay stop) Laplacian, n=784, EC, p=1 SS RTR(3p,RTR stop) SS RTR(3p,Notay stop) Figure 8: SS-RTR(3p,RTR-stop) vs. SS-RTR(3p,Notay-stop) vs., for the Laplacian problem, for no preconditioner, incomplete Cholesky, and exact Cholesky, respectively. 19

20 Dense,n=100,gap after 1 SS RTR(3p,RTR stop) SS RTR(3p,Notay stop) Dense,n=100,no gap SS RTR(3p,RTR stop) SS RTR(3p,Notay stop) Figure 9: SS-RTR(3p,RTR-stop) vs. SS-RTR(3p,Notay-stop) vs., for dense problems, with a gap and without a gap, respectively. 20

21 6.5 Quality of model Hessian We examine the performance of RTR against that of Tracemin, under different preconditioner scenarios. Also demonstrated here is the adaptive model method, a RTR-Tracemin hybrid (called the Adaptive Trust-Region method) which begins with Tracemin and switches to RTR [ABGS05]. Figure 10 tests these on the Laplacian matrix, targeting the leftmost p = 4 eigenvalues, with different preconditioners: no preconditioning, preconditioning using an incomplete Cholesky of A, and preconditioning using an exact Cholesky of A. Figure 11 tests these methods on the gap-controlled matrices, using either no preconditioning or preconditioning using an exact Cholesky factorization of A. 21

22 Laplacian, n=784, no precon, p=4 Laplacian, n=784, IC(0), p=4 RTR TM ATR RTR TM ATR Laplacian, n=784, EC, p=4 RTR TM ATR Figure 10: Classical RTR vs. Tracemin vs. AR, for the Laplacian problem, p = 4, for no preconditioner, incomplete Cholesky, and exact Cholesky, respectively. 22

23 10 4 Dense,n=100,gap after 5,p= Dense,n=100,gap after 5,p=5,EC RTR TM ATR RTR TM ATR Dense,n=100,no gap,p=5 RTR TM ATR 10 4 Dense,n=100,no gap,p=5,ec RTR TM ATR Figure 11: Classical RTR vs. Tracemin vs. AR, for dense problems, with a gap (first row) and without a gap (second row), without a preconditioner (first column) and with a preconditioner from an exact factorization of A (second column). 23

24 References [ABG04a] P.-A. Absil, C. G. Baker, and K. A. Gallivan, A truncated-cg style method for symmetric generalized eigenvalue problems, submitted, [ABG04b], Trust-region methods on Riemannian manifolds, Tech. Report FSU-CSIT-04-13, School of Computational Science, Florida State University, July [ABG04c] [ABG05], Trust-region methods on Riemannian manifolds with applications in numerical linear algebra, Proceedings of the 16th International Symposium on Mathematical Theory of Networks and Systems (MTNS2004), Leuven, Belgium, 5 9 July 2004, 2004., Trust-region methods on Riemannian manifolds, submitted, March [ABGS05] P.-A. Absil, C. G. Baker, K. A. Gallivan, and A. Sameh, Adaptive model trust region methods for generalized eigenvalue problems, Lecture Notes in Compuer Science, vol. 3514, pp , Springer-Verlag, [Hag01] W. W. Hager, Minimizing a quadratic over a sphere, SIAM J. Optim. 12 (2001), no. 1, (electronic). [Kny01] [Not02] [ST00] [SV96] A. V. Knyazev, Toward the optimal preconditioned eigensolver: locally optimal block preconditioned conjugate gradient method, SIAM J. Sci. Comput. 23 (2001), no. 2, Y. Notay, Combination of Jacobi-Davidson and conjugate gradients for the partial symmetric eigenproblem, Numer. Linear Algebra Appl. 9 (2002), no. 1, A. Sameh and Z. Tong, The trace minimization method for the symmetric generalized eigenvalue problem, J. Comput. Appl. Math. 123 (2000), G. L. G. Sleijpen and H. A. Van der Vorst, A Jacobi-Davidson iteration method for linear eigenvalue problems, SIAM J. Matrix Anal. Appl. 17 (1996), no. 2,

25 [SW82] A. H. Sameh and J. A. Wisniewski, A trace minimization algorithm for the generalized eigenvalue problem, SIAM J. Numer. Anal. 19 (1982), no. 6,