How Rutishauser may have found the qd and LR algorithms, t

Transcription

1 How Rutishauser may have found the qd and LR algorithms, the fore-runners of QR Martin H Gutnecht and Beresford N Parlett Seminar for Applied Mathematics ETH Zurich Department of Mathematics University of California Bereley 3rd Biennial Conference on Numerical Analysis, U Strathclyde, Glasgow, Scotland 3 6 June 9 Martin H Gutnecht and Beresford N Parlett

2 Rutishauser s qd algorithm: early papers Rutishauser (954a, ZAMP): Der Quotienten Differenzen Algorithmus Rutishauser (954b, ZAMP): Anwendungen des Quotienten Differenzen Algorithmus Rutishauser (954c, ArchMath): Ein infinitesimales Analogon zum Quotienten Differenzen Algorithmus Rutishauser (955a, ZAMP): Bestimmung der Eigenwerte und Eigenvetoren einer Matrix mit Hilfe des Quotienten Differenzen Algorithmus Rutihauser (957a, Mitt IAM, ETH): Der Quotienten Differenzen Algorithmus Henrici (958, NBS boo): The Quotient-Difference Algorithm Martin H Gutnecht and Beresford N Parlett

3 Eduard Stiefel s suggestion ( 953) Stiefel s suggestion to Rutishauser: Given A, x, y, use the Schwarz constants (= moments = Marov parameters) to find the eigenvalues of A s : y T A x ( =,,,) () Daniel Bernoulli (73), J König (884): s ν+ s ν λ as ν if λ > λ λ Note: It turned out that for the other eigenvalues, Stiefel s proposal was a bad idea, since the dependence of the EVals from the moments is highly ill-conditioned (Gautschi (968)) Martin H Gutnecht and Beresford N Parlett

4 Moments and their generating function Given: N N matrix A and x, y R N, let f(z) : y,(zi A) x = y, z (I z A) x () f is a rational function of type (N, N), so f( ) = The poles of f are eigenvalues of A f can be expanded into a power series in z : where f(z) = = s z + = s z + s z + s + (3) z3 s = y T A x So, f is the generating function of the moments Martin H Gutnecht and Beresford N Parlett

5 Alternative formulations of the problem Clearly, there are several equivalent problems: Find eigenvalues of A Find poles of generating (rational) function f Find zeros of the denominator polynomial of f (Bernoulli) In theory, the problem had been solved before by Hadamard (89) (his PhD thesis!), de Montessus de Ballore (9/95), Aiten (96/93) But none of them had an efficient algorithm Rutishauser cites Hadamard and Aiten, but never de Montessus de Ballore, who proved the convergence of Padé approximants with fixed denominator degree Martin H Gutnecht and Beresford N Parlett

6 Hadamard s theorem (89) Given the power series of f in z of (3), let H (ν) :, and define the Hanel determinants s ν s ν+ s ν+ H (ν) = s ν+ s ν+ s ν+ s ν+ s ν+ s ν+ ( =,,; ν =,, ) THEOREM [Hadamard (89)] If λ + < Λ < λ, then, as ν, ( ) H (ν) Λ ν ] = const (λ λ ) [ ν + O λ For a simpler proof see Henrici (958) or Henrici (974) Martin H Gutnecht and Beresford N Parlett

7 Hadamard s theorem (89) (cont d) COROLLARY If f has N simple poles, then H (ν) ( =,,N) for large enough ν, and H (ν) N+ = ( ν) If λ > λ + then H (ν+) H (ν) λ λ λ as ν (4) 3 If λ > λ > λ + then q (ν) : H (ν+) H (ν) H (ν) H (ν+) λ as ν (5) Martin H Gutnecht and Beresford N Parlett

8 Aiten s scheme (93) (cont d) Computing, for fixed ν, the Hanel determinants H (ν),, H (ν) N (if nonzero) requires the LU decomposition of the matrix H (ν) N Aiten (96, 93) used what is now called Jacobi identity ( theorem of compound determinants ) ( H (ν) ) (ν ) = H H (ν+) + H (ν ) + H (ν+) (6) It had also been nown to Hadamard, but Aiten used it to build up from the left or from the top the table H () H () H () H () H () H () 3 H (3) H () H () 3 H () 4 Martin H Gutnecht and Beresford N Parlett

9 Rutishauser s qd algorithm (QD-Algorithmus) Rutishauser (954a) new Aiten s wor and refers to (4), H (ν+) H (ν) λ λ λ as ν as the ey to computing non-dominant poles But instead of computing the H (ν) table, he headed directly for recurrences for q (ν) : H (ν+) H (ν) H (ν) H (ν+) and : H (ν) + H (ν) H (ν+) H (ν+) (7) In Rutishauser (954a) he derives the formulas needed for q (ν), and then states recursions for general Martin H Gutnecht and Beresford N Parlett

10 Rutishauser s qd algorithm (cont d) qd table (QD Schema): q () q () q () q (3) q (4) e () e () e () e (3) e (4) + q () q () q () q (3) + e () e () e () e (3) e () N e () N e () N q () N q () N e () q () = q () e () q () + e () = e () + q () Martin H Gutnecht and Beresford N Parlett

11 Rutishauser s qd algorithm (cont d) Rhombus rules (called so by Stiefel, 955) of qd algorithm: For building up the table columnwise from left to right: := e (ν+) + q (ν+) q (ν) q (ν) + := q (ν+) e (ν+) ( =,,) (8) For building up the table row-wise, from top to bottom: q (ν+) := q (ν) + e (ν+) e (ν+) := q (ν+) q (ν) + ( =,,) (9) Recursions (9) are the basis of the progressive qd algorithm (the relevant version) Martin H Gutnecht and Beresford N Parlett

12 Rutishauser s qd algorithm (cont d) In Rutishauser (954a) the correctness of the rhombus rules follows later from the connections to continued fractions (probably Stiefel s argument) Originally, Rutishauser derived them probably from Hadamard s Jacobi identity ( H (ν) ) (ν ) = H H (ν+) + H (ν ) + H (ν+) Henrici (958), who was in contact with Rutishauser, pointed out that one rhombus rules (+) can be derived by combining two applications of this formula, the other ( ) just by using the definitions (9) of q (ν) and The details have been wored out in Parlett (996), a TR entitled What Hadamard missed Martin H Gutnecht and Beresford N Parlett

13 From power series to a continued fractions By a standard operation the given power series (3) in z of f, f(z) = = s z + = s z + s z + s z 3 + can be turned into a continued fraction (which typically converges in a much larger region) We may also write f(z) = s z + s z + + s ν z ν + f ν(z) z ν () and expand the remainder f ν (z) of the power series into a continued fractions In each case, two different types of continued fractions can be used So we get two whole series of continued fractions It turns out that their coefficients are related by the rhombus rules Martin H Gutnecht and Beresford N Parlett

14 Continued fractions: J fractions and S fractions f ν (z) : = s ν+ = zν z+ ( ν f(z) = s z + can be expanded both into a Jacobi fraction or J fraction f ν (z) = s ν z q (ν) q(ν) z q (ν) and into a formal Stieltjes fraction or S fraction ) () q(ν) z q (ν) 3 () f ν (z) = s ν z q(ν) e(ν) z q(ν) e(ν) z (3) The J fraction is the so-called even part of the S fraction obtained by merging two successive terms into one Martin H Gutnecht and Beresford N Parlett

15 Continued fractions: J fractions and S fractions The odd part of the S fraction is another formal J fraction, obtained by merging the two differently chosen successive terms into one, f ν(z) = sν z + q (ν) z q (ν) q(ν) z q (ν) By comparing this J fraction with the one for q(ν) 3 z q (ν) 3 3 (4) f ν+ (z) = zf ν (z) s ν, (5) one recovers Rutishauser s rhombus rules of the qd algorithm This is the nicest derivation of the qd algorithm, but not the original one Rutishauser (954a) indicates that it may have been suggested to him by Stiefel Martin H Gutnecht and Beresford N Parlett

16 Continued fractions, Padé approximations, FOPs The partial sums = convergents = approximants of the continued fractions are confluent rational interpolants of f They are Padé approximants (at ) associated with the moments s +ν ( =,,; ν fixed) of the function f ν (z) defined by f(z) = s z + s z + + s ν z ν + f ν(z) z ν For fixed ν, the denominators of the convergents (= Padé approximants) are (formal) orthogonal polynomials p (ν) (z) They can be arranged in a table that he called p table (Analogous to the Padé table) Martin H Gutnecht and Beresford N Parlett

17 Associated polynomials and their p table p table (P Schema): p () p () p () p (3) p (4) p () p () p () p (3) p () In the last column, p () N q () z p() p () p () p () 3 p () 3 p () 3 p () N p () N (z) := z p() (z) q() p() (z) = p() N p () N p () N = is the minimal polynomial Martin H Gutnecht and Beresford N Parlett

18 Continued fractions, Padé approximations, FOPs (cont d) Rutishauser realized that they are also the Lanczos polynomials of the (nonsymmetric) Lanczos algorithm (Lanczos, 95) for A started with the pair (y, A ν x ) Rutishauser never mentions Padé approximants, but he had no need, since they are just the convergents of the J fractions and S fractions For him, actually only the FOPs in the denominator matter Later, , Householder, Gragg, and Stewart stress the connection to Padé approximants in several papers NB: Hadamard s theorem (89) de Montessus de Ballore s theorem (9/95) Martin H Gutnecht and Beresford N Parlett

19 Associated polynomials: recurrences The p table can be built up from the initial column p (ν) by the left-to-right recurrence p (ν) (z) := zp (ν+) (z) q(ν) p (ν) (z) (6) Rutishauser (954a) derived also a top-to-bottom recurrence p (ν+) (z) := p (ν) (z) p (ν+) (z) (7) and the diagonal 3-term recurrence (with :, p (ν) : ) [ ] p (ν) + (z) := z q (ν) + e(ν) p (ν) (z) + q(ν) p (ν) (z) (8) Martin H Gutnecht and Beresford N Parlett

20 Further relations and applications So, in addition to introducing and investigating the qd algorithm Rutishauser (954a) [rec 5 Aug 953] (954b) [rec 8 Sep 953], (955a) [rec 9 July 954)] explained many connections to other topics and gave many applications; eg, in (954a): the connection to continued fractions, the connection to the Lanczos BIO algorithm, the connection to the CG algorithm, computing partial fraction decompositions of rational fcts In (954b): summation of badly converging series, solving algebraic equations = computing zeros of polynomials, quadratic convergence by using shifts / double shifts Martin H Gutnecht and Beresford N Parlett

21 Further relations and applications (cont d) In (955a): computing EVals by combining Lanczos BIO alg and the progressive qd algorithm, computing EVecs (several new algorithms are suggested), EVals and EVecs of infinite matrices Still missing: tridiagonal matrices (except for computing shifts), LU decomposition of these tridiagonal matrices, LR algorithm Martin H Gutnecht and Beresford N Parlett

22 FOPs and tridiagonal matrices Rutishauser new well (see Rutishauser (953) on the Lanczos BIO algorithm) that associated to the 3-term recurrence (8), T (ν) n = [ ] p (ν) + (z) := z q (ν) + e(ν) p (ν) (z) + q(ν) p (ν) (z) (with fixed ν) there is a nested set of tridiagonal matrices q (ν) q(ν) + q (ν) q(ν) + q (ν) 3 n q(ν) n such that p (ν) n (z) is the characteristic polynomial of T (ν) n Since he was interested in the limit of the zeros of p n (ν) ν it was natural to loo at T (ν) n Martin H Gutnecht and Beresford N Parlett n + q(ν) n as

23 LU (LR) decomposition of a tridiagonal matrix Clearly, T (ν) n has the LU decomposition (LR-Zerlegung) T (ν) n = L (ν) n R (ν) n (9) with L (ν) n = n, R (ν) n = q (ν) q (ν) q (ν) 3 q (ν) n Martin H Gutnecht and Beresford N Parlett

24 The LU (LR) transformation At some historic moment in 954, Rutishauser must have realized that his progressive qd algorithm (9) can be interpreted as computing this LU factorization T (ν) n = L (ν) n R (ν) n and then forming + q (ν) R (ν) n L (ν) n = = T (ν+) n q(ν) + q (ν) q(ν) q (ν) 3 n q(ν) n q (ν) n Martin H Gutnecht and Beresford N Parlett

25 The LR transformation (cont d) So, the qd algorithm consists of performing the step T (ν) n = L (ν) n R (ν) n R (ν) n L (ν) n = T (ν+) n called LR transformation, which is a similarity transformation: ( ) T (ν+) n = R (ν) n T (ν) n R (ν) n The liely motivation: Diagonals ( rows ) of qd table correspond to J fractions, FOPS (Lanczos polynomials), and tridiagonal matrices Rhombus rules lead us from one diagonal to the next They are matched by construction with J and S fractions There are corresponding rules for the polynomials Hence, there must be a rule for transforming one tridiagonal matrix into the next Martin H Gutnecht and Beresford N Parlett

26 The LR algorithm LR algorithm: succession of LR transformations (LR steps) Convergence of ( =,,n) as ν means: Convergence of L (ν) n to diagonal matrix as ν, Convergence of T (ν) n to upper bidiagonal matrix as ν, The diagonals of T (ν) n A, and R (ν) n ultimately contain eigenvalues of Generalization to full matrices is immediate, but unimportant Martin H Gutnecht and Beresford N Parlett

27 The LR algorithm: publications The first two publication on the LR algorithm were in French, two two-page notes in the Comptes Rendues: Rutishauser (955e) [séance du 3 janvier 955], Rutishauser/Bauer (955) [séance du 5 avril 955] 956 Rutishauser produced a mimeographed 5-page ETH research report in English, entitled Report on the Solution of Eigenvalue Problems with the LR transformation Two years later it got properly published in a National Bureau of Standards (NBS) boo series (Rutishauser, 958a) In the same issue: Henrici s review paper on the qd algorithm, and Stiefel s paper on ernel polynomials in NLA In 957, Rutishauser included a 5-page appendix on the LR transformation in the (German) boolet that compiled and updated most of his previous wor on qd (Rutishauser, 957a) Martin H Gutnecht and Beresford N Parlett

28 Conclusions The discovery of the qd and the LR algorithms probably evolved in the following steps: Generalizing Aiten s wor qd table / algorithm Considering the corresponding p table (gen Lanczos wor) and finding the diagonal 3-term recurrence for this table Maing the connection to continued fractions and Lanczos polynomials (and as well to many other topics) Maing the connection to tridiagonal matrices Noticing their extremely simple LU decomposition Noticing that qd algorithm = LR algorithm for tridiagonal matrices Generalizing the LR algorithm to full matrices Martin H Gutnecht and Beresford N Parlett