Contribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computa

Transcription

1 Contribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computations ªaw University of Technology Institute of Mathematics and Computer Science Warsaw, October 7, 2006

2 Lanczos method C. Lanczos, An iteration method for the solution of the eigenvalue problem of linear dierential and integral equation, J. Res. Nat. Bur. Standards 45 (1950), Conjugate gradient method M.R. Hestenes, E. Stiefel, Methods of conjugate gradients for solving linear systems, J. Nat. Bur. Standards 49 (1952),

3 Pioneering papers of Wo¹niakowski 1 H. Wo¹niakowski, Numerical stability of the Chebyshev method for the solution of large linear systems, Numer. Math. 28 (1977), H. Wo¹niakowski, Roundo error analysis iterations for large linear systems, Numer. Math. 30 (1978), H. Wo¹niakowski, Roundo-error analysis of a new class of conjugate-gradient algorithms, Linear Alg. Appl. 29 (1980),

4 Pioneering papers of Wo¹niakowski 1 H. Wo¹niakowski, Numerical stability of the Chebyshev method for the solution of large linear systems, Numer. Math. 28 (1977), H. Wo¹niakowski, Roundo error analysis iterations for large linear systems, Numer. Math. 30 (1978), H. Wo¹niakowski, Roundo-error analysis of a new class of conjugate-gradient algorithms, Linear Alg. Appl. 29 (1980),

5 Pioneering papers of Wo¹niakowski 1 H. Wo¹niakowski, Numerical stability of the Chebyshev method for the solution of large linear systems, Numer. Math. 28 (1977), H. Wo¹niakowski, Roundo error analysis iterations for large linear systems, Numer. Math. 30 (1978), H. Wo¹niakowski, Roundo-error analysis of a new class of conjugate-gradient algorithms, Linear Alg. Appl. 29 (1980),

6 Pioneering PhD thesis 1 C.C. Paige, The computation of eigenvalues and eigenvectors of very large sparse matrices, University of London A. Greenbaum, Convergence properties of the conjugate gradient algorithm in exact and nite precision arithmetic, University of California, Berkeley Review paper G. Meurant, Z. Strako², The Lanczos and conjugate gradient algorithms in nite precision arithmetic, Acta Numerica 2006,

7 Pioneering PhD thesis 1 C.C. Paige, The computation of eigenvalues and eigenvectors of very large sparse matrices, University of London A. Greenbaum, Convergence properties of the conjugate gradient algorithm in exact and nite precision arithmetic, University of California, Berkeley Review paper G. Meurant, Z. Strako², The Lanczos and conjugate gradient algorithms in nite precision arithmetic, Acta Numerica 2006,

8 Pioneering PhD thesis 1 C.C. Paige, The computation of eigenvalues and eigenvectors of very large sparse matrices, University of London A. Greenbaum, Convergence properties of the conjugate gradient algorithm in exact and nite precision arithmetic, University of California, Berkeley Review paper G. Meurant, Z. Strako², The Lanczos and conjugate gradient algorithms in nite precision arithmetic, Acta Numerica 2006,

9 Roundo error analysis of iterations for linear systems Ax = b, A Hermitian positive denite α exact solution, y computed solution in Numerical stability of method y α is of order 2 t A A 1 α Well behaviour (A + A)y = b, A is of order 2 t A

10 Numerical stability of iterative method lim x k α k x k computed sequence in is of order 2 t A A 1 α Wo¹niakowski 1977 Chebyshev method is numerically stable, but not well-behaved residuals Ax k b are of order 2 t A 2 A 1 α

11 Numerical stability of iterative method lim x k α k x k computed sequence in is of order 2 t A A 1 α Wo¹niakowski 1977 Chebyshev method is numerically stable, but not well-behaved residuals Ax k b are of order 2 t A 2 A 1 α

12 Numerical stability of iterative method lim x k α k x k computed sequence in is of order 2 t A A 1 α Wo¹niakowski 1977 Chebyshev method is numerically stable, but not well-behaved residuals Ax k b are of order 2 t A 2 A 1 α

13 Chebyshev method x k α = W k (A)(x 0 α) W k (z) = T k(f (z)) T k (f (0)), f (z) = b + a b a 2 z b a T k (z) Chebyshev polynomial r k := Ax k b, x k+1 := x k +[p k 1(x k x k 1) r k ]/q k, new algorithm for computing p k and q k

14 Jacobi, Gauss-Seidel, SOR A = A H > 0 has property A consistently ordered, A = D L U, γd 1 U + γ 1 D 1 U [ ] A = D 1 A 12 A 21 D 2 Wo¹niakowski 1978 Jacobi, Richardson, SOR methods are numerically stable, but not well-behaved Gauss-Seidel is well-behaved

15 Jacobi, Gauss-Seidel, SOR A = A H > 0 has property A consistently ordered, A = D L U, γd 1 U + γ 1 D 1 U [ ] A = D 1 A 12 A 21 D 2 Wo¹niakowski 1978 Jacobi, Richardson, SOR methods are numerically stable, but not well-behaved Gauss-Seidel is well-behaved

16 Conjugate gradient method cg Ax = b A = A H > 0, order n In exact arithmetic cg generates orthogonal residual vectors r k = b Ax k < r i, r j >= 0 in exact arithmetic α = A 1 b is obtained after at most n steps.

17 Conjugate gradient method cg Ax = b A = A H > 0, order n In exact arithmetic cg generates orthogonal residual vectors r k = b Ax k < r i, r j >= 0 in exact arithmetic α = A 1 b is obtained after at most n steps.

18 Hestenes-Stiefel formulation of cg given x 0, r 0 = b Ax 0, p 0 = r 0 for j = 1, 2,... x j = x j 1 + γ j 1p j 1 γ j 1 = < r j 1, r j 1 < p j, Ap j 1 >, δ j = < r j, r j > < r j 1, r j 1 > p j = r j + δ j p j 1, r j = r j 1 γ j 1Ap j 1,

19 Wo¹niakowski 1980 new class of conjugate gradient algorithms numerical stability of these algorithms with iterative renement numerical well behaviour if 2 t [cond2(a)] 2 is at most of order unity Gatlinburg VII at Asilomar 1977

20 Wo¹niakowski 1980 new class of conjugate gradient algorithms numerical stability of these algorithms with iterative renement numerical well behaviour if 2 t [cond2(a)] 2 is at most of order unity Gatlinburg VII at Asilomar 1977

21 Wo¹niakowski 1980 new class of conjugate gradient algorithms numerical stability of these algorithms with iterative renement numerical well behaviour if 2 t [cond2(a)] 2 is at most of order unity Gatlinburg VII at Asilomar 1977

22 Wo¹niakowski algorithm Wo¹niakowski cg r k = Ax k b, z k = x k c k r k y k = x k 1 z k, c k = < r k, r k > < r k, Ar k > x k+1 = z k u k y k algorithm for computation of u k explicitly is not given Classical version of cg: x k+1 = x k + γ k p k

23 In: Accuracy and Stability of Numerical Algorithms Higham writes: 1 Wo¹niakowski (1980) analyses a class of conjugate gradient algorithms (which does not include the usual conjugate gradient method). 2 Wo¹niakowski obtains a forward error bound proportional to [cond(a)] 3/2 and a residual bound proportional to cond(a).

24 In: Accuracy and Stability of Numerical Algorithms Higham writes: 1 Wo¹niakowski (1980) analyses a class of conjugate gradient algorithms (which does not include the usual conjugate gradient method). 2 Wo¹niakowski obtains a forward error bound proportional to [cond(a)] 3/2 and a residual bound proportional to cond(a).

25 In: Accuracy and Stability of Numerical Algorithms Higham writes: 1 Greenbaum (1989) presents a detailed error analysis of the conjugate gradient method, but her concern is with the rate of convergence rather than the attainable accuracy. 2 Notay (1993) analyses how rounding errors inuence the convergence rate of the conjugate gradient method for matrices with isolated eigenvalues at the ends of the spectrum.

26 In: Accuracy and Stability of Numerical Algorithms Higham writes: 1 Greenbaum (1989) presents a detailed error analysis of the conjugate gradient method, but her concern is with the rate of convergence rather than the attainable accuracy. 2 Notay (1993) analyses how rounding errors inuence the convergence rate of the conjugate gradient method for matrices with isolated eigenvalues at the ends of the spectrum.

27 Wo¹niakowski class of cg algorithms For these algorithms there exists computed vector x k such that A 1/2 (x k α) c n 2 t A 1/2 x k If A 1/2 x k is of order A 1/2 x k then these cg algorithms are numerically stable in A-norm: A 1/2 (x k α) = 0(2 t cond(a) A 1/2 x k ), but not well behaved

28 Relationship between cg and Lanczos Krylov subspace A n n, nonsingular sym. posit. def., v 2 = 1 K k (v, A) = span{v, Av,..., A k 1 v}, Ax = b, r 0 = b Ax 0 x k = x 0 + V k y k V k is the matrix of orthonormal basis of K k (v 1, A), where Krylov subspace is generated by Lanczos algorithm with v 1 = r 0 / r 0

29 Conjugate gradient method cg and Lanczos Matrix notation Lanczos algorithm AV k = V k T k + η k+1v k+1e T k, T k sym. tridiag. ( ) T k y k = r 0 e 1, x k = x 0 + V k y k (*) is equivalent to cg method of Hestenes and Stiefel

30 Properties of cg If r k = b Ax k = r 0 AV k y k is orthogonal to V k then r n+1 = 0 r 0 = b Ax 0, v 1 = r 0 / r 0 K k (v 1, A) = span{r 0,..., r k 1} < r i, r j >= 0, r k K k (v 1, A) v k+1 = ( 1) k r k r k

31 Properties of cg If r k = b Ax k = r 0 AV k y k is orthogonal to V k then r n+1 = 0 r 0 = b Ax 0, v 1 = r 0 / r 0 K k (v 1, A) = span{r 0,..., r k 1} < r i, r j >= 0, r k K k (v 1, A) v k+1 = ( 1) k r k r k

32 Properties of cg If r k = b Ax k = r 0 AV k y k is orthogonal to V k then r n+1 = 0 r 0 = b Ax 0, v 1 = r 0 / r 0 K k (v 1, A) = span{r 0,..., r k 1} < r i, r j >= 0, r k K k (v 1, A) v k+1 = ( 1) k r k r k

33 Properties of cg If r k = b Ax k = r 0 AV k y k is orthogonal to V k then r n+1 = 0 r 0 = b Ax 0, v 1 = r 0 / r 0 K k (v 1, A) = span{r 0,..., r k 1} < r i, r j >= 0, r k K k (v 1, A) v k+1 = ( 1) k r k r k

34 Error norms in cg 1 Lanczos and cg can be formulated in terms of orthogonal polynomials and Gauss quadrature of some integral determined by A, v 1 2 Lanczos and cg can be viewed as matrix representations of Gauss quadrature 3 A norm of the error x k α and Euclidean norm of the error in cg can be computed using Gauss quadrature.

35 Error norms in cg 1 Lanczos and cg can be formulated in terms of orthogonal polynomials and Gauss quadrature of some integral determined by A, v 1 2 Lanczos and cg can be viewed as matrix representations of Gauss quadrature 3 A norm of the error x k α and Euclidean norm of the error in cg can be computed using Gauss quadrature.

36 Error norms in cg 1 Lanczos and cg can be formulated in terms of orthogonal polynomials and Gauss quadrature of some integral determined by A, v 1 2 Lanczos and cg can be viewed as matrix representations of Gauss quadrature 3 A norm of the error x k α and Euclidean norm of the error in cg can be computed using Gauss quadrature.

37 Error norms in cg 1 Computing A norm of the error x k α is closely related to approximating quadratic forms. 2 This has been studied extensively by Gene Golub with many collaborators during the last thirty-ve years (see the review paper of Meurant and Strako²)

38 Error norms in cg 1 Computing A norm of the error x k α is closely related to approximating quadratic forms. 2 This has been studied extensively by Gene Golub with many collaborators during the last thirty-ve years (see the review paper of Meurant and Strako²)

39 Lanczos and cg in nite precision 1 What happens numerically to the equivalence of Lanczos and cg as well as to the equivalence with orthogonal polynomials and Gauss quadrature? 2 How do we evaluate convergence of cg in nite precision arithmetic? 3 see Z. Strako² and P. Tichy, On error estim. in cg and why it works in nite precision computation, Electr. Trans. Numer. Anal. 13 (2002).

40 Lanczos and cg in nite precision 1 What happens numerically to the equivalence of Lanczos and cg as well as to the equivalence with orthogonal polynomials and Gauss quadrature? 2 How do we evaluate convergence of cg in nite precision arithmetic? 3 see Z. Strako² and P. Tichy, On error estim. in cg and why it works in nite precision computation, Electr. Trans. Numer. Anal. 13 (2002).

41 Lanczos and cg in nite precision 1 What happens numerically to the equivalence of Lanczos and cg as well as to the equivalence with orthogonal polynomials and Gauss quadrature? 2 How do we evaluate convergence of cg in nite precision arithmetic? 3 see Z. Strako² and P. Tichy, On error estim. in cg and why it works in nite precision computation, Electr. Trans. Numer. Anal. 13 (2002).

42 Conclusion from Strako² and Tichy talk: Hestenes and Stiefel cg (1952) should be celebrated, but also studied, even after 50 years!

43 Most important of all My congratulations to Henryk who was the pioneer in this eld!!!

44 Many happy and fruitful years!!!