Efficient Estimation of the A-norm of the Error in the Preconditioned Conjugate Gradient Method

Transcription

1 Efficient Estimation of the A-norm of the Error in the Preconditioned Conjugate Gradient Method Zdeněk Strakoš and Petr Tichý Institute of Computer Science AS CR, Technical University of Berlin. Emory University, February 2006.

2 A system of linear algebraic equations Consider a system of linear algebraic equations Ax = b where A R n n is symmetric positive definite, b R n. Discretization of elliptic self-adjoint partial differential equations: Energy norm. The conjugate gradient method minimizes at the jth step the energy norm of the error on the given j-dimensional Krylov subspace. Z. Strakoš and P. Tichý 2

3 The conjugate gradient method (CG) Given x 0 R n, r 0 = b Ax 0. CG computes a sequence of iterates x j, so that where x x j A = x j x 0 + K j (A, r 0 ) min x u A, u x 0 +K j (A,r 0 ) K j (A, r 0 ) span {r 0,Ar 0,,A j 1 r 0 }, x x j A ( (x x j ),A(x x j ) ) 1 2. Z. Strakoš and P. Tichý 3

4 CG Hestenes and Stiefel (1952) given x 0, r 0 = b Ax 0, p 0 = r 0, for j = 0, 1, 2,... γ j = (r j, r j ) (p j,ap j ) x j+1 = x j + γ j p j r j+1 = r j γ j Ap j δ j+1 = (r j+1, r j+1 ) (r j, r j ) p j+1 = r j+1 + δ j+1 p j Z. Strakoš and P. Tichý 4

5 Preconditioned Conjugate Gradients (PCG) The CG-iterates are thought of being applied to ˆx = ˆb. We consider symmetric preconditioning Change of variables  = L 1 AL T, ˆb = L 1 b. M LL T, γ j ˆγ j, δ j ˆδ j, x j L T ˆx j, r j L ˆr j, s j M 1 r j, p j L T ˆp j. The preconditioner M is chosen so that a linear system with the matrix M is easy to solve while the matrix L 1 AL T should ensure fast convergence of CG. Z. Strakoš and P. Tichý 5

6 Algorithm of PCG given x 0, r 0 = b Ax 0, s 0 = M 1 r 0, p 0 = s 0, for j = 0, 1, 2,... γ j = (r j, s j ) (p j,ap j ) x j+1 = x j + γ j p j r j+1 = r j γ j Ap j s j+1 = M 1 r j+1 δ j+1 = (r j+1, s j+1 ) (r j, s j ) p j+1 = s j+1 + δ j+1 p j Z. Strakoš and P. Tichý 6

7 When to stop? When to stop? Should we bother? fair comparison of the direct and iterative principles iterative principle direct principle accuracy needed 10 2 accuracy of the approximate solution computational cost Z. Strakoš and P. Tichý 7

8 How to measure a quality of approximation? Measuring accuracy?... it depends on a problem. using residual information, normwise backward error, relative residual norm. using error estimates, estimate of the A-norm of the error, estimate of the Euclidean norm of the error. If the original system is well-conditioned - it does not matter. Z. Strakoš and P. Tichý 8

9 A message from history Using of the residual vector r j as a measure of the goodness of the estimate x j is not reliable [HeSt-52, p. 410]. The function (x x j,a(x x j )) can be used as a measure of the goodness of x j as an estimate of x [HeSt-52, p. 413]. Z. Strakoš and P. Tichý 9

10 Various convergence characteristics 10 4 Example using [GuSt-00], n = normwise relative backward error relative euclidean norm of the error relative residual norm relative A norm of the error Z. Strakoš and P. Tichý 10

11 Outline 1. CG and Gauss Quadrature 2. Construction of estimates in CG and PCG 3. Estimates in finite precision arithmetic 4. Rounding error analysis 5. Numerical experiments 6. Conclusions Z. Strakoš and P. Tichý 11

12 1. CG and Gauss Quadrature Z. Strakoš and P. Tichý 12

13 CG and Gauss Quadrature Ax = b, x 0 T j y j = r 0 e 1 x j = x 0 +Q j y j ξ ζ j i=1 f(λ) dω(λ) ( ω (j) i f θ (j) i ) ω (j) ω(λ) Z. Strakoš and P. Tichý 13

14 CG and Gauss Quadrature At any iteration step j, CG represents the matrix formulation of the j-point Gauss quadrature of the Riemann-Stieltjes integral determined by A and r 0, ξ ζ f(λ) dω(λ) = j i=1 ω (j) i f(θ (j) i ) + R j (f). For f(λ) λ 1 the formula takes the form x x 0 2 A r 0 2 = j-th Gauss quadrature + x x j 2 A r 0 2. This was a base for CG error estimation in [DaGoNa-78, GoFi-93, GoMe-94, GoSt-94, GoMe-97,... ]. Z. Strakoš and P. Tichý 14

15 Equivalent formulas Continued fractions [GoMe-94, GoSt-94, GoMe-97] r 0 2 C n = r 0 2 C j + x x j 2 A. Warnick [Wa-00] r T 0 (x x 0 ) = r T 0 (x j x 0 ) + x x j 2 A. Hestenes and Stiefel [HeSt-52, De-93, StTi-02] x x 0 2 A = j 1 i=0 γ i r i 2 + x x j 2 A. The last formula is derived purely algebraically! Z. Strakoš and P. Tichý 15

16 Local A-orthogonality Standard derivation of this formula uses global A-orthogonality among direction vectors [AxKa-01, p. 274], [Ar-04, p. 8]. It follows, however, from the local A-orthogonality and Pythagorean theorem, Z. Strakoš and P. Tichý 16

17 Hestenes and Stiefel formula (derivation) Using local orthogonality between r i+1 and p i, Then x x i 2 A x x i+1 2 A = γ i r i 2. x x 0 2 A x x j 2 A = = j 1 i=0 j 1 i=0 ( x xi 2 A x x i+1 2 A) γ i r i 2. The approach to derivation of this formula is very important for its justification in finite precision arithmetic. Z. Strakoš and P. Tichý 17

18 2. Construction of estimates in CG and PCG Z. Strakoš and P. Tichý 18

19 Construction of estimate in CG Idea: Consider, for example, x x j 2 A = r 0 2 [C n C j ]. Run d extra steps. Subtracting identity for x x j+d 2 A gives x x j 2 A = r 0 2 [C j+d C j ] }{{} EST 2 + x x j+d 2 A. When x x j 2 A x x j+d 2 A, we have a tight (lower) bound [GoSt-94, GoMe-97]. Z. Strakoš and P. Tichý 19

20 Mathematically equivalent estimates Continued fractions [GoSt-94, GoMe-97] η j,d = r 0 2 [C j+d C j ], Warnick [Wa-00] µ j,d = r T 0 (x j+d x j ), Hestenes and Stiefel [HeSt-52] ν j,d = j+d 1 i=j γ i r i 2. Z. Strakoš and P. Tichý 20

21 Construction of estimate in PCG The A-norm of the error can be estimated similarly as in ordinary CG. Extension of the Gauss Quadrature formulas based on continued fractions was published in [Me-99]. Extension of the HS estimate: use the HS formula for ˆx = ˆb and substitution  = L 1 AL T, ˆx j = L T x j, ˆγ i = γ i, ˆr i = L 1 r i [De-93, AxKa-01, StTi-04, Ar-04] ˆx ˆx j 2  }{{} x x j 2 A = j+d 1 i=j ˆγ i ˆr i 2  }{{} γ i (r i, s i ) 2 + ˆx ˆx j+d 2.  }{{} x x j+d 2 A In many problems it is convenient to use a stopping criterion that relates the relative A-norm of the error to a discretization error, see [Ar-04]. Z. Strakoš and P. Tichý 21

22 Estimating the relative A-norm of the error To estimate the relative A-norm of the error we use the identities x x j 2 A = ν j,d + x x j+d 2 A, x 2 A = ν 0,j+d + 2b T x 0 x 0 2 A }{{} ξ j+d 2 + x x j+d A. Define j,d ν j,d ξ j+d. If x A x x 0 A then j,d > 0 and j,d = x x j 2 A x x j+d 2 A x 2 A x x j+d 2 A x x j 2 A x 2 A. Z. Strakoš and P. Tichý 22

23 3. Estimates in finite precision arithmetic Z. Strakoš and P. Tichý 23

24 CG behavior in finite precision arithmetic orthogonality is lost, convergence is delayed! 10 0 (E) x x j A (FP) x x j A 10 2 (E) I V T j V j F (FP) I V T j V j F Z. Strakoš and P. Tichý 24

25 A principal, often overlooked, problem Are estimates derived assuming exact arithmetic applicable in finite precision computations? sophisticated error estimates and effects of rounding errors finite precision computation exact arithmetic computation 10 2 norm of the error which is to be estimated iteration number Z. Strakoš and P. Tichý 25

26 Estimates need not work The identity x x j 2 A = EST 2 + x x j+d 2 A need not hold during the finite precision CG computations. An example: µ j,d = r T 0 (x j+d x j ) does not work! 10 0 x x j A µ j,d 1/2 I V T j V j F Z. Strakoš and P. Tichý 26

27 4. Rounding error analysis Z. Strakoš and P. Tichý 27

28 Rounding error analysis Without a proper rounding error analysis, there is no justification whatsoever that the proposed estimates work in finite precision arithmetic. Do the estimates give good information in practical computations? estimate CG PCG η j,d (continued fractions) yes yes? [GoSt-94, GoMe-97, Me-99] µ j,d (Warnick) no no [StTi-02] ν j,d (Hestenes and Stiefel) yes yes [StTi-02, StTi-04] Based on [GrSt-92], [Gr-89], ε limit. Z. Strakoš and P. Tichý 28

29 Hestenes and Stiefel estimate (CG) [StTi-02]: Rounding error analysis based on - detailed investigation of preserving local orthogonality in CG, - results [Pa-71, Pa-76, Pa-80], [Gr-89, Gr-97]. Theorem: Let ε κ(a) 1. Then the CG approximate solutions computed in finite precision arithmetic satisfy x x j 2 A x x j+d 2 A = ν j,d + x x j A E j,d + O(ε 2 ), E j,d ( κ(a)) ε x x 0 A. Z. Strakoš and P. Tichý 29

30 Hestenes and Stiefel estimate (PCG) [StTi-04]: Analysis based on: - rounding error analysis from [StTi-02], - solving of Ms j+1 = r j+1 enjoys perfect normwise backward stability [Hi-96, p. 206]. Similar result as for CG: Until x x j A reaches a level close to ε x x 0 A, the estimate ν j,d = j+d 1 i=j γ i (r i, s i ) must work. Z. Strakoš and P. Tichý 30

31 5. Numerical Experiments Z. Strakoš and P. Tichý 31

32 Estimating the A-norm of the error P. Benner: Large-Scale Control Problems, optimal cooling of steel profiles, PCG, κ(a) = 9.7e + 04, n = 5177, d = 4, L = cholinc(a, 0) true residual norm normwise backward error A norm of the error estimate Z. Strakoš and P. Tichý 32

33 Estimating the A-norm of the error R. Kouhia: Cylindrical shell (Matrix Market), matrix s3dkt3m2, PCG, κ(a) = 3.62e + 11, n = 90499, d = 200, L = cholinc(a, 0) true residual norm normwise backward error A norm of the error estimate Z. Strakoš and P. Tichý 33

34 Estimating the relative A-norm of the error R. Kouhia: Cylindrical shell (Matrix Market), matrix s3dkt3m2, PCG, κ(a) = 3.62e + 11, n = 90499, d = 200, L = cholinc(a, 0) d = 200 d = 80 d = 10 d = relative A norm of the error estimate Z. Strakoš and P. Tichý 34

35 Estimating the relative A-norm of the error R. Kouhia: Cylindrical shell (Matrix Market), matrix s3rmt3m3, PCG, κ(a) = 2.40e + 10, n = 5357, d = 50, L = cholinc(a, 1e 5) relative A norm of the error numerically stable estimate numerically unstable estimate Z. Strakoš and P. Tichý 35

36 Reconstruction of the convergence curve R. Kouhia: Cylindrical shell (Matrix Market), matrix s3dkt3m2, PCG, κ(a) = 3.62e + 11, n = 90499, d = 100, L = cholinc(a, 0) relative A norm of the error estimate (d=100) Z. Strakoš and P. Tichý 36

37 6. Conclusions Various formulas (based on Gauss quadrature) are mathematically equivalent to the formulas present (but somehow hidden) in the original Hestenes and Stiefel paper. Hestenes and Stiefel estimate is very simple, it can be computed almost for free and it has been proved numerically stable. We suggest to incorporate the estimates ν 1/2 j,d software realizations of the CG and PCG methods. and 1/2 j,d into any The estimates are tight if the A-norm of the error reasonably decreases. Open problem: The adaptive choice of the parameter d. Z. Strakoš and P. Tichý 37

38 More details can be found in Strakoš, Z. and Tichý, P., On error estimation in the Conjugate Gradient method and why it works in finite precision computations, Electronic Trans. Numer. Anal. (ETNA), 13: pp , (2002). Strakoš, Z. and Tichý, P., Simple estimation of the A-norm of the error in the Preconditioned Conjugate Gradients, BIT Numerical Mathematics, 45: pp , (2005). strakos, tichy Meurant, G. and Strakoš, Z., The Lanczos and conjugate gradient methods in finite precision arithmetic, Acta Numerica, 15 (to appear in 2006). Z. Strakoš and P. Tichý 38

39 Thank you for your kind attention! Z. Strakoš and P. Tichý 39