7.3 The Jacobi and Gauss-Siedel Iterative Techniques. Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.

Transcription

1 7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.

2 7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 0 Matrix splitting a 2,1 0 A = diag a 1,1, a 2,2,, a n,n ) a n 1,1 a n 1,2 0 a n,1 a n,2 a n,n a 1,2 a 1,n 1 a 1,n 0 a 2,n 1 a 2,n a n 1,n 0

3 7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 0 Matrix splitting a 2,1 0 A = diag a 1,1, a 2,2,, a n,n ) a n 1,1 a n 1,2 0 a n,1 a n,2 a n,n a 1,2 a 1,n 1 a 1,n 0 a 2,n 1 a 2,n a n 1,n 0 def = D L U =.

4 Ex: Matrix splitting for A = A = = diag 10, 11, 10, 8)

5 The Jacobi and Gauss-Siedel Methods for solving Ax = b Jacobi Method: With matrix splitting A = D L U, rewrite x = D 1 L + U) x + D 1 b. Jacobi iteration with given x 0), x k+1) = D 1 L + U) x k) + D 1 b, for k = 0, 1, 2,.

6 The Jacobi and Gauss-Siedel Methods for solving Ax = b Jacobi Method: With matrix splitting A = D L U, rewrite x = D 1 L + U) x + D 1 b. Jacobi iteration with given x 0), x k+1) = D 1 L + U) x k) + D 1 b, for k = 0, 1, 2,. Gauss-Siedel Method: Rewrite x = D L) 1 U x + D L) 1 b. Gauss-Siedel iteration with given x 0), x k+1) = D L) 1 U x k) + D L) 1 b, for k = 0, 1, 2,.

7 Ex: Jacobi Method for Ax = b, with A = , b = A = D L U = diag 10, 11, 10, 8)

8 Ex: Jacobi Method for Ax = b, with A = , b = A = D L U = diag 10, 11, 10, 8) Jacobi iteration with x 0) = 0, for k = 0, 1, 2, x k+1) J = D 1 L + U) x k) J + D 1 b = xk) J

9 Ex: Gauss-Siedel Method for Ax = b A = D L U =

10 Ex: Gauss-Siedel Method for Ax = b A = D L U = Gauss-Siedel iteration with x 0) = 0, for k = 0, 1, 2, x k+1) GS = D L) 1 U x GS + D L) 1 b 1 10 = xk) GS

11 Jacobi vs. Gauss-Siedel: solution x = Convergence Comparision, Jacobi vs. G-S 10-2 Jacobi G-S

12 General Iteration Methods To solve A x = b with matrix splitting A = D L U, Jacobi Method: x k+1) J Gauss-Siedel Method: x k+1) GS = D 1 L + U) x k) J + D 1 b. = D L) 1 U x k) GS + D L) 1 b. General Iteration Method: for k = 0, 1, 2, x k+1) = T x k) + c. Next: convergence analysis on General Iteration Method

13 General Iteration: x k+1) = T x k) + c for k = 0, 1, 2, Thm: The following statements are equivalent ρt ) < 1. The equation x = T x + c 1) has a unique solution and {x k) } converges to this solution from any x 0).

14 General Iteration: x k+1) = T x k) + c for k = 0, 1, 2, Thm: The following statements are equivalent ρt ) < 1. The equation x = T x + c 1) has a unique solution and {x k) } converges to this solution from any x 0). Proof: Assume ρt ) < 1. Then 1) has unique solution x ). x k+1) x ) = T x k) x )) = T 2 x k 1) x )) Conversely, if omitted) = = T k+1 x 0) x )) = 0.

15 Jacobi on random upper triangular matrix A = D U. T = D 1 U with ρt ) = 0. 0 A is randn upper triangular with n = nz = 1275 Convergence plot G-S Convergence on upper triangular matrix

16 7.4 Relaxation Techniques for Solving Linear Systems To solve A x = b with matrix splitting A = D L U, rewrite D x = D x, ω L x = ω D U) x ω b, for any ω.

17 7.4 Relaxation Techniques for Solving Linear Systems To solve A x = b with matrix splitting A = D L U, rewrite D x = D x, ω L x = ω D U) x ω b, for any ω. Taking difference of two equations, D ω L) x = 1 ω) D + ω U) x + ω b.

18 7.4 Relaxation Techniques for Solving Linear Systems To solve A x = b with matrix splitting A = D L U, rewrite D x = D x, ω L x = ω D U) x ω b, for any ω. Taking difference of two equations, D ω L) x = 1 ω) D + ω U) x + ω b. Successive Over-Relaxation SOR), for k = 0, 1, 2, x k+1) SOR = D ω L) 1 1 ω) D + ω U) x k) SOR + ω D ω L) 1 b def = T SOR x k) SOR + c SOR.

19 7.4 Relaxation Techniques for Solving Linear Systems To solve A x = b with matrix splitting A = D L U, rewrite D x = D x, ω L x = ω D U) x ω b, for any ω. Taking difference of two equations, D ω L) x = 1 ω) D + ω U) x + ω b. Successive Over-Relaxation SOR), for k = 0, 1, 2, x k+1) SOR = D ω L) 1 1 ω) D + ω U) x k) SOR + ω D ω L) 1 b def = T SOR x k) SOR + c SOR. converges if ρ T SOR ) < 1. Good choice of ω is tricky, but critical for accelerated convergence

20 Optimal SOR parameters Thm: If A is symmetric positive definite and tridiagonal, then ρ T GS ) = ρ T J )) 2 < 1, and the optimal choice of ω for the SOR method is ω OPT = ρ T J )) 2 with ρ T SOR ) = ω OPT 1 = 1 + ρ T J ) 1 ρ T J )) 2 2.

21 A = , b = 1 1 1, x = 1 3 If A is symmetric positive definite and tridiagonal, ) 4 3 det A) = 24, det = 7, 4 >

22 A = , b = 1 1 1, x = 1 3 If A is symmetric positive definite and tridiagonal, ) 4 3 det A) = 24, det = 7, 4 > ρt J ) = T J = D 1 L + U) = Optimal ω: 2 ω OPT = = ω OPT = ρ T J )) 2, b =

23 A = , b = 1 1 1, x = Convergence Comparision, G-S vs. SOR 10-2 G-S SOR

24 7.5 Error Bounds and Iterative Refinement Assume that x is an approximation to the solution x of A x = b. Residual r def = b A x = A x x). Thus small x x implies small r.

25 7.5 Error Bounds and Iterative Refinement Assume that x is an approximation to the solution x of A x = b. Residual r def = b A x = A x x). Thus small x x implies small r. However, big x x can still lead to small r. Ex: τ 2 ) x = τ ) ) 1 3 Exact solution x =. Bad approximation x = 1 0 has a small residual for large τ: ) ) ) r = τ τ = τ ). ).

26 Near Linear Dependence For τ = 4, equations define two nearly parallel lines l 1 : x x 2 = 3, and l 2 : x x 2 = Parallel lines do not have intersections.

27 Let A x = b with non-singular A and non-zero b T hm: Assume x is an approximate solution with r = b A x. Then for any natural norm, x x A 1 r, x x x κ A) r b, where κ A) def = A A 1 is the condition number of A

28 Let A x = b with non-singular A and non-zero b T hm: Assume x is an approximate solution with r = b A x. Then for any natural norm, x x A 1 r, x x x κ A) r b, where κ A) def = A A 1 is the condition number of A A is well-conditioned if κ A) = O1): small residual implies small solution error. A is ill-conditioned if κ A) 1: small residual may still allow large solution error.

29 Ex: Condition Number for A = τ 2 ) Solution: For τ > 0, A = τ. Since A 1 = 10τ τ ) 1 ), we have A 1 = 2 10 τ. Thus κ A) = A A 1 = 6 10 τ + 2. κ A) grows exponentially in τ. A is ill-conditioned for large τ.

30 Iterative Refinement I) Let A x = b with non-singular A and non-zero b. Let F ) be an in-exact equation solver, so F b) is approximate solution. Assume F ) is accurate enough that there exists a ρ < 1 so b A F b) b ρ for any b 0.

31 Iterative Refinement I) Let A x = b with non-singular A and non-zero b. Let F ) be an in-exact equation solver, so F b) is approximate solution. Assume F ) is accurate enough that there exists a ρ < 1 so b A F b) b ρ for any b 0. In practice, F ) could be from an in-exact) LU factorization, F b) = U 1 L 1 b ). Inaccuracies in LU factorization could be due to rounding-error, A L U.

32 Ex: A = randn n, n), b = randn n, 1), n = 3000 LU factorize A to get L, U, x 0 = U 1 L 1 b ), LU without pivoting) r 0) = b A x 0, x 1 = U 1 L 1 ) r 0, r 1 = r 0 A x 1, x = x 0 + x 1 disp normr 0 ), normr 1 )) e e-16

33 Iterative Refinement II) Given a tolerance τ > 0 and x 0) Initialize r 0) = b A x 0). for k = 0, 1, Compute x k) = F r k)), x k+1) = x k) + x k), r k+1) = r k) A x k). If r k+1) τ b stop.

34 Iterative Refinement II) Given a tolerance τ > 0 and x 0) Initialize r 0) = b A x 0). for k = 0, 1, Compute x k) = F r k)), x k+1) = x k) + x k), r k+1) = r k) A x k). If r k+1) τ b stop. Convergence Proof: r k+1) r ρ k) ρ 2 r k 1) ρ k+1 r 0).

35 Perturbation Theory Thm: Let x and x be solutions to A x = b and A + A) x = b + b with perturbations A and b. Then x x x κ A) 1 κ A) A A with κ A) = A A 1. A A + b ). b

36 7.6 The Conjugate Gradient Method CG) for A x = b Assumption: A is symmetric positive definite SPD) A T = A, x T Ax 0 for any x, x T Ax = 0 if and only if x = 0.

37 7.6 The Conjugate Gradient Method CG) for A x = b Assumption: A is symmetric positive definite SPD) A T = A, x T Ax 0 for any x, x T Ax = 0 if and only if x = 0. Thm: The vector x solves the SPD equations A x = b if and only if it minimizes function g x) def = x T Ax 2 x T b.

38 7.6 The Conjugate Gradient Method CG) for A x = b Assumption: A is symmetric positive definite SPD) A T = A, x T Ax 0 for any x, x T Ax = 0 if and only if x = 0. Thm: The vector x solves the SPD equations A x = b if and only if it minimizes function Proof: Let A x = b. Then g x) def = x T Ax 2 x T b. g x) = x T Ax 2 x T A x = x x ) T A x x ) x ) T A x ) = x x ) T A x x ) + g x ). Thus, g x) g x ) for all x; and g x) = g x ) iff x = x.

39 CG for A x = b Thm: The vector x solves the SPD equations A x = b if and only if it minimizes function g x) def = x T Ax 2 x T b. The CG Idea: Starting from an initial vector x 0), quickly compute new vectors x 1),, x k), with g x 0)) > g x 1)) > g x 2)) > > g x k)) > so that the sequence {x k) } will converge to x.

40 search direction and line search The CG Idea: Starting from an initial vector x 0), quickly compute new vectors x 1),, x k), with g x 0)) > g x 1)) > g x 2)) > > g x k)) > so that the sequence {x k) } will converge to x.

41 search direction and line search The CG Idea: Starting from an initial vector x 0), quickly compute new vectors x 1),, x k), with g x 0)) > g x 1)) > g x 2)) > > g x k)) > so that the sequence {x k) } will converge to x. Descent Method: Assume a search direction v k) at iteration x k 1), next iteration with step-size t k x k) def = x k 1) + t k v k) minimizes g x k 1) + tv k)).

42 search direction and line search The CG Idea: Starting from an initial vector x 0), quickly compute new vectors x 1),, x k), with g x 0)) > g x 1)) > g x 2)) > > g x k)) > so that the sequence {x k) } will converge to x. Descent Method: Assume a search direction v k) at iteration x k 1), next iteration with step-size t k x k) def = x k 1) + t k v k) minimizes g x k 1) + tv k)). Optimality Condition: 0 = d d t g x k 1) + tv k)) = v k)) T g x k 1) + tv k)) = v k)) T 2A x k 1) + tv k)) ) 2 b,

43 search direction and line search The CG Idea: Starting from an initial vector x 0), quickly compute new vectors x 1),, x k), with g x 0)) > g x 1)) > g x 2)) > > g x k)) > so that the sequence {x k) } will converge to x. Descent Method: Assume a search direction v k) at iteration x k 1), next iteration with step-size t k x k) def = x k 1) + t k v k) minimizes g x k 1) + tv k)). Optimality Condition: 0 = d d t g x k 1) + tv k)) = v k)) T g x k 1) + tv k)) = v k)) T 2A x k 1) + tv k)) ) 2 b, v k) ) T r k 1) ) t k = ) v k) T ), r k 1) def = b A x k 1) residual). A v k)

44 search direction choices For a small step-size t: g x k 1) + tv k)) g x k 1)) + t v k)) T g x k 1)). steepest descent: Greatest decrease in the value of g x k 1) + tv k)) : v k) = g x k 1)). A-orthogonal directions: non-zero vectors {v i) } n i=1 v i)) T A v j)) = 0 for all i j. A-orthogonal vectors associated with the positive definite matrix A is linearly independent.

45 A orthogonality Craft

46 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n v k) ) T r k 1) ) t k = v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v τ k v k ) = g x 0 + t 1 v t k v k ).

47 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n v k) ) T r k 1) ) t k = v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v τ k v k ) = g x 0 + t 1 v t k v k ). Magic I): min τ1,τ 2 g min x g x) = min τ1,,τ n g min τ1 g x 0 + τ 1 v 1)) = g x 0 + τ 1 v 1) + τ 2 v 2)) = g x 0 + τ 1 v 1) + + τ n v n)) = g x 0 + t 1 v 1)). x 0 + t 1 v 1) + t 2 v 2)). Thus x = x 0 + t 1 v 1) + + t n v n) is solution to A x = b. x 0 + t 1 v 1) + + t n v n)).

48 A orthogonality Craft

49 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n t k = v k) ) T r k 1) ) v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v τ k v k ) = g x 0 + t 1 v t k v k ).

50 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n t k = v k) ) T r k 1) ) v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v τ k v k ) = g x 0 + t 1 v t k v k ). Proof I): Let t = τ 1,, τ k ). Then g x 0 + τ 1 v τ k v k ) = g x 0 ) +t T v 1,, v k ) T A v 1,, v k ) t 2t T v 1,, v k ) T r 0), t g = 2 v 1,, v k ) T A v 1,, v k ) t v 1,, v k ) T r 0))

51 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n t k = v k) ) T r k 1) ) v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v τ k v k ) = g x 0 + t 1 v t k v k ). Proof I): Let t = τ 1,, τ k ). Then g x 0 + τ 1 v τ k v k ) = g x 0 ) +t T v 1,, v k ) T A v 1,, v k ) t 2t T v 1,, v k ) T r 0), t g = 2 v 1,, v k ) T A v 1,, v k ) t v 1,, v k ) T r 0)) min τ1,,τ k g x 0 + τ 1 v τ k v k ) t g = 0.

52 A orthogonality Craft

53 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n v k) ) T r k 1) ) t k = v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v 1) + + τ k v k)) = g x 0 + t 1 v 1) + + t k v k)).

54 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n v k) ) T r k 1) ) t k = v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v 1) + + τ k v k)) = g x 0 + t 1 v 1) + + t k v k)). Proof II): Since vectors {v k) } are A-orthogonal t g = 2 diag v 1)) T A v 1),, v k)) ) T A v k) t t g = 0 t = v 1) ) T r 0) ) v 1) ) T A v 1) ). v k) ) T r 0) ) v k) ) T A v k) ) v 1),, v k)) ) T r 0).

55 A orthogonality Craft

56 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n t k = v k) ) T r k 1) ) v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v 1) + + τ k v k)) = g x 0 + t 1 v 1) + + t k v k)).

57 A orthogonality Craft Thm: Let non-zero vectors {v k) } be A-orthogonal with v 1) = r 0) and for k = 1,, n t k = v k) ) T r k 1) ) v k) ) T A v k) ), r k 1) def = b A x k 1) residual). Then for g x) = x T A x 2 x T b and for k = 1,, n, min τ1,,τ k g x 0 + τ 1 v 1) + + τ k v k)) = g x 0 + t 1 v 1) + + t k v k)). Proof III): Since v k)) T r k 1)) = v k)) T k 1 r 0) t j A v j) = v k) ) T r 0) ) j=1 v k) ) T r k 1) ) t k = ) v k) T ) = ) A v k) v k) T ). A v k) v k)) T r 0)), so

58 A orthogonality vectors I)

59 A orthogonality vectors I) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Assume that {v k) } are non-zero. Then they are A orthogonal.

60 A orthogonality vectors I) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Assume that {v k) } are non-zero. Then they are A orthogonal. Induction Proof: For all 1 i < k, v k)) T A v i)) = r k 1)) T A v i)) = k 1 + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j)) T A v i)) r k 1)) T A v i)) + v i)) T A r k 1)) = 0.

61 A orthogonality vectors II)

62 A orthogonality vectors II) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v j)) T r k)) = 0, j = 1,, k; r j)) T r k)) = 0, j = 1,, k 1.

63 A orthogonality vectors II) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v j)) T r k)) = 0, j = 1,, k; r j)) T r k)) = 0, j = 1,, k 1. Proof: Due to optimality property of x k), for all τ and for 1 j k, g x k)) g x k) + τ v j)) = g x k)) 2τ r k)) T v j) + τ 2 v j)) T A v j)). This is true only when r k)) T v j) = 0.

64 A orthogonality vectors II) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v j)) T r k)) = 0, j = 1,, k; r j)) T r k)) = 0, j = 1,, k 1. Proof: Due to optimality property of x k), for all τ and for 1 j k, g x k)) g x k) + τ v j)) = g x k)) 2τ r k)) T v j) + τ 2 v j)) T A v j)). This is true only when r k)) T v j) = 0. Residual vector orthogonality: r j) = linear combination of v 1),, v j+1).

65 A orthogonality vectors III)

66 A orthogonality vectors III) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v k)) T r j)) = r k 1)) T r k 1)), j = 1,, k 1.

67 A orthogonality vectors III) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v k)) T r j)) = r k 1)) T r k 1)), j = 1,, k 1. Proof I): For j = k 1, v k)) T r k 1)) = = k 1 v r k 1) j) ) T A r k 1) ) T + ) v j) T ) v j) r k 1)) A v j) j=1 r k 1)) T r k 1)).

68 A orthogonality vectors III)

69 A orthogonality vectors III) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v k)) T r j)) = r k 1)) T r k 1)), j = 1,, k 1.

70 A orthogonality vectors III) Thm: Set v 1) = r 0), and for k = 2,, n k 1 v k) = r k 1) + j=1 v j) ) T A r k 1) ) v j) ) T A v j) ) v j). Let x k) = x 0) + t 1 v 1) + + t k v k) and r k) = b A x k). Then v k)) T r j)) = r k 1)) T r k 1)), j = 1,, k 1. Proof II): For j < k 1 v k)) T r j)) = = v k)) T r k 1)) + v k)) T r j) r k 1)) v k)) T r k 1)) + v k)) T = r k 1)) T r k 1)). k 1 i=j+1 t i A v i)

71 A orthogonality: A Gift from Math God

72 A orthogonality: A Gift from Math God Set v 1) = r 0), and for k = 2,, n, write Then k 1 v k) = j=0 = r k 1) v k) = k 1 j=0 r k 1) ) T r k 1) ) v k) ) T r j) ) r j) ) T r j) ) r j) r j) ) T r j) ) r j). r k 1) ) T r k 1) ) r k 2) ) T r k 2) ) k 2 = r k 1) + s k 1 v k 1), with s k 1 = rk 1) ) T r k 1) ) r k 2) ) T r k 2) ). j=0 r k 2) ) T r k 2) ) r j) ) T r j) ) r j)

73 Thm: Let {v i) } n i=1 be A-orthogonal with v1) = r 0) and for k = 1,, n v k) ) T r k 1) ) t k = ) v k) T ) = A v k) r k 1) ) T r k 1) ) v k) ) T A v k) ), x k) def = x k 1) +t k v k). Then A x n) = b in exact arithmetic.

74 Conjugate Gradient Algorithm Thm: For k = 1,, n, define, r v k) = r k 1) + s k 1 v k 1) k 1) ) T r k 1) ) with s k 1 = ) r k 2) T ), r k 2) r x k) = x k 1) + t k v k) k 1) ) T r k 1) ) with t k = ) v k) T ). A v k) Then vectors {v k) } are A-orthogonal and A x n) = b in exact arithmetic. The CG Algorithm: C is for Craft, G is for Gift.

75 Algorithm 1 Conjugate Gradient Algorithm Input: Symmetric positive definite A R n n, b R n, initial guess x 0) R n, and tolerance τ > 0. Output: Approximate solution x. Algorithm: Initialize: r 0) = b A x 0), v 0) = r 0), k = 1 while r k 1) 2 τ do t k = rk 1) ) T r k 1) ) v k) ) T A v k) ). x k) = x k 1) + t k v k) r k) = r k 1) t k A v k). s k = rk) ) T r k) ) r k 1) ) T r k 1) ). v k+1) = r k) + s k v k). k = k + 1. end while