Topics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems

Size: px
Start display at page:

Download "Topics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems"

Transcription

1 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52

2 Conjugate gradient method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps Optimal method for minimizing with A SPD J(x) = 1 2 x t Ax x t b. 2 / 52

3 Descent method Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 for n=0:itmax ( )Choose a descent direction d n α n := arg min α J(x n + αd n ) = d n, r n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, stop, end end 3 / 52

4 Descent method Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 for n=0:itmax ( )Choose a descent direction d n α n := arg min α J(x n + αd n ) = d n, r n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, stop, end end Steepest descent uses d n = r n 3 / 52

5 Descent method Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 for n=0:itmax ( )Choose a descent direction d n α n := arg min α J(x n + αd n ) = d n, r n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, stop, end end CG uses conjugate direction 3 / 52

6 Conjugate search direction Definition Conjugate means orthogonal in A-inner product [ ] b 2 0 A = 0 a 2 and J(x) = x T Ax Level curve is ellipse: J(x) = const or x 2 [ ] a cos θ x = b sin θ [ ] a sin θ Tangent vector t = dx dθ = b cos θ + y 2 a 2 b 2 = const x and t are conjugate: x, At A = x T At = a 2 b 2 sin θ cos θ + a 2 b 2 sin θ cos θ = 0 Good choice d n : vector conjugate to tangent vector 4 / 52

7 Conjugate Gradient Algorithm (294) Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 d 0 = r 0 for n=1:itmax α n 1 = d n 1, r n 1 / d n 1, Ad n 1 x n = x n 1 + α n 1 d n 1 r n = b Ax n if converged, stop, end β n = r n, r n / r n 1, r n 1 d n = r n + β n d n 1 end 5 / 52

8 Features of CG A three term recursion or a coupled two term recursion. Only three vectors are required ( cond(a) ) Worst case: O iterations per significant digit of accuracy. Common cases much faster. In exact arithmetic, CG reaches the exact solution of an N N system in N steps or less. Low FLOP count once the matrix-vector product is complete. 6 / 52

9 Non-SPD case You can have a short recursion (few vectors need to be stored) or you can have reasonably fast convergence. You cannot have both! 7 / 52

10 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); 8 / 52

11 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 8 / 52

12 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. 8 / 52

13 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. iter = 259 flag = 0 8 / 52

14 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. iter = 259 flag = 0 norm(y-x)/norm(x) ans = e-07 8 / 52

15 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. iter = 259 flag = 0 norm(y-x)/norm(x) ans = e-07 jacobi2d requires iterations 8 / 52

16 Speedy! [y,flag,relres,iter]=pcg(a,b,1.e-1,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-2,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-3,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-4,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-5,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-6,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-7,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-8,10000);iter 9 / 52

17 Speedy! [y,flag,relres,iter]=pcg(a,b,1.e-1,10000);iter iter = 3 3 [y,flag,relres,iter]=pcg(a,b,1.e-2,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-3,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-4,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-5,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-6,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-7,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-8,10000);iter iter = more 9 / 52

18 Example Example 196: Jacobi took 35 iterations to get 2-digit accuracy. whose true solution is 2u 1 u 2 = 1 u 1 +2u 2 u 3 = 2 u 2 +2u 3 u 4 = 3 u 3 +2u 4 = 4. u 1 = 4, u 2 = 7, u 3 = 8, u 4 = / 52

19 Example results Iteration α β x n / 52

20 Homework Text Exercise 296, 297, Exercise G. 296 Write a program to do CG for a matrix 297 Modify previous program to do CG for MPP xercise G CG converges in at most N iterations for an N N matrix. Show that for A = I, where I is the N N identity matrix, CG converges in a single iteration, no matter how large N is! Hint: Consider the system Ix = b, where b is an arbitrary vector of length N. Starting from an arbitrary initial condition x 0, follow the CG algorithm (by hand) and show that x 1 is the exact solution. 12 / 52

21 What if A is not SPD Might work! alpha = d n 1, d n 1 / d n 1, Ad n 1 might divide by zero. If don t divide by zero, should converge in N steps in exact arithmetic. 13 / 52

22 Example: converges but A not SPD error= error= error= error= error= error= error= error=1.059 error= error=1.1638e / 52

23 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 15 / 52

24 Algorithmic options 1. An equivalent expression for α n is α n = r n, r n d n, Ad n. 2. The expression r n+1 = b Ax n+1 is equivalent to r n+1 = r n α n Ad n in exact arithmetic. Since x n+1 = x n + α n d n, multiply by A and add b to both sides to get b Ax n+1 }{{} r n+1 = b Ax n }{{} r n α n Ad n, 16 / 52

25 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 17 / 52

26 Definitions span Let z 1,..., z m be m vectors. Then span { z 1,..., z m} is the set of all linear combinations of z 1,..., z m, i.e., the subspace span { { } z 1,..., z m} m = x = α i z i : α i R. Krylov subspace Let x 0 be given and r 0 = b Ax 0. The Krylov subspace determined by r 0 and A is i=1 X n = X n (A; r 0 ) = span{r 0, Ar 0,..., A n 1 r 0 } and the affine Krylov space determined by r 0 and A is K n = K n (A; x 0 ) = x 0 + X n = {x 0 + x : x X n }. 18 / 52

27 Important facts about CG, Proposition 302 The CG iterates x j, residuals r j and search directions d j satisfy x j x 0 + span{r 0, Ar 0,..., A j 1 r 0 }, r j r 0 + A span{r 0, Ar 0,..., A j 1 r 0 } and d j span{r 0, Ar 0,..., A j r 0 }. Note misprint in book! 19 / 52

28 Proof Proof of third result is by induction d 0 = r 0 span{r 0 } Assume d n 1 span{r 0,..., r n 1 } d n = r n + β n d n 1 span{r 0,..., r n } Proof of first and second results similar 20 / 52

29 Homework Exercise F: Complete the proof of Proposition / 52

30 First convergence theorem 304 Let A be SPD. Then the CG method satisfies the following: 1. The n th residual is globally optimal over the affine subspace K n in the A 1 -norm r n A 1 = min r r 0 +AX n r A 1 2. The n th error is globally optimal over K n in the A-norm e n A = 3. J(x n ) is the global minimum over K n min e e 0 +X n e A J(x n ) = min x Kn J(x). 4. Furthermore, the residuals are orthogonal and search directions are A-orthogonal: r n, r k = 0, for k n, d n, d k A = 0, for k n. 22 / 52

31 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A 23 / 52

32 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 23 / 52

33 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 23 / 52

34 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 23 / 52

35 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 = r n, r n /α n 1 + β n 1 d n 1, Ad n 1 23 / 52

36 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 = r n, r n /α n 1 + β n 1 d n 1, Ad n 1 = d n 1, Ad n 1 r n, r n r n 1, r n 1 + r n, r n d n 1, Ad n 1 r n 1, r n 1 23 / 52

37 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 = r n, r n /α n 1 + β n 1 d n 1, Ad n 1 = d n 1, Ad n 1 r n, r n r n 1, r n 1 + r n, r n d n 1, Ad n 1 r n 1, r n 1 = 0 23 / 52

38 Continue proof r n, r k = d n, Ad k = 0 For the case k n 2 Using r n+1 = r n α n Ad n, and d k span{r 0, Ar 0,..., A k 1 r 0 } r n+1, r k = r n, r k α n r n, Ad k = 0 0 Using d n+1 = r n+1 + β n d n, and r k+1 r k = α k Ad k, d n+1, Ad k = r n+1, Ad k + β n d n, Ad k ) = 1 α k r n+1, (r k+1 r k ) β n d n, Ad k ) = 0 24 / 52

39 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) 25 / 52

40 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 25 / 52

41 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 25 / 52

42 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 = J(x 0 ) α r 0, d α2 d 0, Ad 0 25 / 52

43 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) This is minimized when = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 = J(x 0 ) α r 0, d α2 d 0, Ad 0 r 0, d 0 + α d 0, Ad 0 = 0 25 / 52

44 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) This is minimized when = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 = J(x 0 ) α r 0, d α2 d 0, Ad 0 r 0, d 0 + α d 0, Ad 0 = 0 So α = α 0 yields a minimum over K / 52

45 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n / 52

46 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n 1. To see that x n actually minimizes J over K n, suppose ỹ K n. Write ỹ = x n + y, so y X n. Computing J(x n + y) = J(x n + x n, Ay + 1 y, Ay b, y 2 = J(x n ) + Ax n b, y + 1 y, Ay 2 = J(x n ) r n, y + 1 y, Ay 2 26 / 52

47 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n 1. To see that x n actually minimizes J over K n, suppose ỹ K n. Write ỹ = x n + y, so y X n. Computing J(x n + y) = J(x n + x n, Ay + 1 y, Ay b, y 2 = J(x n ) + Ax n b, y + 1 y, Ay 2 = J(x n ) r n, y + 1 y, Ay 2 But y X n = span{r 0,..., A n 1 r 0 } = span{r 0,..., r n 1 }, so that r, y = 0 because r n, r k = 0 for k < n. Hence J(x n + y) = J(x n ) y, Ay > J(x n ) unless y = / 52

48 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n 1. To see that x n actually minimizes J over K n, suppose ỹ K n. Write ỹ = x n + y, so y X n. Computing J(x n + y) = J(x n + x n, Ay + 1 y, Ay b, y 2 = J(x n ) + Ax n b, y + 1 y, Ay 2 = J(x n ) r n, y + 1 y, Ay 2 But y X n = span{r 0,..., A n 1 r 0 } = span{r 0,..., r n 1 }, so that r, y = 0 because r n, r k = 0 for k < n. Hence J(x n + y) = J(x n ) y, Ay > J(x n ) unless y = 0. so that J(x n ) is a minimum over all K n. 26 / 52

49 Finite termination Let A be SPD. Then in exact arithmetic CG produces the exact solution to an N N system in N steps or fewer. Proof. Since the residuals {r 0, r 1,..., r N 1 } are orthogonal they are linearly independent. Thus, r l = 0 for some l N. 27 / 52

50 Remark on development Sections 6.2 and 6.3 in the text present a way to develop the CG method. We will be skipping it because of lack of time. 28 / 52

51 Convergence rate of CG Notation: The set of real polynomials of degree n is denoted Π n. Theorem 327 Let A be SPD. The error at CG step n is bounded by ( ) x x n A min max p(x) e 0 A λ min x λ max Theorem 329 Given any ε > 0 for p n Π n and p(0)=1 n 1 2 cond(a) ln( 2 ε ) + 1 the error in the CG iterations is reduced by ε: x n x A ε x 0 x A. 29 / 52

52 Idea of proof 1. e n e 0 + X n = e n = (polynomial in A)e 0 2. e n is optimal 3. Chebychev polynomials satisfy known bounds 4. CG must be no worse than Chebychev bounds. 30 / 52

53 Polynomial bounds r n r 0 + A(span{r 0, Ar 0,..., A n 1 r 0 }) e n e 0 + span{e 0, Ae 0,..., A n 1 e 0 } e n = [I + a 1 A + a 2 A a n A n ]e 0 = p(a)e 0 e n A = min p n Π n and p(0)=1 p(a)e0 A ( ) min p(a) A e 0 A. p n Π n and p(0)=1 p(a) A = max p(x) max p(x). λ spectrum(a) λ min x λ max 31 / 52

54 Chebychev polynomials, min-max problem The Chebychev polynomials T n (x) = cos ( n cos 1 (x) ) can be scaled and translated to [a, b] (where a = λ min, b = λ max ) ( ) ( ) b + a 2x b + a p n /T n. b a b a p n (x) are known to attain Hence, min p n Π n p(0) = 1 min p n Π n p(0) = 1 max p(x) = max a x b a x b 1 where σ = 1 + a b a b max p(x) λ min x λ max T n ( b+a 2x b a ) T n ( b+a b a ) = 2 σn 1 + σ n ( ) κ 1 1 = = O( 1 κ + 1 κ κ ) 32 / 52

55 Scaled Chebychev polynomials 1 max p(x) p(x) λ min min p(x) λ max 33 / 52

56 Chebychev polynomials do_cheby.m 34 / 52

57 How many iterations? How many iterations to get 2σ n /(1 + σ n ) ɛ? ( ) 1 σ 1 2 κ σ n ɛ/2 n log σ log(ɛ/2) n( 2 1/κ) log(ɛ/2) κ n 2 log 2 ɛ / 52

58 Polynomial error e n = [I + a 1 A + a 2 A a n A n ]e 0 = p(a)e 0 Repeated eigenvalues are treated as single eigenvalue Accelerates convergence K distinct eigenvalues = convergence in K iterations. Recall Exercise G. Clusters of eigenvalues speed up convergence 36 / 52

59 Preconditioning Instead of solving Ax = b, solve MAx = Mb. M = A 1, converges in 1 iteration. M A 1, but computing Mx is fast M L L, approximate factors of A A few iterations of another method (such as Gauss-Seidel) Universe of alternatives 37 / 52

60 PCG Algorithm for solving Ax = b Given a SPD matrix A, preconditioner M, initial guess vector x 0, right side vector b, and maximum number of iterations itmax r 0 = b Ax 0 Solve Md 0 = r 0 z 0 = d 0 for n=0:itmax α n = r n, z n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 ( ) if converged, stop end Solve Mz n+1 = r n+1 β n+1 = r n+1, z n+1 / r n, z n d n+1 = z n+1 + β n+1 d n ( ) end 38 / 52

61 Example Write A = UU, where U is upper triangular Cholesky factorization Factors have fill-in (curse of dimensionality) Only keep nonzeros where A is nonzero: Incomplete Cholesky 39 / 52

62 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; 40 / 52

63 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc 40 / 52

64 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc iter0 >> Elapsed time is seconds. >> iter0 = / 52

65 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc iter0 >> Elapsed time is seconds. >> iter0 = 73 U=chol(A); U(A==0)=0; tic;[x,flag,relres,iter] = pcg(a,b,tol,maxit,u,u);toc 40 / 52

66 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc iter0 >> Elapsed time is seconds. >> iter0 = 73 U=chol(A); U(A==0)=0; tic;[x,flag,relres,iter] = pcg(a,b,tol,maxit,u,u);toc iter >> Elapsed time is seconds. >> iter = / 52

67 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 41 / 52

68 GMRES is most popular Youcef Saad and Martin H. Schultz, GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems, SIAM J. Sci. and Stat. Comput. 7, pp x n is the vector that minimizes Ax b over the Krylov space K n = span{r (0), Ar (0), A 2 r (0),..., A n 1 r (0) }. The basis for K n is formed according to a modified Gram-Schmidt orthogonalization, called an Arnoldi process. w (n) = Av (n) for k = 1,..., n w (n) = w (n) (w (n), v (k) )v (k) end v (n+1) = w (n) / w (n) 42 / 52

69 Features of GMRES A not SPD = no 3-term relation All of the basis vectors must be kept Usually restart whole thing after m steps: GMRES(m). Yousef Saad, Iterative Methods for Sparse Linear Systems, Second edition, SIAM, 2003, www-users.cs.umn.edu/%7esaad/itermethbook_2nded.pdf 43 / 52

70 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 44 / 52

71 Conjugate gradient squared (cgs) Conjugate gradient applied to A H Ax = A H b Hermitian is transpose plus complex conjugate Convergence behavior can be irregular Two (not independent) matrix-vector multiplies per iteration A H A: condition number is squared Works on general matrices, even complex 45 / 52

72 Conjugate gradient applied to normal equations (cgn) Conjugate gradient applied to A H Ax = A H b directly A H is formed Two matrix-vector multiplies per iteration 46 / 52

73 minres and symmlq Variants of the CG method for indefinite systems MINRES minimizes the residual in the 2-norm. SYMMLQ solves the projected system, but does not minimize anything. SYMMLQ keeps the residual orthogonal to all previous ones. SYMMLQ uses LQ decomposition to solve an intermediate system 47 / 52

74 Bi-conjugate gradient (bicg) r (n) = r (n 1) α n Ap (n) p (n) = r (n 1) + β n 1 p (n 1) r (n) = r (n 1) α n A T p (n) p (n) = r (n 1) + β n 1 p (n 1) α n = r (n 1),r (n 1) p (n),ap (n) β n = r (n),r (n) r (n 1),r (n 1) p (n) are conjugate-orthogonal to p (k) No minimization principle = irregular convergence Two matrix-vector products per iteration, one is with A T Works for non-symmetric matrices Can break down 48 / 52

75 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 49 / 52

76 GMRES x n is the vector that minimizes Ax b over the Krylov space K n = span{r (0), Ar (0), A 2 r (0),..., A n 1 r (0) }. Must keep all basis vectors for K n Restart to keep history size under control 50 / 52

77 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 51 / 52

78 How do I choose? Symmetric? yes Definite? yes CG no MINRES or SYMMLQ or CR no GMRES or CGN Best to use no preconditioner at first good methods will be better when preconditioned. 52 / 52

Conjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)

Conjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294) Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps

More information

Summary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method

Summary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method Summary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method Leslie Foster 11-5-2012 We will discuss the FOM (full orthogonalization method), CG,

More information

Some definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts

Some definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric

More information

ITERATIVE METHODS BASED ON KRYLOV SUBSPACES

ITERATIVE METHODS BASED ON KRYLOV SUBSPACES ITERATIVE METHODS BASED ON KRYLOV SUBSPACES LONG CHEN We shall present iterative methods for solving linear algebraic equation Au = b based on Krylov subspaces We derive conjugate gradient (CG) method

More information

Applied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic

Applied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give

More information

6.4 Krylov Subspaces and Conjugate Gradients

6.4 Krylov Subspaces and Conjugate Gradients 6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P

More information

EECS 275 Matrix Computation

EECS 275 Matrix Computation EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview

More information

9.1 Preconditioned Krylov Subspace Methods

9.1 Preconditioned Krylov Subspace Methods Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 18 Outline

More information

Lecture 11: CMSC 878R/AMSC698R. Iterative Methods An introduction. Outline. Inverse, LU decomposition, Cholesky, SVD, etc.

Lecture 11: CMSC 878R/AMSC698R. Iterative Methods An introduction. Outline. Inverse, LU decomposition, Cholesky, SVD, etc. Lecture 11: CMSC 878R/AMSC698R Iterative Methods An introduction Outline Direct Solution of Linear Systems Inverse, LU decomposition, Cholesky, SVD, etc. Iterative methods for linear systems Why? Matrix

More information

The Conjugate Gradient Method

The Conjugate Gradient Method The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large

More information

Algorithms that use the Arnoldi Basis

Algorithms that use the Arnoldi Basis AMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Arnoldi Methods Dianne P. O Leary c 2006, 2007 Algorithms that use the Arnoldi Basis Reference: Chapter 6 of Saad The Arnoldi Basis How to

More information

Iterative Methods for Solving A x = b

Iterative Methods for Solving A x = b Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http

More information

Iterative methods for Linear System

Iterative methods for Linear System Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and

More information

Notes on Some Methods for Solving Linear Systems

Notes on Some Methods for Solving Linear Systems Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms

More information

Chapter 7 Iterative Techniques in Matrix Algebra

Chapter 7 Iterative Techniques in Matrix Algebra Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition

More information

Lecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University

Lecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University Lecture 17 Methods for System of Linear Equations: Part 2 Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of

More information

Computational Linear Algebra

Computational Linear Algebra Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 3: Iterative Methods PD

More information

4.6 Iterative Solvers for Linear Systems

4.6 Iterative Solvers for Linear Systems 4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often

More information

M.A. Botchev. September 5, 2014

M.A. Botchev. September 5, 2014 Rome-Moscow school of Matrix Methods and Applied Linear Algebra 2014 A short introduction to Krylov subspaces for linear systems, matrix functions and inexact Newton methods. Plan and exercises. M.A. Botchev

More information

The amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A.

The amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A. AMSC/CMSC 661 Scientific Computing II Spring 2005 Solution of Sparse Linear Systems Part 2: Iterative methods Dianne P. O Leary c 2005 Solving Sparse Linear Systems: Iterative methods The plan: Iterative

More information

Krylov Space Solvers

Krylov Space Solvers Seminar for Applied Mathematics ETH Zurich International Symposium on Frontiers of Computational Science Nagoya, 12/13 Dec. 2005 Sparse Matrices Large sparse linear systems of equations or large sparse

More information

FEM and sparse linear system solving

FEM and sparse linear system solving FEM & sparse linear system solving, Lecture 9, Nov 19, 2017 1/36 Lecture 9, Nov 17, 2017: Krylov space methods http://people.inf.ethz.ch/arbenz/fem17 Peter Arbenz Computer Science Department, ETH Zürich

More information

4.8 Arnoldi Iteration, Krylov Subspaces and GMRES

4.8 Arnoldi Iteration, Krylov Subspaces and GMRES 48 Arnoldi Iteration, Krylov Subspaces and GMRES We start with the problem of using a similarity transformation to convert an n n matrix A to upper Hessenberg form H, ie, A = QHQ, (30) with an appropriate

More information

Preconditioning Techniques Analysis for CG Method

Preconditioning Techniques Analysis for CG Method Preconditioning Techniques Analysis for CG Method Huaguang Song Department of Computer Science University of California, Davis hso@ucdavis.edu Abstract Matrix computation issue for solve linear system

More information

CME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication.

CME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication. CME342 Parallel Methods in Numerical Analysis Matrix Computation: Iterative Methods II Outline: CG & its parallelization. Sparse Matrix-vector Multiplication. 1 Basic iterative methods: Ax = b r = b Ax

More information

A DISSERTATION. Extensions of the Conjugate Residual Method. by Tomohiro Sogabe. Presented to

A DISSERTATION. Extensions of the Conjugate Residual Method. by Tomohiro Sogabe. Presented to A DISSERTATION Extensions of the Conjugate Residual Method ( ) by Tomohiro Sogabe Presented to Department of Applied Physics, The University of Tokyo Contents 1 Introduction 1 2 Krylov subspace methods

More information

Lab 1: Iterative Methods for Solving Linear Systems

Lab 1: Iterative Methods for Solving Linear Systems Lab 1: Iterative Methods for Solving Linear Systems January 22, 2017 Introduction Many real world applications require the solution to very large and sparse linear systems where direct methods such as

More information

A short course on: Preconditioned Krylov subspace methods. Yousef Saad University of Minnesota Dept. of Computer Science and Engineering

A short course on: Preconditioned Krylov subspace methods. Yousef Saad University of Minnesota Dept. of Computer Science and Engineering A short course on: Preconditioned Krylov subspace methods Yousef Saad University of Minnesota Dept. of Computer Science and Engineering Universite du Littoral, Jan 19-3, 25 Outline Part 1 Introd., discretization

More information

Iterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009)

Iterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009) Iterative methods for Linear System of Equations Joint Advanced Student School (JASS-2009) Course #2: Numerical Simulation - from Models to Software Introduction In numerical simulation, Partial Differential

More information

DELFT UNIVERSITY OF TECHNOLOGY

DELFT UNIVERSITY OF TECHNOLOGY DELFT UNIVERSITY OF TECHNOLOGY REPORT 16-02 The Induced Dimension Reduction method applied to convection-diffusion-reaction problems R. Astudillo and M. B. van Gijzen ISSN 1389-6520 Reports of the Delft

More information

Parallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1

Parallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1 Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations

More information

KRYLOV SUBSPACE ITERATION

KRYLOV SUBSPACE ITERATION KRYLOV SUBSPACE ITERATION Presented by: Nab Raj Roshyara Master and Ph.D. Student Supervisors: Prof. Dr. Peter Benner, Department of Mathematics, TU Chemnitz and Dipl.-Geophys. Thomas Günther 1. Februar

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis

More information

Computational Linear Algebra

Computational Linear Algebra Computational Linear Algebra PD Dr. rer. nat. habil. Ralf-Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2018/19 Part 4: Iterative Methods PD

More information

SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS. Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA

SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS. Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA 1 SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA 2 OUTLINE Sparse matrix storage format Basic factorization

More information

Notes on PCG for Sparse Linear Systems

Notes on PCG for Sparse Linear Systems Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova e-mail luca.bergamaschi@unipd.it webpage www.dmsa.unipd.it/

More information

Iterative Methods and Multigrid

Iterative Methods and Multigrid Iterative Methods and Multigrid Part 3: Preconditioning 2 Eric de Sturler Preconditioning The general idea behind preconditioning is that convergence of some method for the linear system Ax = b can be

More information

Preface to the Second Edition. Preface to the First Edition

Preface to the Second Edition. Preface to the First Edition n page v Preface to the Second Edition Preface to the First Edition xiii xvii 1 Background in Linear Algebra 1 1.1 Matrices................................. 1 1.2 Square Matrices and Eigenvalues....................

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)

AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical

More information

Numerical Methods I Non-Square and Sparse Linear Systems

Numerical Methods I Non-Square and Sparse Linear Systems Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant

More information

Iterative Methods for Sparse Linear Systems

Iterative Methods for Sparse Linear Systems Iterative Methods for Sparse Linear Systems Luca Bergamaschi e-mail: berga@dmsa.unipd.it - http://www.dmsa.unipd.it/ berga Department of Mathematical Methods and Models for Scientific Applications University

More information

The Lanczos and conjugate gradient algorithms

The Lanczos and conjugate gradient algorithms The Lanczos and conjugate gradient algorithms Gérard MEURANT October, 2008 1 The Lanczos algorithm 2 The Lanczos algorithm in finite precision 3 The nonsymmetric Lanczos algorithm 4 The Golub Kahan bidiagonalization

More information

PETROV-GALERKIN METHODS

PETROV-GALERKIN METHODS Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based

More information

Iterative Linear Solvers

Iterative Linear Solvers Chapter 10 Iterative Linear Solvers In the previous two chapters, we developed strategies for solving a new class of problems involving minimizing a function f ( x) with or without constraints on x. In

More information

Solving Sparse Linear Systems: Iterative methods

Solving Sparse Linear Systems: Iterative methods Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccs Lecture Notes for Unit VII Sparse Matrix Computations Part 2: Iterative Methods Dianne P. O Leary c 2008,2010

More information

Solving Sparse Linear Systems: Iterative methods

Solving Sparse Linear Systems: Iterative methods Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 2: Iterative Methods Dianne P. O Leary

More information

Conjugate Gradient (CG) Method

Conjugate Gradient (CG) Method Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous

More information

7.2 Steepest Descent and Preconditioning

7.2 Steepest Descent and Preconditioning 7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider

More information

Numerical Methods in Matrix Computations

Numerical Methods in Matrix Computations Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices

More information

Iterative Methods for Linear Systems of Equations

Iterative Methods for Linear Systems of Equations Iterative Methods for Linear Systems of Equations Projection methods (3) ITMAN PhD-course DTU 20-10-08 till 24-10-08 Martin van Gijzen 1 Delft University of Technology Overview day 4 Bi-Lanczos method

More information

Krylov Space Methods. Nonstationary sounds good. Radu Trîmbiţaş ( Babeş-Bolyai University) Krylov Space Methods 1 / 17

Krylov Space Methods. Nonstationary sounds good. Radu Trîmbiţaş ( Babeş-Bolyai University) Krylov Space Methods 1 / 17 Krylov Space Methods Nonstationary sounds good Radu Trîmbiţaş Babeş-Bolyai University Radu Trîmbiţaş ( Babeş-Bolyai University) Krylov Space Methods 1 / 17 Introduction These methods are used both to solve

More information

Conjugate Gradient Method

Conjugate Gradient Method Conjugate Gradient Method direct and indirect methods positive definite linear systems Krylov sequence spectral analysis of Krylov sequence preconditioning Prof. S. Boyd, EE364b, Stanford University Three

More information

The Conjugate Gradient Method

The Conjugate Gradient Method The Conjugate Gradient Method The minimization problem We are given a symmetric positive definite matrix R n n and a right hand side vector b R n We want to solve the linear system Find u R n such that

More information

In order to solve the linear system KL M N when K is nonsymmetric, we can solve the equivalent system

In order to solve the linear system KL M N when K is nonsymmetric, we can solve the equivalent system !"#$% "&!#' (%)!#" *# %)%(! #! %)!#" +, %"!"#$ %*&%! $#&*! *# %)%! -. -/ 0 -. 12 "**3! * $!#%+,!2!#% 44" #% &#33 # 4"!#" "%! "5"#!!#6 -. - #% " 7% "3#!#3! - + 87&2! * $!#% 44" ) 3( $! # % %#!!#%+ 9332!

More information

Course Notes: Week 1

Course Notes: Week 1 Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues

More information

Krylov Subspace Methods that Are Based on the Minimization of the Residual

Krylov Subspace Methods that Are Based on the Minimization of the Residual Chapter 5 Krylov Subspace Methods that Are Based on the Minimization of the Residual Remark 51 Goal he goal of these methods consists in determining x k x 0 +K k r 0,A such that the corresponding Euclidean

More information

Linear Solvers. Andrew Hazel

Linear Solvers. Andrew Hazel Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction

More information

From Stationary Methods to Krylov Subspaces

From Stationary Methods to Krylov Subspaces Week 6: Wednesday, Mar 7 From Stationary Methods to Krylov Subspaces Last time, we discussed stationary methods for the iterative solution of linear systems of equations, which can generally be written

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 9 Minimizing Residual CG

More information

ITERATIVE METHODS FOR SPARSE LINEAR SYSTEMS

ITERATIVE METHODS FOR SPARSE LINEAR SYSTEMS ITERATIVE METHODS FOR SPARSE LINEAR SYSTEMS YOUSEF SAAD University of Minnesota PWS PUBLISHING COMPANY I(T)P An International Thomson Publishing Company BOSTON ALBANY BONN CINCINNATI DETROIT LONDON MADRID

More information

OUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative methods ffl Krylov subspace methods ffl Preconditioning techniques: Iterative methods ILU

OUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative methods ffl Krylov subspace methods ffl Preconditioning techniques: Iterative methods ILU Preconditioning Techniques for Solving Large Sparse Linear Systems Arnold Reusken Institut für Geometrie und Praktische Mathematik RWTH-Aachen OUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative

More information

The conjugate gradient method

The conjugate gradient method The conjugate gradient method Michael S. Floater November 1, 2011 These notes try to provide motivation and an explanation of the CG method. 1 The method of conjugate directions We want to solve the linear

More information

FEM and Sparse Linear System Solving

FEM and Sparse Linear System Solving FEM & sparse system solving, Lecture 7, Nov 3, 2017 1/46 Lecture 7, Nov 3, 2015: Introduction to Iterative Solvers: Stationary Methods http://people.inf.ethz.ch/arbenz/fem16 Peter Arbenz Computer Science

More information

CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3

CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3 CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3 Felix Kwok February 27, 2004 Written Problems 1. (Heath E3.10) Let B be an n n matrix, and assume that B is both

More information

Preconditioned inverse iteration and shift-invert Arnoldi method

Preconditioned inverse iteration and shift-invert Arnoldi method Preconditioned inverse iteration and shift-invert Arnoldi method Melina Freitag Department of Mathematical Sciences University of Bath CSC Seminar Max-Planck-Institute for Dynamics of Complex Technical

More information

IDR(s) as a projection method

IDR(s) as a projection method Delft University of Technology Faculty of Electrical Engineering, Mathematics and Computer Science Delft Institute of Applied Mathematics IDR(s) as a projection method A thesis submitted to the Delft Institute

More information

Contribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computa

Contribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computa Contribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computations ªaw University of Technology Institute of Mathematics and Computer Science Warsaw, October 7, 2006

More information

Sparse matrix methods in quantum chemistry Post-doctorale cursus quantumchemie Han-sur-Lesse, Belgium

Sparse matrix methods in quantum chemistry Post-doctorale cursus quantumchemie Han-sur-Lesse, Belgium Sparse matrix methods in quantum chemistry Post-doctorale cursus quantumchemie Han-sur-Lesse, Belgium Dr. ir. Gerrit C. Groenenboom 14 june - 18 june, 1993 Last revised: May 11, 2005 Contents 1 Introduction

More information

PROJECTED GMRES AND ITS VARIANTS

PROJECTED GMRES AND ITS VARIANTS PROJECTED GMRES AND ITS VARIANTS Reinaldo Astudillo Brígida Molina rastudillo@kuaimare.ciens.ucv.ve bmolina@kuaimare.ciens.ucv.ve Centro de Cálculo Científico y Tecnológico (CCCT), Facultad de Ciencias,

More information

Contents. Preface... xi. Introduction...

Contents. Preface... xi. Introduction... Contents Preface... xi Introduction... xv Chapter 1. Computer Architectures... 1 1.1. Different types of parallelism... 1 1.1.1. Overlap, concurrency and parallelism... 1 1.1.2. Temporal and spatial parallelism

More information

Conjugate Gradient Method

Conjugate Gradient Method Conjugate Gradient Method Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 10, 2011 T.M. Huang (NTNU) Conjugate Gradient Method October 10, 2011 1 / 36 Outline 1 Steepest

More information

Chapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices

Chapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices Chapter 7 Iterative methods for large sparse linear systems In this chapter we revisit the problem of solving linear systems of equations, but now in the context of large sparse systems. The price to pay

More information

Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods

Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 March 2015 1 / 70 Topics Introduction to Iterative Methods

More information

Lecture 9: Krylov Subspace Methods. 2 Derivation of the Conjugate Gradient Algorithm

Lecture 9: Krylov Subspace Methods. 2 Derivation of the Conjugate Gradient Algorithm CS 622 Data-Sparse Matrix Computations September 19, 217 Lecture 9: Krylov Subspace Methods Lecturer: Anil Damle Scribes: David Eriksson, Marc Aurele Gilles, Ariah Klages-Mundt, Sophia Novitzky 1 Introduction

More information

CONVERGENCE BOUNDS FOR PRECONDITIONED GMRES USING ELEMENT-BY-ELEMENT ESTIMATES OF THE FIELD OF VALUES

CONVERGENCE BOUNDS FOR PRECONDITIONED GMRES USING ELEMENT-BY-ELEMENT ESTIMATES OF THE FIELD OF VALUES European Conference on Computational Fluid Dynamics ECCOMAS CFD 2006 P. Wesseling, E. Oñate and J. Périaux (Eds) c TU Delft, The Netherlands, 2006 CONVERGENCE BOUNDS FOR PRECONDITIONED GMRES USING ELEMENT-BY-ELEMENT

More information

Bindel, Fall 2016 Matrix Computations (CS 6210) Notes for

Bindel, Fall 2016 Matrix Computations (CS 6210) Notes for 1 Iteration basics Notes for 2016-11-07 An iterative solver for Ax = b is produces a sequence of approximations x (k) x. We always stop after finitely many steps, based on some convergence criterion, e.g.

More information

1 Extrapolation: A Hint of Things to Come

1 Extrapolation: A Hint of Things to Come Notes for 2017-03-24 1 Extrapolation: A Hint of Things to Come Stationary iterations are simple. Methods like Jacobi or Gauss-Seidel are easy to program, and it s (relatively) easy to analyze their convergence.

More information

Charles University Faculty of Mathematics and Physics DOCTORAL THESIS. Krylov subspace approximations in linear algebraic problems

Charles University Faculty of Mathematics and Physics DOCTORAL THESIS. Krylov subspace approximations in linear algebraic problems Charles University Faculty of Mathematics and Physics DOCTORAL THESIS Iveta Hnětynková Krylov subspace approximations in linear algebraic problems Department of Numerical Mathematics Supervisor: Doc. RNDr.

More information

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012

Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.

More information

Math 504 (Fall 2011) 1. (*) Consider the matrices

Math 504 (Fall 2011) 1. (*) Consider the matrices Math 504 (Fall 2011) Instructor: Emre Mengi Study Guide for Weeks 11-14 This homework concerns the following topics. Basic definitions and facts about eigenvalues and eigenvectors (Trefethen&Bau, Lecture

More information

Lecture # 20 The Preconditioned Conjugate Gradient Method

Lecture # 20 The Preconditioned Conjugate Gradient Method Lecture # 20 The Preconditioned Conjugate Gradient Method We wish to solve Ax = b (1) A R n n is symmetric and positive definite (SPD). We then of n are being VERY LARGE, say, n = 10 6 or n = 10 7. Usually,

More information

Research Article Some Generalizations and Modifications of Iterative Methods for Solving Large Sparse Symmetric Indefinite Linear Systems

Research Article Some Generalizations and Modifications of Iterative Methods for Solving Large Sparse Symmetric Indefinite Linear Systems Abstract and Applied Analysis Article ID 237808 pages http://dxdoiorg/055/204/237808 Research Article Some Generalizations and Modifications of Iterative Methods for Solving Large Sparse Symmetric Indefinite

More information

The Conjugate Gradient Method

The Conjugate Gradient Method The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)

More information

RESIDUAL SMOOTHING AND PEAK/PLATEAU BEHAVIOR IN KRYLOV SUBSPACE METHODS

RESIDUAL SMOOTHING AND PEAK/PLATEAU BEHAVIOR IN KRYLOV SUBSPACE METHODS RESIDUAL SMOOTHING AND PEAK/PLATEAU BEHAVIOR IN KRYLOV SUBSPACE METHODS HOMER F. WALKER Abstract. Recent results on residual smoothing are reviewed, and it is observed that certain of these are equivalent

More information

ON ORTHOGONAL REDUCTION TO HESSENBERG FORM WITH SMALL BANDWIDTH

ON ORTHOGONAL REDUCTION TO HESSENBERG FORM WITH SMALL BANDWIDTH ON ORTHOGONAL REDUCTION TO HESSENBERG FORM WITH SMALL BANDWIDTH V. FABER, J. LIESEN, AND P. TICHÝ Abstract. Numerous algorithms in numerical linear algebra are based on the reduction of a given matrix

More information

Introduction. Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods. Example: First Order Richardson. Strategy

Introduction. Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods. Example: First Order Richardson. Strategy Introduction Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 Solve system Ax = b by repeatedly computing

More information

Conjugate Gradient Method

Conjugate Gradient Method Conjugate Gradient Method Hung M Phan UMass Lowell April 13, 2017 Throughout, A R n n is symmetric and positive definite, and b R n 1 Steepest Descent Method We present the steepest descent method for

More information

STEEPEST DESCENT AND CONJUGATE GRADIENT METHODS WITH VARIABLE PRECONDITIONING

STEEPEST DESCENT AND CONJUGATE GRADIENT METHODS WITH VARIABLE PRECONDITIONING SIAM J. MATRIX ANAL. APPL. Vol.?, No.?, pp.?? c 2007 Society for Industrial and Applied Mathematics STEEPEST DESCENT AND CONJUGATE GRADIENT METHODS WITH VARIABLE PRECONDITIONING ANDREW V. KNYAZEV AND ILYA

More information

Key words. linear equations, polynomial preconditioning, nonsymmetric Lanczos, BiCGStab, IDR

Key words. linear equations, polynomial preconditioning, nonsymmetric Lanczos, BiCGStab, IDR POLYNOMIAL PRECONDITIONED BICGSTAB AND IDR JENNIFER A. LOE AND RONALD B. MORGAN Abstract. Polynomial preconditioning is applied to the nonsymmetric Lanczos methods BiCGStab and IDR for solving large nonsymmetric

More information

Numerical Methods for Solving Large Scale Eigenvalue Problems

Numerical Methods for Solving Large Scale Eigenvalue Problems Peter Arbenz Computer Science Department, ETH Zürich E-mail: arbenz@inf.ethz.ch arge scale eigenvalue problems, Lecture 2, February 28, 2018 1/46 Numerical Methods for Solving Large Scale Eigenvalue Problems

More information

ITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD

ITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD ITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD Heinrich Voss voss@tu-harburg.de Hamburg University of Technology Institute of Numerical Simulation

More information

The Conjugate Gradient Method

The Conjugate Gradient Method The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011 Lecture Objectives describe when CG can be used to solve Ax

More information

Solving Ax = b, an overview. Program

Solving Ax = b, an overview. Program Numerical Linear Algebra Improving iterative solvers: preconditioning, deflation, numerical software and parallelisation Gerard Sleijpen and Martin van Gijzen November 29, 27 Solving Ax = b, an overview

More information

AMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Krylov Minimization and Projection (KMP) Dianne P. O Leary c 2006, 2007.

AMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Krylov Minimization and Projection (KMP) Dianne P. O Leary c 2006, 2007. AMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Krylov Minimization and Projection (KMP) Dianne P. O Leary c 2006, 2007 This unit: So far: A survey of iterative methods for solving linear

More information

AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 24: Preconditioning and Multigrid Solver Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 5 Preconditioning Motivation:

More information

Iterative Solvers in the Finite Element Solution of Transient Heat Conduction

Iterative Solvers in the Finite Element Solution of Transient Heat Conduction Iterative Solvers in the Finite Element Solution of Transient Heat Conduction Mile R. Vuji~i} PhD student Steve G.R. Brown Senior Lecturer Materials Research Centre School of Engineering University of

More information

On the influence of eigenvalues on Bi-CG residual norms

On the influence of eigenvalues on Bi-CG residual norms On the influence of eigenvalues on Bi-CG residual norms Jurjen Duintjer Tebbens Institute of Computer Science Academy of Sciences of the Czech Republic duintjertebbens@cs.cas.cz Gérard Meurant 30, rue

More information

Conjugate Gradients I: Setup

Conjugate Gradients I: Setup Conjugate Gradients I: Setup CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Conjugate Gradients I: Setup 1 / 22 Time for Gaussian Elimination

More information

APPLIED NUMERICAL LINEAR ALGEBRA

APPLIED NUMERICAL LINEAR ALGEBRA APPLIED NUMERICAL LINEAR ALGEBRA James W. Demmel University of California Berkeley, California Society for Industrial and Applied Mathematics Philadelphia Contents Preface 1 Introduction 1 1.1 Basic Notation

More information