Topics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems
|
|
- Daniela Norris
- 5 years ago
- Views:
Transcription
1 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52
2 Conjugate gradient method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps Optimal method for minimizing with A SPD J(x) = 1 2 x t Ax x t b. 2 / 52
3 Descent method Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 for n=0:itmax ( )Choose a descent direction d n α n := arg min α J(x n + αd n ) = d n, r n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, stop, end end 3 / 52
4 Descent method Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 for n=0:itmax ( )Choose a descent direction d n α n := arg min α J(x n + αd n ) = d n, r n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, stop, end end Steepest descent uses d n = r n 3 / 52
5 Descent method Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 for n=0:itmax ( )Choose a descent direction d n α n := arg min α J(x n + αd n ) = d n, r n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 if converged, stop, end end CG uses conjugate direction 3 / 52
6 Conjugate search direction Definition Conjugate means orthogonal in A-inner product [ ] b 2 0 A = 0 a 2 and J(x) = x T Ax Level curve is ellipse: J(x) = const or x 2 [ ] a cos θ x = b sin θ [ ] a sin θ Tangent vector t = dx dθ = b cos θ + y 2 a 2 b 2 = const x and t are conjugate: x, At A = x T At = a 2 b 2 sin θ cos θ + a 2 b 2 sin θ cos θ = 0 Good choice d n : vector conjugate to tangent vector 4 / 52
7 Conjugate Gradient Algorithm (294) Given an SPD A, x 0 and a maximum number of iterations itmax r 0 = b Ax 0 d 0 = r 0 for n=1:itmax α n 1 = d n 1, r n 1 / d n 1, Ad n 1 x n = x n 1 + α n 1 d n 1 r n = b Ax n if converged, stop, end β n = r n, r n / r n 1, r n 1 d n = r n + β n d n 1 end 5 / 52
8 Features of CG A three term recursion or a coupled two term recursion. Only three vectors are required ( cond(a) ) Worst case: O iterations per significant digit of accuracy. Common cases much faster. In exact arithmetic, CG reaches the exact solution of an N N system in N steps or less. Low FLOP count once the matrix-vector product is complete. 6 / 52
9 Non-SPD case You can have a short recursion (few vectors need to be stored) or you can have reasonably fast convergence. You cannot have both! 7 / 52
10 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); 8 / 52
11 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 8 / 52
12 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. 8 / 52
13 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. iter = 259 flag = 0 8 / 52
14 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. iter = 259 flag = 0 norm(y-x)/norm(x) ans = e-07 8 / 52
15 Example A=gallery( poisson,100); x=rand(10000,1); b=a*x; [y,flag,relres,iter]=pcg(a,b,1.e-8,10000,[],[],x); flag = 0 iter = 0 [y,flag,relres,iter]=pcg(a,b,1.e-8,10000); Default: initial guess is zero. iter = 259 flag = 0 norm(y-x)/norm(x) ans = e-07 jacobi2d requires iterations 8 / 52
16 Speedy! [y,flag,relres,iter]=pcg(a,b,1.e-1,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-2,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-3,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-4,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-5,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-6,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-7,10000);iter [y,flag,relres,iter]=pcg(a,b,1.e-8,10000);iter 9 / 52
17 Speedy! [y,flag,relres,iter]=pcg(a,b,1.e-1,10000);iter iter = 3 3 [y,flag,relres,iter]=pcg(a,b,1.e-2,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-3,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-4,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-5,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-6,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-7,10000);iter iter = more [y,flag,relres,iter]=pcg(a,b,1.e-8,10000);iter iter = more 9 / 52
18 Example Example 196: Jacobi took 35 iterations to get 2-digit accuracy. whose true solution is 2u 1 u 2 = 1 u 1 +2u 2 u 3 = 2 u 2 +2u 3 u 4 = 3 u 3 +2u 4 = 4. u 1 = 4, u 2 = 7, u 3 = 8, u 4 = / 52
19 Example results Iteration α β x n / 52
20 Homework Text Exercise 296, 297, Exercise G. 296 Write a program to do CG for a matrix 297 Modify previous program to do CG for MPP xercise G CG converges in at most N iterations for an N N matrix. Show that for A = I, where I is the N N identity matrix, CG converges in a single iteration, no matter how large N is! Hint: Consider the system Ix = b, where b is an arbitrary vector of length N. Starting from an arbitrary initial condition x 0, follow the CG algorithm (by hand) and show that x 1 is the exact solution. 12 / 52
21 What if A is not SPD Might work! alpha = d n 1, d n 1 / d n 1, Ad n 1 might divide by zero. If don t divide by zero, should converge in N steps in exact arithmetic. 13 / 52
22 Example: converges but A not SPD error= error= error= error= error= error= error= error=1.059 error= error=1.1638e / 52
23 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 15 / 52
24 Algorithmic options 1. An equivalent expression for α n is α n = r n, r n d n, Ad n. 2. The expression r n+1 = b Ax n+1 is equivalent to r n+1 = r n α n Ad n in exact arithmetic. Since x n+1 = x n + α n d n, multiply by A and add b to both sides to get b Ax n+1 }{{} r n+1 = b Ax n }{{} r n α n Ad n, 16 / 52
25 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 17 / 52
26 Definitions span Let z 1,..., z m be m vectors. Then span { z 1,..., z m} is the set of all linear combinations of z 1,..., z m, i.e., the subspace span { { } z 1,..., z m} m = x = α i z i : α i R. Krylov subspace Let x 0 be given and r 0 = b Ax 0. The Krylov subspace determined by r 0 and A is i=1 X n = X n (A; r 0 ) = span{r 0, Ar 0,..., A n 1 r 0 } and the affine Krylov space determined by r 0 and A is K n = K n (A; x 0 ) = x 0 + X n = {x 0 + x : x X n }. 18 / 52
27 Important facts about CG, Proposition 302 The CG iterates x j, residuals r j and search directions d j satisfy x j x 0 + span{r 0, Ar 0,..., A j 1 r 0 }, r j r 0 + A span{r 0, Ar 0,..., A j 1 r 0 } and d j span{r 0, Ar 0,..., A j r 0 }. Note misprint in book! 19 / 52
28 Proof Proof of third result is by induction d 0 = r 0 span{r 0 } Assume d n 1 span{r 0,..., r n 1 } d n = r n + β n d n 1 span{r 0,..., r n } Proof of first and second results similar 20 / 52
29 Homework Exercise F: Complete the proof of Proposition / 52
30 First convergence theorem 304 Let A be SPD. Then the CG method satisfies the following: 1. The n th residual is globally optimal over the affine subspace K n in the A 1 -norm r n A 1 = min r r 0 +AX n r A 1 2. The n th error is globally optimal over K n in the A-norm e n A = 3. J(x n ) is the global minimum over K n min e e 0 +X n e A J(x n ) = min x Kn J(x). 4. Furthermore, the residuals are orthogonal and search directions are A-orthogonal: r n, r k = 0, for k n, d n, d k A = 0, for k n. 22 / 52
31 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A 23 / 52
32 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 23 / 52
33 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 23 / 52
34 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 23 / 52
35 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 = r n, r n /α n 1 + β n 1 d n 1, Ad n 1 23 / 52
36 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 = r n, r n /α n 1 + β n 1 d n 1, Ad n 1 = d n 1, Ad n 1 r n, r n r n 1, r n 1 + r n, r n d n 1, Ad n 1 r n 1, r n 1 23 / 52
37 Prove r n, r k = d n, Ad k = 0 for k < n By induction: Vacuously true for n = 0 Assue true for n 1 First, prove r n, r n 1 and d n, d n 1 A Using r n = r n 1 α n 1 Ad n 1 and α n 1 = r n 1, r n 1 / d n 1, Ad n 1 r n, r n 1 = r n 1, r n 1 α n 1 Ad n 1, d n 1 = 0 Using d n = r n + β n 1 d n 1, r n r n 1 = α n 1 Ad n 1, and β n 1 = r n, r n / r n 1, r n 1 d n, Ad n 1 = r n, Ad n 1 + β n 1 d n 1, Ad n 1 = r n, r n r n 1 /α n 1 + β n 1 d n 1, Ad n 1 = r n, r n /α n 1 + β n 1 d n 1, Ad n 1 = d n 1, Ad n 1 r n, r n r n 1, r n 1 + r n, r n d n 1, Ad n 1 r n 1, r n 1 = 0 23 / 52
38 Continue proof r n, r k = d n, Ad k = 0 For the case k n 2 Using r n+1 = r n α n Ad n, and d k span{r 0, Ar 0,..., A k 1 r 0 } r n+1, r k = r n, r k α n r n, Ad k = 0 0 Using d n+1 = r n+1 + β n d n, and r k+1 r k = α k Ad k, d n+1, Ad k = r n+1, Ad k + β n d n, Ad k ) = 1 α k r n+1, (r k+1 r k ) β n d n, Ad k ) = 0 24 / 52
39 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) 25 / 52
40 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 25 / 52
41 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 25 / 52
42 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 = J(x 0 ) α r 0, d α2 d 0, Ad 0 25 / 52
43 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) This is minimized when = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 = J(x 0 ) α r 0, d α2 d 0, Ad 0 r 0, d 0 + α d 0, Ad 0 = 0 25 / 52
44 Prove J(x n ) = min x Kn J(x), n = 1 Induction proof: Assume n = 1, x 1 = x 0 + α 0 d 0 K 1 For arbitrary α J(x 0 + αd 0 ) = 1 2 (x 0 + αd 0 ), A(x 0 + αd 0 ) b, (x 0 + αd 0 ) This is minimized when = 1 2 x 0, Ax 0 + α x 0, Ad α2 d 0, Ad 0 b, x 0 α b, d 0 = J(x 0 ) + α Ax 0 b, d α2 d 0, Ad 0 = J(x 0 ) α r 0, d α2 d 0, Ad 0 r 0, d 0 + α d 0, Ad 0 = 0 So α = α 0 yields a minimum over K / 52
45 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n / 52
46 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n 1. To see that x n actually minimizes J over K n, suppose ỹ K n. Write ỹ = x n + y, so y X n. Computing J(x n + y) = J(x n + x n, Ay + 1 y, Ay b, y 2 = J(x n ) + Ax n b, y + 1 y, Ay 2 = J(x n ) r n, y + 1 y, Ay 2 26 / 52
47 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n 1. To see that x n actually minimizes J over K n, suppose ỹ K n. Write ỹ = x n + y, so y X n. Computing J(x n + y) = J(x n + x n, Ay + 1 y, Ay b, y 2 = J(x n ) + Ax n b, y + 1 y, Ay 2 = J(x n ) r n, y + 1 y, Ay 2 But y X n = span{r 0,..., A n 1 r 0 } = span{r 0,..., r n 1 }, so that r, y = 0 because r n, r k = 0 for k < n. Hence J(x n + y) = J(x n ) y, Ay > J(x n ) unless y = / 52
48 Prove J(x n ) = min x Kn J(x), induction step For x = x n 1 + αd n 1, previous calculation yields α n = r 0, d 0 /α d 0, Ad 0 so x n minimizes J over x n 1 + αd n 1. To see that x n actually minimizes J over K n, suppose ỹ K n. Write ỹ = x n + y, so y X n. Computing J(x n + y) = J(x n + x n, Ay + 1 y, Ay b, y 2 = J(x n ) + Ax n b, y + 1 y, Ay 2 = J(x n ) r n, y + 1 y, Ay 2 But y X n = span{r 0,..., A n 1 r 0 } = span{r 0,..., r n 1 }, so that r, y = 0 because r n, r k = 0 for k < n. Hence J(x n + y) = J(x n ) y, Ay > J(x n ) unless y = 0. so that J(x n ) is a minimum over all K n. 26 / 52
49 Finite termination Let A be SPD. Then in exact arithmetic CG produces the exact solution to an N N system in N steps or fewer. Proof. Since the residuals {r 0, r 1,..., r N 1 } are orthogonal they are linearly independent. Thus, r l = 0 for some l N. 27 / 52
50 Remark on development Sections 6.2 and 6.3 in the text present a way to develop the CG method. We will be skipping it because of lack of time. 28 / 52
51 Convergence rate of CG Notation: The set of real polynomials of degree n is denoted Π n. Theorem 327 Let A be SPD. The error at CG step n is bounded by ( ) x x n A min max p(x) e 0 A λ min x λ max Theorem 329 Given any ε > 0 for p n Π n and p(0)=1 n 1 2 cond(a) ln( 2 ε ) + 1 the error in the CG iterations is reduced by ε: x n x A ε x 0 x A. 29 / 52
52 Idea of proof 1. e n e 0 + X n = e n = (polynomial in A)e 0 2. e n is optimal 3. Chebychev polynomials satisfy known bounds 4. CG must be no worse than Chebychev bounds. 30 / 52
53 Polynomial bounds r n r 0 + A(span{r 0, Ar 0,..., A n 1 r 0 }) e n e 0 + span{e 0, Ae 0,..., A n 1 e 0 } e n = [I + a 1 A + a 2 A a n A n ]e 0 = p(a)e 0 e n A = min p n Π n and p(0)=1 p(a)e0 A ( ) min p(a) A e 0 A. p n Π n and p(0)=1 p(a) A = max p(x) max p(x). λ spectrum(a) λ min x λ max 31 / 52
54 Chebychev polynomials, min-max problem The Chebychev polynomials T n (x) = cos ( n cos 1 (x) ) can be scaled and translated to [a, b] (where a = λ min, b = λ max ) ( ) ( ) b + a 2x b + a p n /T n. b a b a p n (x) are known to attain Hence, min p n Π n p(0) = 1 min p n Π n p(0) = 1 max p(x) = max a x b a x b 1 where σ = 1 + a b a b max p(x) λ min x λ max T n ( b+a 2x b a ) T n ( b+a b a ) = 2 σn 1 + σ n ( ) κ 1 1 = = O( 1 κ + 1 κ κ ) 32 / 52
55 Scaled Chebychev polynomials 1 max p(x) p(x) λ min min p(x) λ max 33 / 52
56 Chebychev polynomials do_cheby.m 34 / 52
57 How many iterations? How many iterations to get 2σ n /(1 + σ n ) ɛ? ( ) 1 σ 1 2 κ σ n ɛ/2 n log σ log(ɛ/2) n( 2 1/κ) log(ɛ/2) κ n 2 log 2 ɛ / 52
58 Polynomial error e n = [I + a 1 A + a 2 A a n A n ]e 0 = p(a)e 0 Repeated eigenvalues are treated as single eigenvalue Accelerates convergence K distinct eigenvalues = convergence in K iterations. Recall Exercise G. Clusters of eigenvalues speed up convergence 36 / 52
59 Preconditioning Instead of solving Ax = b, solve MAx = Mb. M = A 1, converges in 1 iteration. M A 1, but computing Mx is fast M L L, approximate factors of A A few iterations of another method (such as Gauss-Seidel) Universe of alternatives 37 / 52
60 PCG Algorithm for solving Ax = b Given a SPD matrix A, preconditioner M, initial guess vector x 0, right side vector b, and maximum number of iterations itmax r 0 = b Ax 0 Solve Md 0 = r 0 z 0 = d 0 for n=0:itmax α n = r n, z n / d n, Ad n x n+1 = x n + α n d n r n+1 = b Ax n+1 ( ) if converged, stop end Solve Mz n+1 = r n+1 β n+1 = r n+1, z n+1 / r n, z n d n+1 = z n+1 + β n+1 d n ( ) end 38 / 52
61 Example Write A = UU, where U is upper triangular Cholesky factorization Factors have fill-in (curse of dimensionality) Only keep nonzeros where A is nonzero: Incomplete Cholesky 39 / 52
62 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; 40 / 52
63 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc 40 / 52
64 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc iter0 >> Elapsed time is seconds. >> iter0 = / 52
65 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc iter0 >> Elapsed time is seconds. >> iter0 = 73 U=chol(A); U(A==0)=0; tic;[x,flag,relres,iter] = pcg(a,b,tol,maxit,u,u);toc 40 / 52
66 Example N=50; A=gallery( poisson,n); xact=sin(1:n^2) ; b=a*xact; tol=1.e-6; maxit=n^2; tic;[x,flag,relres,iter0] = pcg(a,b,tol,maxit);toc iter0 >> Elapsed time is seconds. >> iter0 = 73 U=chol(A); U(A==0)=0; tic;[x,flag,relres,iter] = pcg(a,b,tol,maxit,u,u);toc iter >> Elapsed time is seconds. >> iter = / 52
67 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 41 / 52
68 GMRES is most popular Youcef Saad and Martin H. Schultz, GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems, SIAM J. Sci. and Stat. Comput. 7, pp x n is the vector that minimizes Ax b over the Krylov space K n = span{r (0), Ar (0), A 2 r (0),..., A n 1 r (0) }. The basis for K n is formed according to a modified Gram-Schmidt orthogonalization, called an Arnoldi process. w (n) = Av (n) for k = 1,..., n w (n) = w (n) (w (n), v (k) )v (k) end v (n+1) = w (n) / w (n) 42 / 52
69 Features of GMRES A not SPD = no 3-term relation All of the basis vectors must be kept Usually restart whole thing after m steps: GMRES(m). Yousef Saad, Iterative Methods for Sparse Linear Systems, Second edition, SIAM, 2003, www-users.cs.umn.edu/%7esaad/itermethbook_2nded.pdf 43 / 52
70 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 44 / 52
71 Conjugate gradient squared (cgs) Conjugate gradient applied to A H Ax = A H b Hermitian is transpose plus complex conjugate Convergence behavior can be irregular Two (not independent) matrix-vector multiplies per iteration A H A: condition number is squared Works on general matrices, even complex 45 / 52
72 Conjugate gradient applied to normal equations (cgn) Conjugate gradient applied to A H Ax = A H b directly A H is formed Two matrix-vector multiplies per iteration 46 / 52
73 minres and symmlq Variants of the CG method for indefinite systems MINRES minimizes the residual in the 2-norm. SYMMLQ solves the projected system, but does not minimize anything. SYMMLQ keeps the residual orthogonal to all previous ones. SYMMLQ uses LQ decomposition to solve an intermediate system 47 / 52
74 Bi-conjugate gradient (bicg) r (n) = r (n 1) α n Ap (n) p (n) = r (n 1) + β n 1 p (n 1) r (n) = r (n 1) α n A T p (n) p (n) = r (n 1) + β n 1 p (n 1) α n = r (n 1),r (n 1) p (n),ap (n) β n = r (n),r (n) r (n 1),r (n 1) p (n) are conjugate-orthogonal to p (k) No minimization principle = irregular convergence Two matrix-vector products per iteration, one is with A T Works for non-symmetric matrices Can break down 48 / 52
75 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 49 / 52
76 GMRES x n is the vector that minimizes Ax b over the Krylov space K n = span{r (0), Ar (0), A 2 r (0),..., A n 1 r (0) }. Must keep all basis vectors for K n Restart to keep history size under control 50 / 52
77 Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 51 / 52
78 How do I choose? Symmetric? yes Definite? yes CG no MINRES or SYMMLQ or CR no GMRES or CGN Best to use no preconditioner at first good methods will be better when preconditioned. 52 / 52
Conjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)
Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps
More informationSummary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method
Summary of Iterative Methods for Non-symmetric Linear Equations That Are Related to the Conjugate Gradient (CG) Method Leslie Foster 11-5-2012 We will discuss the FOM (full orthogonalization method), CG,
More informationSome definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts
Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric
More informationITERATIVE METHODS BASED ON KRYLOV SUBSPACES
ITERATIVE METHODS BASED ON KRYLOV SUBSPACES LONG CHEN We shall present iterative methods for solving linear algebraic equation Au = b based on Krylov subspaces We derive conjugate gradient (CG) method
More informationApplied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 18 Outline
More informationLecture 11: CMSC 878R/AMSC698R. Iterative Methods An introduction. Outline. Inverse, LU decomposition, Cholesky, SVD, etc.
Lecture 11: CMSC 878R/AMSC698R Iterative Methods An introduction Outline Direct Solution of Linear Systems Inverse, LU decomposition, Cholesky, SVD, etc. Iterative methods for linear systems Why? Matrix
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationAlgorithms that use the Arnoldi Basis
AMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Arnoldi Methods Dianne P. O Leary c 2006, 2007 Algorithms that use the Arnoldi Basis Reference: Chapter 6 of Saad The Arnoldi Basis How to
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationIterative methods for Linear System
Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and
More informationNotes on Some Methods for Solving Linear Systems
Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationLecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University
Lecture 17 Methods for System of Linear Equations: Part 2 Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 3: Iterative Methods PD
More information4.6 Iterative Solvers for Linear Systems
4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often
More informationM.A. Botchev. September 5, 2014
Rome-Moscow school of Matrix Methods and Applied Linear Algebra 2014 A short introduction to Krylov subspaces for linear systems, matrix functions and inexact Newton methods. Plan and exercises. M.A. Botchev
More informationThe amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A.
AMSC/CMSC 661 Scientific Computing II Spring 2005 Solution of Sparse Linear Systems Part 2: Iterative methods Dianne P. O Leary c 2005 Solving Sparse Linear Systems: Iterative methods The plan: Iterative
More informationKrylov Space Solvers
Seminar for Applied Mathematics ETH Zurich International Symposium on Frontiers of Computational Science Nagoya, 12/13 Dec. 2005 Sparse Matrices Large sparse linear systems of equations or large sparse
More informationFEM and sparse linear system solving
FEM & sparse linear system solving, Lecture 9, Nov 19, 2017 1/36 Lecture 9, Nov 17, 2017: Krylov space methods http://people.inf.ethz.ch/arbenz/fem17 Peter Arbenz Computer Science Department, ETH Zürich
More information4.8 Arnoldi Iteration, Krylov Subspaces and GMRES
48 Arnoldi Iteration, Krylov Subspaces and GMRES We start with the problem of using a similarity transformation to convert an n n matrix A to upper Hessenberg form H, ie, A = QHQ, (30) with an appropriate
More informationPreconditioning Techniques Analysis for CG Method
Preconditioning Techniques Analysis for CG Method Huaguang Song Department of Computer Science University of California, Davis hso@ucdavis.edu Abstract Matrix computation issue for solve linear system
More informationCME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication.
CME342 Parallel Methods in Numerical Analysis Matrix Computation: Iterative Methods II Outline: CG & its parallelization. Sparse Matrix-vector Multiplication. 1 Basic iterative methods: Ax = b r = b Ax
More informationA DISSERTATION. Extensions of the Conjugate Residual Method. by Tomohiro Sogabe. Presented to
A DISSERTATION Extensions of the Conjugate Residual Method ( ) by Tomohiro Sogabe Presented to Department of Applied Physics, The University of Tokyo Contents 1 Introduction 1 2 Krylov subspace methods
More informationLab 1: Iterative Methods for Solving Linear Systems
Lab 1: Iterative Methods for Solving Linear Systems January 22, 2017 Introduction Many real world applications require the solution to very large and sparse linear systems where direct methods such as
More informationA short course on: Preconditioned Krylov subspace methods. Yousef Saad University of Minnesota Dept. of Computer Science and Engineering
A short course on: Preconditioned Krylov subspace methods Yousef Saad University of Minnesota Dept. of Computer Science and Engineering Universite du Littoral, Jan 19-3, 25 Outline Part 1 Introd., discretization
More informationIterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009)
Iterative methods for Linear System of Equations Joint Advanced Student School (JASS-2009) Course #2: Numerical Simulation - from Models to Software Introduction In numerical simulation, Partial Differential
More informationDELFT UNIVERSITY OF TECHNOLOGY
DELFT UNIVERSITY OF TECHNOLOGY REPORT 16-02 The Induced Dimension Reduction method applied to convection-diffusion-reaction problems R. Astudillo and M. B. van Gijzen ISSN 1389-6520 Reports of the Delft
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationKRYLOV SUBSPACE ITERATION
KRYLOV SUBSPACE ITERATION Presented by: Nab Raj Roshyara Master and Ph.D. Student Supervisors: Prof. Dr. Peter Benner, Department of Mathematics, TU Chemnitz and Dipl.-Geophys. Thomas Günther 1. Februar
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf-Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2018/19 Part 4: Iterative Methods PD
More informationSOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS. Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA
1 SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA 2 OUTLINE Sparse matrix storage format Basic factorization
More informationNotes on PCG for Sparse Linear Systems
Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova e-mail luca.bergamaschi@unipd.it webpage www.dmsa.unipd.it/
More informationIterative Methods and Multigrid
Iterative Methods and Multigrid Part 3: Preconditioning 2 Eric de Sturler Preconditioning The general idea behind preconditioning is that convergence of some method for the linear system Ax = b can be
More informationPreface to the Second Edition. Preface to the First Edition
n page v Preface to the Second Edition Preface to the First Edition xiii xvii 1 Background in Linear Algebra 1 1.1 Matrices................................. 1 1.2 Square Matrices and Eigenvalues....................
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationIterative Methods for Sparse Linear Systems
Iterative Methods for Sparse Linear Systems Luca Bergamaschi e-mail: berga@dmsa.unipd.it - http://www.dmsa.unipd.it/ berga Department of Mathematical Methods and Models for Scientific Applications University
More informationThe Lanczos and conjugate gradient algorithms
The Lanczos and conjugate gradient algorithms Gérard MEURANT October, 2008 1 The Lanczos algorithm 2 The Lanczos algorithm in finite precision 3 The nonsymmetric Lanczos algorithm 4 The Golub Kahan bidiagonalization
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationIterative Linear Solvers
Chapter 10 Iterative Linear Solvers In the previous two chapters, we developed strategies for solving a new class of problems involving minimizing a function f ( x) with or without constraints on x. In
More informationSolving Sparse Linear Systems: Iterative methods
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccs Lecture Notes for Unit VII Sparse Matrix Computations Part 2: Iterative Methods Dianne P. O Leary c 2008,2010
More informationSolving Sparse Linear Systems: Iterative methods
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 2: Iterative Methods Dianne P. O Leary
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More information7.2 Steepest Descent and Preconditioning
7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider
More informationNumerical Methods in Matrix Computations
Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices
More informationIterative Methods for Linear Systems of Equations
Iterative Methods for Linear Systems of Equations Projection methods (3) ITMAN PhD-course DTU 20-10-08 till 24-10-08 Martin van Gijzen 1 Delft University of Technology Overview day 4 Bi-Lanczos method
More informationKrylov Space Methods. Nonstationary sounds good. Radu Trîmbiţaş ( Babeş-Bolyai University) Krylov Space Methods 1 / 17
Krylov Space Methods Nonstationary sounds good Radu Trîmbiţaş Babeş-Bolyai University Radu Trîmbiţaş ( Babeş-Bolyai University) Krylov Space Methods 1 / 17 Introduction These methods are used both to solve
More informationConjugate Gradient Method
Conjugate Gradient Method direct and indirect methods positive definite linear systems Krylov sequence spectral analysis of Krylov sequence preconditioning Prof. S. Boyd, EE364b, Stanford University Three
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method The minimization problem We are given a symmetric positive definite matrix R n n and a right hand side vector b R n We want to solve the linear system Find u R n such that
More informationIn order to solve the linear system KL M N when K is nonsymmetric, we can solve the equivalent system
!"#$% "&!#' (%)!#" *# %)%(! #! %)!#" +, %"!"#$ %*&%! $#&*! *# %)%! -. -/ 0 -. 12 "**3! * $!#%+,!2!#% 44" #% ! # 4"!#" "%! "5"#!!#6 -. - #% " 7% "3#!#3! - + 87&2! * $!#% 44" ) 3( $! # % %#!!#%+ 9332!
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationKrylov Subspace Methods that Are Based on the Minimization of the Residual
Chapter 5 Krylov Subspace Methods that Are Based on the Minimization of the Residual Remark 51 Goal he goal of these methods consists in determining x k x 0 +K k r 0,A such that the corresponding Euclidean
More informationLinear Solvers. Andrew Hazel
Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction
More informationFrom Stationary Methods to Krylov Subspaces
Week 6: Wednesday, Mar 7 From Stationary Methods to Krylov Subspaces Last time, we discussed stationary methods for the iterative solution of linear systems of equations, which can generally be written
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 9 Minimizing Residual CG
More informationITERATIVE METHODS FOR SPARSE LINEAR SYSTEMS
ITERATIVE METHODS FOR SPARSE LINEAR SYSTEMS YOUSEF SAAD University of Minnesota PWS PUBLISHING COMPANY I(T)P An International Thomson Publishing Company BOSTON ALBANY BONN CINCINNATI DETROIT LONDON MADRID
More informationOUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative methods ffl Krylov subspace methods ffl Preconditioning techniques: Iterative methods ILU
Preconditioning Techniques for Solving Large Sparse Linear Systems Arnold Reusken Institut für Geometrie und Praktische Mathematik RWTH-Aachen OUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative
More informationThe conjugate gradient method
The conjugate gradient method Michael S. Floater November 1, 2011 These notes try to provide motivation and an explanation of the CG method. 1 The method of conjugate directions We want to solve the linear
More informationFEM and Sparse Linear System Solving
FEM & sparse system solving, Lecture 7, Nov 3, 2017 1/46 Lecture 7, Nov 3, 2015: Introduction to Iterative Solvers: Stationary Methods http://people.inf.ethz.ch/arbenz/fem16 Peter Arbenz Computer Science
More informationCS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3
CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3 Felix Kwok February 27, 2004 Written Problems 1. (Heath E3.10) Let B be an n n matrix, and assume that B is both
More informationPreconditioned inverse iteration and shift-invert Arnoldi method
Preconditioned inverse iteration and shift-invert Arnoldi method Melina Freitag Department of Mathematical Sciences University of Bath CSC Seminar Max-Planck-Institute for Dynamics of Complex Technical
More informationIDR(s) as a projection method
Delft University of Technology Faculty of Electrical Engineering, Mathematics and Computer Science Delft Institute of Applied Mathematics IDR(s) as a projection method A thesis submitted to the Delft Institute
More informationContribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computa
Contribution of Wo¹niakowski, Strako²,... The conjugate gradient method in nite precision computations ªaw University of Technology Institute of Mathematics and Computer Science Warsaw, October 7, 2006
More informationSparse matrix methods in quantum chemistry Post-doctorale cursus quantumchemie Han-sur-Lesse, Belgium
Sparse matrix methods in quantum chemistry Post-doctorale cursus quantumchemie Han-sur-Lesse, Belgium Dr. ir. Gerrit C. Groenenboom 14 june - 18 june, 1993 Last revised: May 11, 2005 Contents 1 Introduction
More informationPROJECTED GMRES AND ITS VARIANTS
PROJECTED GMRES AND ITS VARIANTS Reinaldo Astudillo Brígida Molina rastudillo@kuaimare.ciens.ucv.ve bmolina@kuaimare.ciens.ucv.ve Centro de Cálculo Científico y Tecnológico (CCCT), Facultad de Ciencias,
More informationContents. Preface... xi. Introduction...
Contents Preface... xi Introduction... xv Chapter 1. Computer Architectures... 1 1.1. Different types of parallelism... 1 1.1.1. Overlap, concurrency and parallelism... 1 1.1.2. Temporal and spatial parallelism
More informationConjugate Gradient Method
Conjugate Gradient Method Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 10, 2011 T.M. Huang (NTNU) Conjugate Gradient Method October 10, 2011 1 / 36 Outline 1 Steepest
More informationChapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices
Chapter 7 Iterative methods for large sparse linear systems In this chapter we revisit the problem of solving linear systems of equations, but now in the context of large sparse systems. The price to pay
More informationMath 1080: Numerical Linear Algebra Chapter 4, Iterative Methods
Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 March 2015 1 / 70 Topics Introduction to Iterative Methods
More informationLecture 9: Krylov Subspace Methods. 2 Derivation of the Conjugate Gradient Algorithm
CS 622 Data-Sparse Matrix Computations September 19, 217 Lecture 9: Krylov Subspace Methods Lecturer: Anil Damle Scribes: David Eriksson, Marc Aurele Gilles, Ariah Klages-Mundt, Sophia Novitzky 1 Introduction
More informationCONVERGENCE BOUNDS FOR PRECONDITIONED GMRES USING ELEMENT-BY-ELEMENT ESTIMATES OF THE FIELD OF VALUES
European Conference on Computational Fluid Dynamics ECCOMAS CFD 2006 P. Wesseling, E. Oñate and J. Périaux (Eds) c TU Delft, The Netherlands, 2006 CONVERGENCE BOUNDS FOR PRECONDITIONED GMRES USING ELEMENT-BY-ELEMENT
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Iteration basics Notes for 2016-11-07 An iterative solver for Ax = b is produces a sequence of approximations x (k) x. We always stop after finitely many steps, based on some convergence criterion, e.g.
More information1 Extrapolation: A Hint of Things to Come
Notes for 2017-03-24 1 Extrapolation: A Hint of Things to Come Stationary iterations are simple. Methods like Jacobi or Gauss-Seidel are easy to program, and it s (relatively) easy to analyze their convergence.
More informationCharles University Faculty of Mathematics and Physics DOCTORAL THESIS. Krylov subspace approximations in linear algebraic problems
Charles University Faculty of Mathematics and Physics DOCTORAL THESIS Iveta Hnětynková Krylov subspace approximations in linear algebraic problems Department of Numerical Mathematics Supervisor: Doc. RNDr.
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationMath 504 (Fall 2011) 1. (*) Consider the matrices
Math 504 (Fall 2011) Instructor: Emre Mengi Study Guide for Weeks 11-14 This homework concerns the following topics. Basic definitions and facts about eigenvalues and eigenvectors (Trefethen&Bau, Lecture
More informationLecture # 20 The Preconditioned Conjugate Gradient Method
Lecture # 20 The Preconditioned Conjugate Gradient Method We wish to solve Ax = b (1) A R n n is symmetric and positive definite (SPD). We then of n are being VERY LARGE, say, n = 10 6 or n = 10 7. Usually,
More informationResearch Article Some Generalizations and Modifications of Iterative Methods for Solving Large Sparse Symmetric Indefinite Linear Systems
Abstract and Applied Analysis Article ID 237808 pages http://dxdoiorg/055/204/237808 Research Article Some Generalizations and Modifications of Iterative Methods for Solving Large Sparse Symmetric Indefinite
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More informationRESIDUAL SMOOTHING AND PEAK/PLATEAU BEHAVIOR IN KRYLOV SUBSPACE METHODS
RESIDUAL SMOOTHING AND PEAK/PLATEAU BEHAVIOR IN KRYLOV SUBSPACE METHODS HOMER F. WALKER Abstract. Recent results on residual smoothing are reviewed, and it is observed that certain of these are equivalent
More informationON ORTHOGONAL REDUCTION TO HESSENBERG FORM WITH SMALL BANDWIDTH
ON ORTHOGONAL REDUCTION TO HESSENBERG FORM WITH SMALL BANDWIDTH V. FABER, J. LIESEN, AND P. TICHÝ Abstract. Numerous algorithms in numerical linear algebra are based on the reduction of a given matrix
More informationIntroduction. Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods. Example: First Order Richardson. Strategy
Introduction Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 Solve system Ax = b by repeatedly computing
More informationConjugate Gradient Method
Conjugate Gradient Method Hung M Phan UMass Lowell April 13, 2017 Throughout, A R n n is symmetric and positive definite, and b R n 1 Steepest Descent Method We present the steepest descent method for
More informationSTEEPEST DESCENT AND CONJUGATE GRADIENT METHODS WITH VARIABLE PRECONDITIONING
SIAM J. MATRIX ANAL. APPL. Vol.?, No.?, pp.?? c 2007 Society for Industrial and Applied Mathematics STEEPEST DESCENT AND CONJUGATE GRADIENT METHODS WITH VARIABLE PRECONDITIONING ANDREW V. KNYAZEV AND ILYA
More informationKey words. linear equations, polynomial preconditioning, nonsymmetric Lanczos, BiCGStab, IDR
POLYNOMIAL PRECONDITIONED BICGSTAB AND IDR JENNIFER A. LOE AND RONALD B. MORGAN Abstract. Polynomial preconditioning is applied to the nonsymmetric Lanczos methods BiCGStab and IDR for solving large nonsymmetric
More informationNumerical Methods for Solving Large Scale Eigenvalue Problems
Peter Arbenz Computer Science Department, ETH Zürich E-mail: arbenz@inf.ethz.ch arge scale eigenvalue problems, Lecture 2, February 28, 2018 1/46 Numerical Methods for Solving Large Scale Eigenvalue Problems
More informationITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD
ITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD Heinrich Voss voss@tu-harburg.de Hamburg University of Technology Institute of Numerical Simulation
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011 Lecture Objectives describe when CG can be used to solve Ax
More informationSolving Ax = b, an overview. Program
Numerical Linear Algebra Improving iterative solvers: preconditioning, deflation, numerical software and parallelisation Gerard Sleijpen and Martin van Gijzen November 29, 27 Solving Ax = b, an overview
More informationAMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Krylov Minimization and Projection (KMP) Dianne P. O Leary c 2006, 2007.
AMSC 600 /CMSC 760 Advanced Linear Numerical Analysis Fall 2007 Krylov Minimization and Projection (KMP) Dianne P. O Leary c 2006, 2007 This unit: So far: A survey of iterative methods for solving linear
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 24: Preconditioning and Multigrid Solver Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 5 Preconditioning Motivation:
More informationIterative Solvers in the Finite Element Solution of Transient Heat Conduction
Iterative Solvers in the Finite Element Solution of Transient Heat Conduction Mile R. Vuji~i} PhD student Steve G.R. Brown Senior Lecturer Materials Research Centre School of Engineering University of
More informationOn the influence of eigenvalues on Bi-CG residual norms
On the influence of eigenvalues on Bi-CG residual norms Jurjen Duintjer Tebbens Institute of Computer Science Academy of Sciences of the Czech Republic duintjertebbens@cs.cas.cz Gérard Meurant 30, rue
More informationConjugate Gradients I: Setup
Conjugate Gradients I: Setup CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Conjugate Gradients I: Setup 1 / 22 Time for Gaussian Elimination
More informationAPPLIED NUMERICAL LINEAR ALGEBRA
APPLIED NUMERICAL LINEAR ALGEBRA James W. Demmel University of California Berkeley, California Society for Industrial and Applied Mathematics Philadelphia Contents Preface 1 Introduction 1 1.1 Basic Notation
More information