Notes on PCG for Sparse Linear Systems

Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova e-mail luca.bergamaschi@unipd.it webpage www.dmsa.unipd.it/ berga 1 The method of the steepest descent Let us now focus on symmetric positive definite matrices. Let us define a function φ R n R φ(x) = 1 2 (x h)t A(x h) = 1 2 et Ae φ(x) 0 x 0 (by definition of spd matrix). Moreover φ(x) = 0 x = h = A 1 b. Summarizing, φ has a unique minimum x = h, so that minimizing φ can be regarded as a method to solve the linear system Ax = b. Let us suppose to have an approximate solution x k. φ(x k ) = constant represents a quadratic form in n dimensions. Geometrically it is an hyperellipsoid in the n-dimensional subspace whose geometric center is the solution h. The method of the steepest descent is aimed to go from x k toward the center of the hyperellipsoid going in the direction of the gradient. The gradient of φ at the point x k is φ(x k ) = 1 ( 1 2 et k Ae k = 2 xt k Ax k x T b + 1 ) 2 ht h = Ax k b = r k Since the direction of the gradient vector is outward, the wanted direction coincides with the residual of our linear system in the current approximation x k. Once we have obtained the search direction (WITHOUT knowing the exact solution) we can write our next approximation as x k+1 = x k + α k r k where α k is a suitable constant to be determined so as to minimize function φ(x) along the search direction r k. This is an easier problem as compared to the initial one. We have to perform a monodimensional optimization of function φ. Formally α k = argmin t φ(x k + tr k ) 1

1 The method of the steepest descent 2 Let us write in detail φ(x k + tr k ): it turns out to be a second order polynomial in the variable t. φ(x k + tr k ) = (x k + tr k h) T A(x k + tr k h) = (x k + tr k h) T (Ax k + tar k b) = (x k + tr k h) T (tar k r k ) = (tr k e k ) T (tar k r k ) Being r T k Ar k > 0 the minimum of φ is attained for Alternatively:solve φ t 1.1 Steepest descent. Algorithm = t 2 r T k Ar k t ( r T k r k + e T k Ar k ) + h T r k = t 2 r T k Ar k 2tr T k r k + const. rt k r k α k t = r T k Ar k = 0 (use the chain rule to differentiate). The k + 1th iterate of the method of the steepest descent can be written as r k = b Ax k α k = rt k r k r T k Ar k x k+1 = x k + α k r k Note that this iteration costs mainly two matrix-vector products. We can even do better by observing that from x k+1 = x k + α k r k it follows that r k+1 = b Ax k+1 = b Ax k Aα k r k = r k α k Ar k. This allows us to write a more efficient algorithm for the SD method. Consecutive residuals r k+1, r k are orthogonal (proof by Exercise). The error e k+1 is A orthogonal to the search direction r k. (proof by Exercise). Algorithm. Steepest Descent Input: x 0, A, b, k max, tol r 0 = b Ax 0, k = 0 while r k > tol b and k < k max do 1. z = Ar k 2. α k = rt k r k z T r k 3. x k+1 = x k + α k r k 4. r k+1 = r k α k z 5. k = k + 1 end while

1 The method of the steepest descent 3 1.2 Convergence of the SD Theorem 1.1. (proof not given) φ(xk ) = e T k Ae k = e k A,2 µ k e 0 A,2, where µ = κ(a) 1 κ(a) + 1 Order the eigenvalues of A as 0 < λ n λ 2 λ 1, then κ(a) = λ 1. λ n Again when dealing with system arising from PDEs, κ(a) may be very large. It is useful to compute an estimate of the number of iteration needed to gain p digits in the approximation of the solution: e k A e 0 A 10 p by solving the inequality ( ) k κ(a) 1 10 p κ(a) + 1 Taking logarithms and using the first order Taylor approximation log κ(a) 1 κ(a) + 1 2, we obtain κ(a) + 1 log 10 k p (κ(a) + 1) 2 ( ) 1 0 Exercise 1.1. 1. Solve the system Ax = 0 with A = using the SD method. with initial 0 9 guess x 0 = (9, 1) T. Find x 1, x 2. prove that x k = 0.8 k (9, 1) T. Verify the convergence bounds of previous frame ( ) 1 0 2. Solve the system Ax = 0 with A = using the SD method. with initial guess x 0 9 0 = (9, 1/9) T. Find x 1, x 2. 2 1.5 1 0.5 0 0.5 1 1.5 2 2 1 0 1 2 Slow convergence: consecutive residuals are orthogonal. It depends on relation between e 0 and the eigenvectors of A. Note that (exercise) if e 0 is an eigenvector then x 1 = h.

2 The Conjugate Gradient (CG) Method 4 2 1.5 1 0.5 0 0.5 1 1.5 2 2 1 0 1 2 Faster convergence: why? 2 The Conjugate Gradient (CG) Method The Conjugate Gradient method can be viewed as an improvement of the SD method. We start from a similar recurrence relation between consecutive iterates. x k+1 = x k + α k p k Now we allow our search direction to be different from the gradient, namely p 0 = r 0 p k+1 = r k+1 + β k p k, k 0 Differently from SD we require that the error at step k+1 is also A-orthogonal to the previous direction p k. To this aim we impose that wich gives 0 = p T k Ap k+1 = p T k A (r k+1 + β k p k ) = p T k Ar k+1 + β k p T k Ap k, β k = rt k+1 Ap k p T k Ap k From this definition of β k we have that p T k Ap k+1 = 0 i.e. p k+1 and p k are A-orthogonal. Exercise 2.1. Prove the following results: a) As in the SD case choice of α k is aimed to minimize φ(x k+1 ) = φ(x k + α k p k ) giving the following formula rt k p k α k = p T k Ap k b) Consecutive residuals as in the SD satisfy a recurrence relation c) Consecutive residuals are orthogonal namely r k+1 = r k α k Ap k r T k+1r k = 0.

2 The Conjugate Gradient (CG) Method 5 2.1 The finite termination property of CG Theorem 2.1. The CG algorithm generates the exact solution h in at most n steps. Moreover the sequences e k, r k and p k satisfy: Proof. Assume that {r j } 0, j k. Basis of the induction. e T k+1ap j = 0, p T k+1ap j = 0, r T k+1r j = 0, j k p T 1 Ap 0 = 0 by the definition of β 0 r T 0 r 1 = r T 0 (r 0 α 0 Ar 0 ) = r T 0 r 0 α 0 r T 0 Ar 0 = 0 since α 0 = rt 0 p 0 p T 0 Ap = rt 0 r 0 0 r T 0 Ar 0 e T 1 Ap 0 = r T 1 r 0 = 0. We assume as inductive hypothesis that e T k Ap j = 0, p T k Ap j = 0, r T k r j = 0, j k 1 Note that the first relation also implies that r T k p j = 0, j k 1. Then for j k 1 we have e T k+1ap j = (e k + α k p k ) T Ap j = e T k Ap j + α k p T k Ap j = 0 (= r T k+1p j = 0); r T k+1r j = r T ( ) k+1 pj β j 1 p j 1 = r T k+1 p j β j 1 r T k p j 1 = 0 p T k+1ap j = (r k+1 + β k p k ) T Ap j = r T k+1ap j + β k p T k Ap j = r T 1 k+1 (r j+1 r j ) = α j = 1 α j ( r T k+1 r j+1 r T k+1r j ) = 0 (by Exercise 2.1 c) and the previous relation) It remains to prove that e T k+1ap k = 0, p T k+1ap k = 0, r T k+1r k = 0. The relation p T k+1 Ap k = 0 immediately follow from the definition of β k. Moreover r T k+1 r k = 0 by Exercise 2.1. Finally by definition of α k. e T k+1ap k = (e k + α k p k ) T Ap k = e T k Ap k + α k p T k Ap k = r T k p k + α k p T k Ap k = 0, 2.2 Consequences of Theorem 2.1 From the Theorem we have that r n = 0, p n = 0 which implies x n = h. This is the finite termination property. CG stops (in exact arithmetics!!) after at most n iterations. The error at step k + 1 can be defined in terms of the initial error and a polynomial of degree k in matrix A. In fact the following result holds: Theorem 2.2. with Q k (0) = 1. k+1 e k+1 = e 0 + c j A j e 0 = (I + AP k (A))e 0 = Q k+1 (A)e 0 j=1

2 The Conjugate Gradient (CG) Method 6 Proof. By induction. e 1 = e 0 + α 0 p 0 = e 0 + α 0 r 0 = e 0 α 0 Ae 0 = Q 1 (A)e 0 With Q 1 (x) = 1 α 0 x, Q 1 (0) = 1. Suppose e j = Q j (A)e 0, j k, then e k+1 = e k + α k p k = e k + α k ( rk + β k 1 p k 1 ) = by setting Q k+1 (A) = (1 α k A)Q k (A) + c 1 Q k 1 (A). = e k α k Ae k + β k 1 α k 1 (e k e k 1 ) = = (1 α k A)e k + c 1 e k 1 = Q k+1 e 0 Minimization property ( of the CG. Actually, e k+1 minimizes the A norm on all the vectors which k+1 ) can be written as e 0 + j=1 c ja j e 0. This is the outcome of the following Theorem 2.3. The sequences of errors produce by the CG method satisfies: e k+1 A = min (I + AP k (A))e 0 A P k Proof. From we have, recursively that x k+1 = x k + α k p k, x k+1 = x 0 + with S k = [ p 0 p 1 p k ] k α k p k = x 0 + S k z = e k+1 = e 0 + S k z. j=0 α 0 z = α 1. α k We have to prove that the A norm of the error is the smallest possible over all choices of vector z. Hence g(z) = e k+1 2 A = e T k+1ae k+1 = e T 0 Ae 0 + 2e T 0 AS k z + z T S T k AS k z = e 0 2 A 2r T 0 S k z + z T Λ k z where the (i, j) entry of Λ k is given by p T i Ap j, which is zero if i j. Matrix Λ k is therefore diagonal. Function g(z) takes its minimum value in ẑ such that g(z) = 0. Componentwise g = 2Λ k z 2S T k r 0 ; g = 0 z = Λ 1 k ST k r 0 α k = z k = pt k r 0 p T k Ap, k which are exactly the α s provided by the CG algorithm, if we finally observe that k 1 r k = r k 1 α k 1 Ap k 1 = = r 0 α j Ap j, hence p T k r k = p T k r 0. j=0

2 The Conjugate Gradient (CG) Method 7 2.3 Other useful relations. 1. p T k r k = r T k r k. From e T k Ap j = 0 immediately follows r T k p j = 0 and hence 2. r T k Ap k = p T k Ap k (left by Exercise). p T k r k = ( r k + β k 1 p k 1 ) T rk = r T k r k 3. Combining 1. and 2. we obtain an alternative definition of α k : rt k p k α k = p T k Ap k = rt k r k r T k Ap k 4. Alternative (and more numerically stable) formulation for scalars α k, β k p T k Ap k = p T k 1 α k (r k r k+1 ) = 1 α k r T k r k = α k = pt k Ap k r T k r k Therefore r T k+1ap k = r T 1 k+1 (r k r k+1 ) = 1 r T α k α k+1r k+1 k β k = rt k+1 Ap k p T k Ap k = rt k+1 r k+1 r T k r k After these rewritings we are able to show the final, and most efficient, version of the Conjugate Gradient method: Algorithm: Conjugate Gradient Input: x 0, A, b, k max, tol r 0 = p 0 = b Ax 0, k = 0 ρ 0 = r T 0 r 0 while r k > tol b and k < k max do 1. z = Ap k 2. α k = ρ k z T p k = rt k r k p T k Ap k 3. x k+1 = x k + α k p k 4. r k+1 = r k α k z 5. ρ k+1 = r T k+1 r k+1 6. β k = ρ k+1 ρ k = rt k+1 r k+1 r T k r k 7. p k+1 = r k+1 + β k p k 8. k = k + 1 end while

2 The Conjugate Gradient (CG) Method 8 2.4 Convergence rate of the CG Method Finite termination property does not have practical utility (on a computer no termination due to loss of orthogonality) + n iterations are too many. Can we measure how fast do iterate approach the solution? Start from: e k A = min P k 1 (I + AP k 1 (A))e 0 A We define Q k (A) = I + AP k 1 (A) (that is Q k (0) = 1). Now being A normal we can orthogonally diagonalize it as A = UΛU T and since A is also spd it exists A = Udiag ( λ 1,, λ n ) U T. Then e k 2 A = e T k Ae k = ( Ae k ) T ( Ae k ) = Ae k 2 e k A = min Q k = min Q k AQ k (A)e 0 = min Q k max Q k(λ i ) e 0 A i [1,n] UQ k (Λ)U T Ae 0 min Q k (Λ) e 0 A Q k If λ n and λ 1 (the smallest and largest eigenvalue, respectively) are known then a bound of the error norm reduction can be given in terms of shifted and scaled Chebyshev polynomials which satisfy a so called minimax condition. Theorem 2.4. The A-norm of the error at the k-th CG iterate satissfies: ( ) k e k A κ(a) 1 2. e 0 A κ(a) + 1 Proof. See Luenberger, 1969 and many other books on iteriatve methods. It is useful to compute an estimate of the number of iteration needed to gain p digits in the approximation of the solution: ( ) k e k A κ(a) 1 10 p by solving the inequality 10 p e 0 A κ(a) + 1 Taking logarithms and using the first order Taylor approximation log κ(a) 1 κ(a) + 1 2 κ(a) + 1, we log 10 obtain k p ( κ(a) + 1) 2 Example. Optimality of Chebyshev polynomials. Suppose eigenvalues of A are 1, 2, 10. Scaled and shifted 2nd order Chebyshev polynomial on [1, 10] has the following form T 2 (x) = 2 ( ) 2x 11 2 9 1 2 ( ) 11 2, max{ T 2 (λ i ) } = T 2 (1) (or T 2 (10) ) = 81 9 1 161 0.503 It satisfies T 2 (x) = arg min max P 2(x). What we really want is something that minimizes the P 2(x) x [1,10] absolute value of the polynomial at the eigenvalues not on the whole spectral interval. So our Q 2 (x) = 1 + ax + bx 2. We can find optimal a, b by imposing the conditions Q 2 (1) = Q 2 (2) = Q 2 (10) i.e. solving the following linear system (Q 2 (1) =) 1 + a + b = γ (Q 2 (2) =) 1 + 2a + 4b = γ (Q 2 (10) =) 1 + 10a + 100b = γ

3 Preconditioning 9 which gives a = 33 42, b = 1 14, γ = max Q 2(λ i ) = 0.2857 Note that, as known, the Chebyshev polynomial T 2 (x) provides the minimum of the maximum value 1.5 Chebyshev polynomial 1 (33/42)x + x*x/14 1 0.5 0 0 2 4 6 8 10 in the interval [1, 10] while the computed polynomial Q 2 (x), though achieving larger values within the interval minimizes max{ Q 2 (1), Q 2 (2), Q 2 (10) }. Summing up: e 2 0.503 Chebyshev polynomials A 0.2857 optimal polynomial e 0 A = 0.1147 Computed ratio (A = diag(1, 2, 10), b = (1, 2, 10) T, x 0 = (0, 0, 0) T.) 3 Preconditioning Fast convergence of the CG method is guaranteed if κ(a) is sufficiently small or, at least a sort of partial condition number λ s+1 λ n p 1 very large and separated eigenvalues. To obtain a faster convergence we would like either to reduce the condition number is small in the case matrix A has s n very small and p n or to obtain a clustering of the majority of the eigenvalues around a single value In both cases we are trying to round the ellipsoid. Preconditioning a linear system is nothing but (pre)multiplying the system by a nonsingular matrix, which we call M 1, thus yielding an equivalent linear system. M 1 Ax = M 1 b (1)

3 Preconditioning 10 The preconditioner M 1, should be also easily invertible, or in other words, should allow a fast solution of a linear system of the form My = c. The CG method is applicable only if system matrix is spd. However, the preconditioned system is no longer symmetric. We now show that system (1) is equivalent to another one in which the coefficient matrix is spd provided that also M is. If M is spd then m admits a real square root that we call X so that M = XX. System (1) can be written as X 1 X 1 Ax = X 1 X 1 b or, premultiplying by X, X 1 Ax = X 1 b Let us now define: x = Xx, b = X 1 b, B = X 1 AX 1 ; we can rewrite the system as Bx = b (2) with B spd matrix (also note that B and M 1 A, being similar, have the same eigenvalues). We then can apply the CG method to the transformed system 2 whose coefficient matrix is now SPD. Unfortunately we are not able to compute explicitly the square root of a large and sparse matrix, so computation of X can not be done in practice. However, the Preconditioned Conjugate Gradient method can be efficiently implemented as we will see later. Let us now write in detail the PCG in the new variables. x k = Xx k, p k = Xp k, r k = b Bx k = X 1 b X 1 AX 1 Xx k = X 1 r k α k = r T k p k p T k Bp k r T k = X 1 Xp k p T = α k XX 1 AX 1 k Xp k β k = r T k+1r k+1 = rt k+1 M 1 r k+1 r T k r r T k k M 1 r k x k+1 = x k + α k p k = Xx k+1 = Xx k + α k Xp k = x k+1 = x k + α k p k r k+1 = r k α k Bp k = X 1 r k+1 = X 1 r k α k X 1 AX 1 Xp k = r k+1 = r k α k Ap k p k+1 = r k+1 + β p k = Xp k+1 = X 1 r k+1 + β Xp k = p k+1 = M 1 r k+1 + β p k It turns out that every CG equation in the new variable can be rewritten in terms of the old variable. The result is that matrix X completely disappears in the algorithm. To be able to implement the PCG method we must compute efficiently the operation M 1 r k+1, which is needed to compute the residual and the scalar β. 3.1 Computational issues A single iteration of PCG is made of 1. 3 scalar products + 3 daxpys 1 = 6 O(n) operations 2. 1 matrix vector product 3. 1 application of preconditioner (solution of Mg k+1 = r k+1 ). 1 A daxpy operation is of the form x = x + αy, with x, p vectors anf α a scalar.

4 Classical preconditioners 11 Algorithm. Preconditioned Conjugate Gradient (PCG) In Input: x 0, A, M, b, k max, tol r 0 = b Ax 0, p 0 = M 1 r 0, k = 0, ρ 0 = r T 0 M 1 r 0 while r k > tol b and k < k max do 1. z k = Ap k 2. α k = pt k r k z T k p k 3. x k+1 = x k + α k p k 4. r k+1 = r k α k z k 5. g k+1 = M 1 r k+1, ρ k+1 = r T k+1 g k+1 6. β k = rt k+1 M 1 r k+1 r T k M 1 r k = ρ k+1 ρ k 7. p k+1 = g k+1 + β k p k 8. k = k + 1 end while 4 Classical preconditioners How to choose the preconditioner? It should mediate between two opposite requirements: 1. Reducing the number of iterations (by clustering eigenvalues). 2. Allowing cheap solution of system Mg k+1 = r k+1. Jacobi preconditioner M = D where D is the (spd) diagonal matrix satisfying d ii = a ii, i = 1,..., n Computation of M 1 r k+1 costs O(n) operations and hence is very cheap. Does this preconditioner improve the condition number of the system? This is the case e.g. for diagonally dominant matrices. In general we can expect only a slight reduction in the iteration number using this preconditioner. Incomplete Cholesky factorization M = L L T where L is an approximation (sparsification) of the triangular factor obtained by the Cholesky factorization. Since we need to solve a system with matrix M we require L be as sparse as possible. A common choice (which leads to the IC(0) preconditioner) is to allow that L has nonzeros only in those position where also A is different from zero, that is a ij = 0 = l ij 0 Fixed a subset S [1,..., n] [1,..., n] of positions in the matrix then the factor L is initialized to the lower triangular factor of A and then updated according to the following procedure:

5 Numerical Example 12 for k=1:n A(k,k)=sqrt(A(k,k)) for i=k+1:n if (A(i,k) ~=0) A(i,k)=A(i,k)/A(k,k) end end for j=k+1:n for i=j:n if (A(i,j) ~=0) A(i,j)=A(i,j)-A(i,k)*A(j,k) end end end end The incomplete Cholesky factorization may not be properly defined even if A is spd (Existence results are known for M-matrices and H-matrices). Breakdown may occur when A kk = 0. However in practice very seldom IC(0) breaks down. Here the keyword is sparsity since computation of g k+1 = M 1 r k+1 is performed by a sequential solution of two sparse triangular linear systems: from g k+1 = (LL T ) 1 r k+1 we obtain LL T g k+1 = r k+1 and hence: Ly = r k+1 L T g k+1 = y When using IC(0) the computational cost of a single iteration is roughly doubled with respect to the non-preconditioned CG. Moreover the condition number of the preconditioned system may decrease of a few orders of magnitude thus producing an effective acceleration of the iterative method. Approximate inverse preconditioners They build M 1 = Z T Z where Z is a sparse lower triangular matrix approximating L 1. In the AINV approach (Benzi et al.) Z is constructed by means of a sparse Gram-Schmidt orthogonalization. 5 Numerical Example A is the result of a Mixed Finite Element discretization of a 2D diffusion equation. n = 77120, nz = 384320, κ 2 (A) = 4 10 6. Results on a Pentium 1GHz (PCG Fortran implementation) with tol=10 7 : Preconditioner iterations cpu cpu per iter - 6022 60.78 0.010 Jacobi 1299 14.00 0.011 Cholesky 246 6.76 0.027 Unfortunately, not even IC(0) avoids dependence of condition number on h. Preconditioners which do this are multigrid/domain decomposition based preconditioners.

5 Numerical Example 13 5.1 An important example: the diffusion equation A number of different physical processes can be described by the diffusion equation: u( x, t) t (K u( x, t)) = f( x), x Ω, t (0, T ) Here u might represent the temperature distribution at time t in an object Ω, to which an external heat source f is applied. The positive definite matrix K( x) is the thermal conductivity of the material. To determine the temperature at time t, we need to know an initial temperature distribution u( x, 0) and some boundary conditions, of the form u( x, t) = 0 on Ω 1 u( x, t) n = 0 on Ω 2 where n is the outward normal to the domain Ω. Two standard method are commonly used for obtaining approximate solutions to such partial differential equations: The Finite Difference method. 1. The region Ω is divided into small pieces 2. At each point of a grid on Ω the derivatives are replaced by difference quotients as a linear combination of the unknown values at the gridpoints. The Finite Element method ff 1. The region Ω is divided into small pieces 2. The differential equation is projected onto a finite dimensional space and its solution is find so as to minimize the residual onto another finite dimensional space. A simplified form of the diffusion equation is the Laplace equation whose solution gives the steady state solution. 2 u x 2 + 2 u y 2 = 0 x T=0 o C i,j+1 1 4 7 y T=100 o C 2 5 8 T=0 o C i-1,j i,j i+1,j 3 6 9 Ap- i,j-1 o T=100 C proximations of second derivatives can be made by 2nd order Taylor expansion as: 2 u x 2 = u i+1,j 2u i,j + u i 1,j x 2, 2 u y 2 = u i,j+1 2u i,j + u i,j 1 y 2

5 Numerical Example 14 Assuming x = y the Laplace equation on node i, j is approximated by u i+1,j + u i 1,j + u i,j+1 + u i,j 1 4u i,j = 0 The result is a system with as many rows as the interior nodes of the grid. Every interior node produces a linear equation with the values of temperature at nodes as unknowns. Temperature on boundary nodes is known and contributes to the rhs. 4 1 0 1 0 0 0 0 0 1 4 1 0 1 0 0 0 0 0 1 4 0 0 1 0 0 0 1 0 0 4 1 0 1 0 0 0 1 0 1 4 1 0 1 0 0 0 1 0 1 4 0 0 1 0 0 0 1 0 0 4 1 0 0 0 0 0 1 0 1 4 1 0 0 0 0 0 1 0 1 4 5.2 Discretized diffusion equation u 1 u 2 u 3 u 4 u 5 u 6 u 7 u 8 u 9 = After discretization in the space, the PDE is converted in a system of ODEs like P u t = Hu + b 100 100 200 0 0 100 0 0 100 where P = I in FD discretization. Stiffness matrix H is symmetric definite negative, capacity or mass matrix P is symmetric positive definite. Finally discretization in time transforms the system of ODEs to a linear system of size equal to the number of gridpoints to be solved at each time step. Time discretization divides the time interval (0, T ] into subintervals of length t i so that solution in time is computed at t k = k i=1 i and uses Finite Differences to approximate the time derivative. For example applying the implicit Euler method (u u(t k+1) u(t k ) t k ). (I + t k H)u(t k+1 ) = t k P u(t k ) + t k b The 2D FD discretization of the Laplace equation takes a general form (in the case x = y = h): S I 4 1 H = 1 I S I 1 4 1 h 2......... where S =......... I S I 1 4 1 I S 1 4 Eigenvalues of S and hence those of H are explicitly known: λ j,k (H) = 4 ( ) jπh h 2 sin2 + 4 ( ) kπh 2 h 2 sin2, j, k = 1,..., n x = 1 2 h 1 Theorem 5.1. The smallest and largest eigenvalues of H behave like: λ n = 2π 2 + O(h 2 ), λ 1 = 8h 2 + O(1).

5 Numerical Example 15 Proof. λ n = 8 ( ) πh h 2 sin2 = 8 2 h 2 (hπ/2 + O(h3 )) 2 = 2π 2 + O(h 2 ). λ 1 = 8 ( ) nx πh h 2 sin2 = 8 ( π 2 h 2 sin2 2 πh ) = 2 = 8 ( ) πh h 2 cos2 = 8 2 h 2 (1 O(h2 )) 2 = 8h 2 + O(1). Corollary 5.2. The condition number of H behaves like Corollary 5.3. κ(h) = 4 π 2 h 2 + O(1) (κ(h) = 2d π d h 2 when Ω R d ) The number of iteration of the CG for solving Hx = b is proportional to n 1D discretizations h 1 = n 2D discretizations. 3 n 3D discretizations 5.3 Numerical Results Finite Difference discretization of the steady-state Laplace equation on a unitary square domain with n x = n y = {100, 200, 400, 800}. The system has been solved by the Conjugate Gradient method with no preconditioner (CG) or accelerated by the Incomplete Cholesky preconditioner provided by MatLAB L = ichol(a,struct( type, ict, droptol,tol)); base on drop tolerances of tol = 10 2, 10 3. n x N CG PCG (10 2 ) PCG (10 3 ) 98 9 604 370 65 27 198 39 204 719 120 47 398 158 404 1365 230 88 798 636 804 2655 409 169 Tab. 1: Number of iterations of CG and PCG with the IC factorization Note from the table the dipendence of the number of iterations on h 1 N.