Iterative methods for linear systems

Transcription

1 Iterative methods for linear sstems Chris H. Rcroft November 9th, 26 Introduction For man elliptic PDE problems, finite-difference and finite-element methods are the techniques of choice. In a finite-difference approach, a solution u k on a set of discrete gridpoints,..., k is searched for. The discretized partial differential equation and boundar conditions create linear relationships between the different values of u k. In the finite-element method, the solution is epressed as a linear combination u k of basis functions λ k on the domain, and the corresponding finite-element variational problem again gives linear relationships between the different values of u k. Regardless of the precise details, all of these approaches ultimatel end up with having to find the u k that satisf all the linear relationships prescribed b the PDE. This can be written as a matri equation of the form Au = b () where one aims to find a solution u, given that A is a matri capturing the differentiation operator, and b corresponds to an source or boundar terms. Theoreticall, this problem could be solved on a computer b an of the standard methods for dealing with matrices. However, the real challenge for PDEs is that frequentl the dimensionalit of the problem can be enormous. For eample, for a two-dimensional PDE problem, a grid would be a perfectl reasonable size to consider. Thus u would be a vector with 4 elements, and A would be a matri with 8 elements. Even allocating memor for such a large matri ma be problematic. Direct approaches, such as the eplicit construction of A, are impractical. The ke to making progress is to note that in general the matri A is etremel sparse, since the linear relationships usuall onl relate nearb gridpoints together. It is therefore natural to seek methods that do not require ever eplicitl specifing all the elements of A, but eploit its special structure directl. Man of these methods are iterative starting with Electronic address: chr@seas.harvard.edu.

2 a guess u k, a process is applied that ields a closer solution u k+. Tpicall, these iterative methods are based on a splitting of A. This is a decomposition A = M K where M is non-singular. An splitting creates a possible iterative process. Equation can be rewritten as and hence a possible iteration is (M K)u = b, (2) Mu = Ku + b, (3) u = M Ku + M b, (4) u k+ = M Ku k + M b. (5) Of course, there is no guarantee that an arbitrar splitting will result in an iterative method that converges. To stud convergence, one must look at the properties of the matri R = M K. For convergence analsis, it is helpful to introduce the spectral radius ρ(r) = ma { λ j, (6) j where the λ j are the eigenvalues of R. It can be shown [2] that an iterative scheme converges if and onl if ρ(r) <. The size of the spectral radius determines the convergence rate, and ideall one would like to find splittings that result in as small a ρ(r) as possible. An eample: a two-dimensional Poisson problem In the convergence analsis later, we will consider a two-dimensional Poisson problem on the square,, given b the equation 2 u = f, (7) subject to the Dirichlet conditions that u(, ) vanishes on the boundar. We use a source function of the form { if <.5 and <.5 f (, ) = otherwise. (8) This is plotted on a grid in figure. For convergence properties, the eigenfunctions and eigenvalues of this function are ver important, and to determine these, it is helpful to consider an associated one-dimensional Poisson problem on the interval, d2 u = f (), (9) d2 2

3 subject to Dirichlet boundar conditions u( ) = u() =. We consider a discretization into N + 2 gridpoints such that j = + 2j/(N + ) for j =,..., N +. When constructing the corresponding matri problem, u and u N+ need not be considered, since their values are alwas fied to zero. B discretizing the second derivative according to d 2 u d 2 = u j + u j+ 2u j =j h 2 () where h = 2/(N + ), the corresponding linear sstem is 2 u. u T N. =... u... N 2 2 u u. u N = h2 f f. f N. () We epect that the eigenvectors of T N ma be based on sine functions. A reasonable guess for the jth eigenfunction is 2 z j (k) = N + sin πkj N +. (2) To verif this is an eigenfunction, and find its eigenvalue, we appl T N to obtain ( ( ) 2 TN z j (k) = 2 sin πkj ) π(k + )j π(k )j sin sin. (3) N + N + N + N + Note that this epression will alwas be valid for the range k =, 2,..., N, and the boundar values will automaticall work out. The last two sine functions can be rewritten using a trigonometric identit to give ( ( ) 2 TN z j (k) = 2 sin πkj N + N + 2 sin ( 2 = N + 2 πj cos N + = 2 ( cos πj N + ) sin πkj N + cos πkj N + ) πj N + ) z j (k) (4) and hence z j is an eigenvector with eigenvalue λ j = 2( cos πj/(n + )). The smallest eigenvalue is λ = 2( cos π/(n + )) and the largest is λ N = 2( cos Nπ/(N + )). Returning to the two-dimensional problem, the corresponding derivative matri T N N can be written as the tensor product of two one-dimensional problems T N. Its eigenvectors are the tensor products of the one-dimensional eigenvectors, namel z i,j (k, l) = z i (k)z j (l), (5) 3

4 f Figure : A sample source function f (, ) on a grid. u Figure 2: The eact solution to the 2D Poisson problem 2 u = f, with zero boundar conditions and a source term given in figure. 4

5 . u Figure 3: The solution to the eample 2D Poisson problem after ten iterations of the Jacobi method. and the corresponding eigenvalues are λ i,j = λ i + λ j. (6) The Jacobi Method The Jacobi method is one of the simplest iterations to implement. While its convergence properties make it too slow for use in man problems, it is worthwhile to consider since it forms the basis of other methods. Starting with an initial guess u, one successivel applies an iteration of the form for j = to N do ) u m+,j = b j k =j a jk u m,k end for a jj ( In other words, the jth component of u is set so that it eactl satisfies equation j of the linear sstem. For the two-dimensional Poisson problem considered above, this corresponds to an iteration of the form for i = to N do for j = to N do 5

6 u m+,i,j = ( h 2 f j + u m,i,j+ + u m,i,j + u m,i+,j + u m,i,j ) /4 end for end for To find the corresponding matri form, write A = D L U where D is diagonal, L is lower-triangular, and U is upper-triangular. Then the above iteration is u m+ = D (L + U)u m + D b. (7) The convergence properties, discussed later, are then set b the matri R J = D (L + U). The Jacobi method has the advantage that for each m, the order in which the components of u m+ are computed has no effect this ma be a favorable propert to have in some parallel implementations. However, it can also be seen that u m must be retained until after u m+ is constructed, meaning we must store u m+ in a different part of the memor. The listing given in Appendi A. carries out the Jacobi iteration on the Poisson test function. It makes use of two arras for the storage of u, computing the odd u k in one and the even u k in the other. Figure 3 shows a the progress of the Jacobi method after ten iterations. The Gauss Seidel Method The Gauss Seidel method improves on the Jacobi algorithm, b noting that when updating a particular point u m+,j, one might as well reference the alread updated values u m+,,...,u m+,j in the calculation, rather than using the original values u m,,..., u m,j. The iteration can be written as for j = to N ( do ) u m+,j = a b jj j j k= a jku m+,k k=j+ N u m,k end for The Gauss Seidel algorithm has the advantage that in a computer implementation, it is not necessar to allocate two arras for u m+ and u m. Instead, a single arra for u m can be used and all updates can be carried out in situ. However, the Gauss Seidel implementation introduces an additional complication that the order in which the updates are applied affects the values of u m. For a two-dimensional problem, two particular orderings are worth special attention: Natural ordering this is the tpical ordering that would result in a for loop. We first loop successivel through all elements of the first row (, ),..., (, n) before moving onto the second row, and so on. Since this frequentl corresponds to the order that the grid is stored within the computer memor, this order can lead to performance improvements due to good cache efficienc. 6

7 .2 u Figure 4: The Gauss Seidel solution to the eample 2D Poisson problem after ten iterations. The crinkles in the solution are due to the Red Black update procedure. Red Black ordering this is the ordering that results b coloring the gridpoints red and black in a checkerboard pattern. Specificall, a gridpoint (i, j) is colored red if i + j is even and colored black if i + j is odd. During the Gauss Seidel update, all red points are updated before the black points. For the two-dimensional Poisson problem, updating a red grid point onl requires information from the black gridpoints, and vice versa. Hence the order in which points in each set are updated does not matter. The whole Gauss Seidel update is divided into a red gridpoint update and black gridpoint update, and this can be helpful in the convergence analsis. From the algorithm above, the corresponding matri splitting for the Gauss Seidel method is (D L)u m+ = Uu m + b, (8) u m+ = (D L) Uu m + (D L) b. (9) Appendi A.2 contains a C++ code to carr out a Gauss Seidel method on the eample problem, and the result after ten iterations is shown in figure 4. Successive Over-Relaation Successive Over-Relaation (SOR) is a refinement to the Gauss Seidel algorithm. At each stage in the Gauss Seidel algorithm, a value u m,j is updated to a new one u m+,j, which is 7

8 u Figure 5: The SOR solution (using the theoreticall optimal ω) to the eample 2D Poisson problem after ten iterations. The solution is closer to the answer than the Jacobi or Gauss Seidel methods. equivalent to displacing u m,j b an amount u = u m+,j u m,j. The SOR algorithm works b displacing the values b an amount ω u, where tpicall ω >, in the hope that if u is a good direction to move in, one might as well move further in that direction. The iteration is for j = to N do ( ) u m+,j = ( ω)u m,j + a ω b jj j j k= a jku m+,k k=j+ N u m,k end for The corresponding matri form is (D + ωl)u m+ = [( ω)d Uω] u m + ωb, (2) u m+ = (D + ωl) [( ω)d Uω] u m + (D + ωl) ωb. (2) Appendi A.3 contains a C++ code to carr out the SOR iteration on the eample problem, and the result is shown in figure 5. In the SOR algorithm, the value of ω can be chosen freel, and the best choices can be determined b considering the eigenfunctions of the associated problem. This is discussed in more detail below. 8

9 Convergence analsis and compleit To eamine the convergence properties of the different methods, we need to look at the associated spectral radii. For the Jacobi method, the matri of interest is R J = D (L + U). For the 2D Poisson problem, D = 4I and hence R J = (4I) (4I T N N ) = I T N N. (22) 4 The largest eigenvalue of R J corresponds to the smallest of T N N, namel ( ) ( ) π π λ, = 4 2 cos 2 cos N + N + ( ) π = 4 4 cos. (23) N + Hence ( ) π ρ(r J ) = cos. (24) N + Epanding the cosine as a Talor series shows that ρ(r J ) π2. The time to gain 2(N+) an etra digit of accurac is approimatel 2 log ρ(r J ) N2, (25) so the algorithm must be run for O(N 2 ) iterations to attain a specific level of accurac. Since there are O(N 2 ) gridpoints for the 2D problem, the total running time is O(N 4 ). For detailed proofs of the convergence properties of the other methods, the reader should consult the book b Demmel [2]. It can be shown that ( ) π ρ(r GS ) = cos 2, (26) N + so that one iteration of the Gauss Seidel method is equivalent to two Jacobi iterations. Note however the compleit is the same: O(N 2 ) iterations are still required to reach a desired level of accurac. For the SOR algorithm, it can be shown that the optimal value of ω is 2. (27) + ρ(r J ) 2 and that for this value ρ(r SOR ) 2 2π N +. (28) Since there is a factor of N in the denominator as opposed to N 2, the order of computation decreases to O(N) per gridpoint. 9

10 5 Jacobi Gauss Seidel Optimal SOR L 2 -norm error Iterations Figure 6: Errors versus the number of effective iterations for the Jacobi, Gauss Seidel, and SOR methods, applied to the eample 2D Poisson problem on a grid. The plots are in line with the theoretical results of the tet. The Gauss Seidel method is faster than the Jacobi method, but has still not reached double numerical precision after iterations. The SOR method is significantl faster, but still requires 2 iterations to reach double numerical precision. Figure 6 shows a plot of mean square error against the number of iterations for the model problem with the Jacobi, Gauss Seidel, and optimal SOR method. The lines agree with the above results. The SOR method reaches numerical precision within 2 iterations, while the other two methods have not full converged even after 4 iterations. Multigrid One of the major problems with the three methods considered so far is that the onl appl locall. Information about different cell values onl propagates b one or two gridpoints per iteration. However, for man elliptic problems, a point source ma cause an effect over the entire domain. The above methods have a fundamental limitation that the will need to be applied for at least as man iterations as it takes for information to propagate across the grid. As such, we should not epect to ever do better than O(N) operations per point. This can also be seen b considering the eigenvalues. The maimal eigenvalue of R J was set b the λ,, corresponding to the lowest order mode. While the methods ma effectivel damp out high frequenc oscillations, it takes a ver long time to correctl capture the lowest modes with the largest wavelengths. The multigrid method circumvents these limitations b introducing a hierarch of

11 f () f () f (2) f (3) Figure 7: The restriction of the eample source term f using the multigrid method with v down =, v up = 2. Top left: the initial grid, with gridpoints. Top right: grid, with 7 7 gridpoints. Bottom left: grid 2, with 9 9 gridpoints. Bottom right: grid 3, with 5 5 gridpoints. coarser and coarser grids. Tpicall, at each level, the number of gridpoints is reduced b a factor of two in each direction, with the coarsest grid having roughl ten to twent points. To find a solution, we restrict the source term to the coarse grids, refine the solution on each, and interpolate up to the original grid. On the coarser grids, the lower frequenc modes in the final solution can be dealt with much more effectivel. Since the coarser grids have progressivel fewer gridpoints, the time spent computing them is minimal. Because of this, the multigrid algorithm requires onl O() computation per point, which is the best order of compleit that can be achieved. To be more specific, let the original problem be on grid, and we then introduce a sequence of successivel coarser grids,..., g. Write u (i) and b (i) to represent the solution and source terms on the ith grid. A multigrid algorithm requires the following: A solution operator S(u (i), b (i) ) that returns a better approimation u (i) to the solution on the ith level, An interpolation operator T(u (i) ) that returns an interpolation u (i ) on the (i )th level, A restriction operator R(b (i) ) that returns a restriction b (i+) on the (i + )th level. The interpolation and restriction operators can be thought of as rectangular matrices. As an eample, consider a problem with nine equall-spaced gridpoints on the unit interval,

12 u (3) u (2) u () u () Figure 8: The solution of the v in the first multigrid V-ccle for the eample 2D Poisson problem. Top left: the eact solution on grid 3 to the f (3) source term in figure 7. Top right: the interpolation and refinement on grid 2. Bottom left: the interpolation and refinement on grid. Bottom right: the interpolation and refinement on grid. Even after a single V-ccle, the solution is closer to the eact solution than the plots of 3, 4, and 5. at (, /8, 2/8,..., ). Let grid have five points at (, /4, 2/4, 3/4, ), and let grid 2 have three gridpoints at (, /2, ). Interpolation operators between the grids can be written as T () = /2 /2 /2 /2 /2 /2 /2 /2, T (2) /2 /2 /2 /2 where the function values at gridpoints common between the two levels are kept, and the function values at midpoints are calculated using linear interpolation. Similarl, the (29) 2

13 restriction operators can be written as R () = /4 /2 /4 /4 /2 /4 /4 /2 /4 R () = /4 /2 /4 (3) Without the boundar points, R (i) is the proportional to the transpose of T (i+), and this propert is ver useful in proving the convergence of the multigrid method. However, this propert is not strictl necessar to create an efficient multigrid algorithm. Other methods of interpolating and restricting are also possible, and it should also be noted that while a grid of size 2 n + has a particularl convenient multigrid formulation, the multigrid method can be applied to grids of arbitrar size. Given a differential operator matri A = A () on the top level, corresponding matrices on the lower levels can be calculated according to A (i) = R (i ) A (i ) T (i). (3) With this definition, a solution operator S(u (i), b (i) ) can be constructed as a single red black Gauss Seidel sweep. The multigrid algorithm is then given b function Multi(u (i), b (i) ) if i = g then compute eact solution to A (i) u (i) = b (i) return u (i) else for j = to v down do u (i) = S(u (i), b (i) ) end for r (i) = b (i) A (i) u (i) d (i) = T(Multi( (i+), R(r (i) ))) u (i) = u (i) + d (i) for j = to v up do u (i) = S(u (i), b (i) ) end for end if Here, (i+) represents a zero vector on grid level i +. The multigrid function is applied recursivel, and at each stage, the remainder of the problem on the level above is sent to the lower level. The algorithm starts on level, descends to level g, and then ascends to level again, following the shape of a V. It is therefore referred to as the multigrid V-ccle. Other more elaborate methods of moving between grids are possible, although the V-ccle is ver efficient in man situations. In the algorithm, we are free to choose the number of. 3

14 5 L 2 -norm error Jacobi Gauss Seidel Optimal SOR Multigrid, v down =, v up = 2 Multigrid, v down = v up = Effective iterations Figure 9: Errors versus the number of effective iterations for the several different iteration techniques. Here, to allow a direct comparison, effective iterations for the multigrid methods is defined b the number of Gauss Seidel iterations that are applied on the top grid level, since the Gauss Seidel iterations on the coarser grids are small in comparison. For the v down =, v up = 2 method, the effective number of iterations is twice the number of V-ccles. For the v down = v up = 2 method, the algorithm was slightl modified, so that onl two Gauss Seidel iterations were applied at the top level each time, instead of the epected four. Thus the effective number of iterations is also twice the number of V-ccles. The speed of the multigrid methods is startling when compared to an of the other three iterations. times the solution operator is applied on the wa down and on the wa up, and tpical good values to tr ma be v down = v up = 2, or even v down =, v up = 2. It ma also be worthwhile to carr out more iterations on the coarser grids, since the computation is much cheaper there. Figure 7 shows the restriction of the source term in the test problem on the coarser grids. Figure 8 shows the solution being successivel refined on the grids. Even after a single V-ccle, the solution is close to the eact answer. Figure 9 shows the computation times for two different multigrid algorithms, compared with the previous three methods considered. The multigrid algorithms reach numerical precision etremel quickl, much faster even than SOR. Onl twent Gauss Seidel iterations are applied at the top level before double numerical precision has been reached. A Code listings The following codes were used to generate the Jacobi, Gauss Seidel, and SOR diagrams in these notes. The are written in C++ and were compiled using the GNU C++ compiler. 4

15 Each of the first three routines calls a common code listed in appendi A.4 for setting up useful constants and defining common routines. This common code also contains a function for outputting the 2D matrices in a matri binar format that is readable b the plotting program Gnuplot []. This output routine could be replaced in order to save to different plotting programs. A. Jacobi method jacobi.cc // Load common routines and constants #include "common.cc" int main() { int i,j,ij,k; double error,u[m n],v[m n],z; double a, b; // Set initial guess to be identicall zero for(ij=;ij<m n;ij++) u[ij]=v[ij]=; output and error("jacobi out",u,); // Carr out Jacobi iterations for(k=;k<=total iters;k++) { // Alternatel flip input and output matrices if (k%2==) {a=u;b=v; else {a=v;b=u; // Compute Jacobi iteration for(j=;j<n ;j++) { for(i=;i<m ;i++) { ij=i+m j; a[ij]=(f(i,j)+dinv (b[ij ]+b[ij+]) +dinv (b[ij m]+b[ij+m])) dcent; // Save and compute error if necessar output and error("jacobi out",a,k); A.2 Gauss Seidel gsrb.cc // Load common routines and constants #include "common.cc" int main() { 5

16 int i,j,ij,k; double error,u[m n],z; // Set initial guess to be identicall zero for(ij=;ij<m n;ij++) u[ij]=; output and error("gsrb out",u,); // Compute Red Black Gauss Seidel iteration for(k=;k<=total iters;k++) { for(j=;j<n ;j++) { for(i=+(j&);i<m ;i+=2) { ij=i+m j; u[ij]=(f(i,j)+dinv (u[ij ]+u[ij+]) +dinv (u[ij m]+u[ij+m])) dcent; for(j=;j<n ;j++) { for(i=2 (j&);i<m ;i+=2) { ij=i+m j; u[ij]=(f(i,j)+dinv (u[ij ]+u[ij+]) +dinv (u[ij m]+u[ij+m])) dcent; // Save the result and compute error if necessar output and error("gsrb out",u,k); A.3 Successive Over-Relaation sor.cc // Load common routines and constants #include "common.cc" int main() { int i,j,ij,k; double error,u[m n],z; // Set initial guess to be identicall zero for(ij=;ij<m n;ij++) u[ij]=; output and error("sor out",u,); // Compute SOR Red Black iterations for(k=;k<=total iters;k++) { for(j=;j<n ;j++) { for(i=+(j&);i<m ;i+=2) { 6

17 ij=i+m j; u[ij]=u[ij] ( omega)+omega (f(i,j) +dinv (u[ij ]+u[ij+]) +dinv (u[ij m]+u[ij+m])) dcent; for(j=;j<n ;j++) { for(i=2 (j&);i<m ;i+=2) { ij=i+m j; u[ij]=u[ij] ( omega)+omega (f(i,j) +dinv (u[ij ]+u[ij+]) +dinv (u[ij m]+u[ij+m])) dcent; // Save the result and compute error if necessar output and error("sor out",u,k); A.4 Common routine for setup and output common.cc // Load standard libraries #include <cstdio> #include <cstdlib> #include <iostream> #include <fstream> #include <cmath> using namespace std; // Set grid size and number of iterations const int save iters=2; const int total iters=2; const int error ever=2; const int m=33,n=33; const double min=,ma=; const double min=,ma=; // Compute useful constants const double pi= ; const double omega=2/(+sin(2 pi/n)); const double d=(ma min)/(m ); const double d=(ma min)/(n ); const double dinv=/(d d); const double dinv=/(d d); const double dcent=/(2 (dinv+dinv)); 7

18 // Input function inline double f(int i,int j) { double =min+i d,=min+j d; return abs() >.5 abs()>.5?:; // Common output and error routine void output and error(char filename,double a,const int sn) { // Computes the error if sn%error ever== if(sn%error ever==) { double z,error=;int ij; for(int j=;j<n ;j++) { for(int i=;i<m ;i++) { ij=i+m j; z=f(i,j) a[ij] (2 dinv+2 dinv) +dinv (a[ij ]+a[ij+]) +dinv (a[ij m]+a[ij+m]); error+=z z; cout << sn << " " << error d d << endl; // Saves the matri if sn<=save iters if(sn<=save iters) { int i,j,ij=,ds=sizeof(float); float,,data float;const char pfloat; pfloat=(const char )&data float; ofstream outfile; static char fname[256]; sprintf(fname,"%s.%d",filename,sn); outfile.open(fname,fstream::out fstream::trunc fstream::binar); data float=m;outfile.write(pfloat,ds); for(i=;i<m;i++) { =min+i d; data float=;outfile.write(pfloat,ds); for(j=;j<n;j++) { =min+j d; data float=; 8

19 outfile.write(pfloat,ds); for(i=;i<m;i++) { data float=a[ij++]; outfile.write(pfloat,ds); outfile.close(); References [] [2] J. W. Demmel, Applied numerical linear algebra, SIAM, 997. [3] G. H. Golub and C. H. Van Loan, Matri computations, Johns Hopkins Universit Publishers,