6.3. Poisson's Equation

Transcription

1 Iterative Methods for Liear Systems Poisso's Equatio Poisso's Equatio i Oe Dimesio We begi with a oe-dimesioal versio of Poisso's equatio, where f(x) is a give fuctio ad v(x) is the ukow fuctio that we wat to compute. v(x) must also satisfy the boudary coditios 24 v(0) = v(l) = 0. We discretize the problem by tryig to compute a approximate solutio at N + 2 evely spaced poits Xi betwee 0 ad 1: Xi = ih, where h 1/N+1 ad 0 i N + 1. We abbreviate Vi = v(x i ) ad f i = f(x i ). To covert differetial equatio (6.1) ito a liear equatio for the ukows v 1,...,v N, we use fiite differeces to approximate differ- Subtractig these approximatios ad dividig by h yield the cetered ece approximatio where T i, the so-called trucatio error, ca be show to be O(h 2 d 4 v/dx 4 )- We may ow rewrite equatio (6.1) at x = x i { as where 0 < i < N+l. Sice the boudary coditios imply that V o = V N+I = 0, we have N equatios i N ukows v 1,..., v N: 24 These are called Dirichlet boudary coditios. Other kids of boudary coditios are also possible.

2 268 Applied Numerical Liear Algebra Fig Eigevalues of T 21. or To solve this equatio, we will igore, sice it is small compared to /, to get (We boud the error v v later.) The coefficiet matrix T N plays a cetral role i all that follows, so we will examie it i some detail. First, we will compute its eigevalues ad eigevectors. Oe ca easily use trigoometric idetities to cofirm the followig lemma (see Questio 6.1). LEMMA 6.1. The eigevalues of T N are j 2(1 cos )- The eigevectors are Z j, where Z j (k] = 2/N+1 si(jk /(N + 1)). Z j has uit two-orm. Let Z = \z \,..., z ] be the orthogoal matrix whose colums are the eigevectors, ad A diag( 1,..., ), so we ca write T N = Z Z T. Figure 6.1 is a plot of the eigevalues of T N for N = 21. The largest eigevalue is N = 2(1 COS = 4. The smallest eigevalue 25 is 1, where for small i 25 Note that N is the largest eigevalue ad 1 is the smallest eigevalue, the opposite of the covetio of Chapter 5.

3 Iterative Methods for Liear Systems 269 Fig Eigevectors of T 21. Thus T N is positive defiite with coditio umber N/ 1 = 4(N+ l) 2 / 2 for large N. The eigevectors are siusoids with lowest frequecy at j = 1 ad highest at j = N, show i Figure 6.2 for N = 21. Now we kow eough to boud the error, i.e., the differece betwee the solutio of TNV = h 2 f ad the true solutio v of the differetial equatio: Subtract equatio (6.5) from equatio (6.4) to get v v = h 2 T -l N T. Takig orms yields so the error v v goes to zero proportioally to h 2, provided that the solutio is smooth eough. ( d 4 v/dx 4 oo is bouded.) From ow o we will ot distiguish betwee v ad its approximatio v ad so will simplify otatio by lettig T N v = h 2 f. I additio to the solutio of the liear system h~ 2 T N v = f approximatig the solutio of the differetial equatio (6.1), it turs out that the eigevalues ad eigevectors of h~ 2 T N also approximate the eigevalues ad eigefuctios of the differetial equatio: We say that i is a eigevalue ad Z i (x) is a eigefuctio of the differetial equatio if

4 270 Applied Numerical Liear Algebra Let us solve for i ad Zi(x}: It is easy to see that Zi(x) must equal a. si(\f\ix}-\- ficos(\/\ix) for some costats a ad. The boudary coditio z i (0) = 0 implies (3 0, ad the boudary coditio Zi(l) = 0 implies that i is a iteger multiple of, which we ca take to be i. Thus i = i 2 2 ad Zi(x) = a si(i x} for ay ozero costat a (which we ca set to 1). Thus the eigevector Zi is precisely equal to the eigefuctio Zi(x) evaluated at the sample poits Xj = jh (whe scaled by 2/N+1). Ad whe iissmall, i = i 2 2 is well approximated by h~ 2. i = (N+l) 2. 2(l-cos i /N+1) = i 2 2 +O((iV+l)- 2 ). Thus we see there is a close correspodece betwee T N (or h~ 2 T N ) ad the secod derivative operator d 2 /dx 2.This correspodece will be the motivatio for the desig ad aalysis of later algorithms. It is also possible to write dow simple formulas for the Cholesky ad LU factors of T N ; see Questio 6.2 for details Poisso's Equatio i Two Dimesios Now we tur to Poisso's equatio i two dimesios: o the uit square {(x,y) : 0 < x, y < 1}, with boudary coditio v = 0 o the boudary of the square. We discretize at the grid poits i the square which are at (x i,y j ) with X i = ih ad y j = jh, with h = 1/N+1 We abbreviate Vij = v(ih,jh) ad f ij,- = f(ih,jh), as show below for N = 3: From equatio (6.2), we kow that we ca approximate

5 Iterative Methods for Liear Systems 271 Addig these approximatios lets us write where T ij {J is agai a trucatio error bouded by O(h 2 ). The heavy (blue) cross i the middle of the above figure is called the (5-poit} stecil of this equatio, because it coects all (5) values of v preset i equatio (6.9). From the boudary coditios we kow V oj = VN+IJ V i o = V i N+I = 0 so that equatio (6.9) defies a set of = N 2 liear equatios i the ukows Vij for 1 i,j N: There are two ways to rewrite the equatios represeted by (6.10) as a sigle matrix equatio, both of which we will use later. The first way is to thik of the ukows v ij as occupyig a N-by-N matrix V with etries v^ ad the right-had sides h 2 f ij as similarly occupyig a N-by-N matrix h 2 F. The trick is to write the matrix with i,j etry 4vij Vi-i,j Vi+1,j Vi,j-1 Vi,j + 1 i a simple way i terms of V ad T N :- Simply ote that 2vij - vi-1,j - V i+1,j = (T N V)ij, 2vij -vi,j-1- v i,j+1 = (V T N )IJ, so addig these two equatios yields This is a liear system of equatios for the ukow etries of the matrix V, eve though it is ot writte i the usual "Ax = 6" format, with the ukows formig a vector x. (We will write the "Ax = b" format below.) Still, it is eough to tell us what the eigevalues ad eigevectors of the uderlyig matrix A are, because "Ax = Ax" is the same as "T N V + VT N = V." Now suppose that T N ZI izi ad T NZJ = jzj are ay two eigepairs of T N, ad let V = ZiZT i. The

6 272 Applied Numerical Liear Algebra so V = ZizT j is a "eigevector" ad i + j is a eigevalue. Sice V has N 2 etries, we expect N 2 eigevalues ad eigevectors, oe for each pair of eigevalues i ad \j of T N. I particular, the smallest eigevalue is 2 1 ad the largest eigevalue is 2 N, so the coditio umber is the same as i the oe-dimesioal case. We rederive this result below usig the "Ax = b" format. See Figure 6.3 for plots of some eigevectors, represeted as surfaces defied by the matrix etries of ZizT J. Just as the eigevalues ad eigevectors of h~ 2 T N were good approximatios to the eigevalues ad eigefuctios of oe-dimesioal Poisso's equatio, the same is true of two-dimesioal Poisso's equatio, whose eigevalues ad eigefuctios are as follows (see Questio 6.3): The secod way to write the equatios represeted by equatio (6.10) as a sigle matrix equatio is to write the ukows Vij i a sigle log N 2 - by-1 vector. This requires us to choose a order for them, ad we (somewhat arbitrarily) choose to umber them as show i Figure 6.4, columwise from the upper left to the lower right. For example, whe N = 3 oe gets a colum vector v [v 1,..., V 9 ] T. If we umber / accordigly, we ca trasform equatio (6.10) to get The 1's immediately ext to the diagoal correspod to subtractig the top ad bottom eighbors v i,j -\ v i j+1. The 1's farther away away from the diagoal correspod to subtractig the left ad right eighbors Vi-ij Vi+1,j. For geeral N, we cofirm i the ext sectio that we get a N 2 -by-n 2 liear system

7 Iterative Methods for Liear Systems 273 Fig Three-dimesioal ad cotour plots of first four eigevectors of the 10-by-10 Poisso equatio.

8 274 Applied Numerical Liear Algebra Fig Numberig the ukows i Poisso's equatio. where T N X N has N N-by-N blocks of the form T N + 2I N o its diagoal ad IN blocks o its offdiagoals: Expressig Poisso's Equatio with Kroecker Products Here is a systematic way to derive equatios (6.15) ad (6.16) as well as to compute the eigevalues ad eigevectors of T NxN. The method works equally well for Poisso's equatio i three or more dimesios. DEFINITION 6.1. Let X be m-by-. The vec(x) is defied to be a colum vector of size m made of the colums of X stacked atop oe aother from left to right. Note that N 2 -by-l vector v defied i Figure 6.4 ca also be writte v vec(v). To express T NXN V as well as compute its eigevalues ad eigevectors, we eed to itroduce Kroecker products. DEFINITION 6.2. Let A be a m-by- matrix ad B be a p-by-q matrix. The A 0 B, the Kroecker product of A ad B, is the (m p)-by-( q) matrix The followig lemma tells us how to rewrite the Poisso equatio i terms of Kroecker products ad the vec(.) operator.

9 Iterative Methods for Liear Systems 275 LEMMA 6.2. Let A be m-by-m, B be -by-, ad X ad C be m-by-. The the followig properties hold: 1. vec(ax) = (I A) vec(x). 2. vec(xb) = (B T I m ) vec(x). 3. The Poisso equatio T N V + VT N = h 2 F is equivalet to Proof. We prove oly part 3, leavig the other parts to Questio 6.4. We start with the Poisso equatio T N V +VT N = h 2 F as expressed i equatio (6.11), which is clearly equivalet to vec(t N V + VT N ) = vec(t N V) + vec(vt N ) = vec(h 2 F). By part 1 of the lemma By part 2 of the lemma ad the symmetry of T N, Addig the last two expressios completes the proof of part 3. The reader ca cofirm that the expressio from equatio (6.17) agrees with equatio (6.16). 26 To compute the eigevalues of matrices defied by Kroecker products, like T NXN, we eed the followig lemma, whose proof is also part of Questio 6.4. LEMMA 6.3. The followig facts about Kroecker products hold: 1. Assume that the products A C ad B D are well defied. The (A B).(C D) = (A.C) (B.D). 26 We ca use this formula to compute T N X N i two lies of Matlab: T N = 2*eye(N) - diag(oes(n-l,l),1) - diag(oes(n-1,1),-1); TNxN = kro(eye(n),tn) + kro(tn,eye(n));

10 276 Applied Numerical Liear Algebra 2. If A ad B are ivertible, the (A B}~ 1 = A~ l B~ l. 3. (A B) T = A T B T. PROPOSITION 6.1. Let T N = Z Z T be the eigedecompositio of T N, with Z = [Z I,...,Z N ] the orthogoal matrix whose colums are eigevectors, ad A = diag( 1,..., XN). The the eigedecompositio Of T N X N = I T N +T N I is is a diagoal matrix whose (in + j)th diagoal etry, the (i,j)th eigevalue of T N x N, is i,j i + j. Z 0 Z is a orthogoal matrix whose (in + j)th colum, the correspodig eigevector, is Zi Zj. Proof. Prom parts 1 ad 3 of Lemma 6.3, it is easy to verify that Z Z is orthogoal, sice I I = I. We ca ow verify equatio (6.18): by part 3 of Lemma 6.3 by part 1 of Lemma 6.3 Also, it is easy to verify that + I is diagoal, with diagoal etry (in+ j) give by j+ i, so that equatio (6.18) really is the eigedecompositio of T N x N. Fially, from the defiitio of Kroecker product, oe ca see that colum in + j of Z Z is Zi{ Z j. The reader ca cofirm that the eigevector Z i Z j vec(zjz T i), thus matchig the expressio for a eigevector i equatio (6.12). For a geeralizatio of Propositio 6.1 to the matrix A I + B T I, which arises whe solvig the Sylvester equatio AX X B = C, see Questio 6.5 (ad Questio 4.6). Similarly, Poisso's equatio i three dimesios leads to with eigevalues all possible triple sums of eigevalues of T N, ad eigevector matrix Z Z Z. Poisso's equatio i higher dimesios is represeted aalogously.

11 Iterative Methods for Liear Systems 277 Method Dese Cholesky Explicit iverse Bad Cholesky Jacobi's Gauss-Seidel Sparse Cholesky Cojugate gradiets Successive overrelaxatio SSOR with Chebyshev accel. Fast Fourier trasform Block cyclic reductio Multigrid Lower boud Serial Time / 2 3 / 2 3 / 2 5 /4 log - log Space Direct or Sectio Iterative / 2 log D D D I I D I I I D D I Table 6.1. ( = N 2 }. Order of complexity of solvig Poisso's equatio o a N-by-N grid 6.4. Summary of Methods for Solvig Poisso's Equatio Table 6.1 lists the costs of various direct ad iterative methods for solvig the model problem o a N-by-N grid. The variable = N 2, the umber of ukows. Sice direct methods provide the exact aswer (i the absece of roudoff), whereas iterative methods provide oly approximate aswers, we must be careful whe comparig their costs, sice a low-accuracy aswer ca be computed more cheaply by a iterative method tha a high-accuracy aswer. Therefore, we compare costs, assumig that the iterative methods iterate ofte eough to make the error at most some fixed small value 27 (say, 10~ 6 ). The secod ad third colums of Table 6.1 give the umber of arithmetic operatios (or time) ad space required o a serial machie. Colum 4 idicates whether the method is direct (D) or iterative (I). All etries are meat i the O(.) sese; the costats deped o implemetatio details ad the stoppig criterio for the iterative methods (say, 10~ 6 ). For example, the etry for Cholesky also applies to Gaussia elimiatio, sice this chages the costat oly by a factor of two. The last colum idicates where the algorithm is discussed i the text. The methods are listed i icreasig order of speed, from slowest (dese 27 Alteratively, we could iterate util the error is O(h 2 ) O((N + I) - 2 ), the size of the trucatio error. Oe ca show that this would icrease the costs of the iterative methods i Table 6.1 by a factor of O(log ).

12 278 Applied Numerical Liear Algebra Cholesky) to fastest (multigrid), edig with a lower boud applyig to ay method. The lower boud is because at least oe operatio is required per solutio compoet, sice otherwise they could ot all be differet ad also deped o the iput. The methods are also, roughly speakig, i order of decreasig geerality, with dese Cholesky applicable to ay symmetric positive defiite matrix ad later algorithms applicable (or at least provably coverget) oly for limited classes of matrices. I later sectios we will describe the applicability of various methods i more detail. The "explicit iverse" algorithm refers to precomputig the explicit iverse of T N X N, ad computig v = T -1 N x N f by a sigle matrix-vector multiplicatio (ad ot coutig the flops to precompute T -1 N x N). Alog with dese Cholesky, it uses 2 space, vastly more tha the other methods. It is ot a good method. Bad Cholesky was discussed i sectio 2.7.3; this is just Cholesky takig advatage of the fact that there are o etries to compute or store outside a bad of 2N + 1 diagoals. Jacobi's ad Gauss-Seidel are classical iterative methods ad ot particularly fast, but they form the basis for other faster methods: successive overrelaxatio, symmetric successive overrelaxatio, ad multigrid, our fastest algorithm. So we will study them i some detail i sectio 6.5. Sparse Cholesky refers to the algorithm discussed i sectio 2.7.4: it is a implemetatio of Cholesky that avoids storig or operatig o the zero etries of T N X N or its Cholesky factor. Furthermore, we are assumig the rows ad colums of T N X N have bee "optimally ordered" to miimize work ad storage (usig ested dissectio [112, 113]). While sparse Cholesky is reasoably fast o Poisso's equatio i two dimesios, it it sigificatly worse i three dimesios (usig O(N 6 ) = 0( 2 ) time ad O(N 4 ) = 0( 4 / 3 ) space), because there is more "fill-i" of zero etries durig the algorithm. Cojugate gradiets are a represetative of a much larger class of methods, called Krylov subspace methods, which are very widely applicable both for liear system solvig ad fidig eigevalues of sparse matrices. We will discuss these methods i more detail i sectio 6.6. The fastest methods are block cyclic reductio, the fast Fourier trasform (FFT), ad multigrid. I particular, multigrid does oly 0(1) operatios per solutio compoet, which is asymptotically optimal. A fial warig is that this table does ot give a complete picture, sice the costats are missig. For a particular size problem o a particular machie, oe caot immediately deduce which method is fastest. Still, it is clear that iterative methods such as Jacobi's, Gauss-Seidel, cojugate gradiets, ad successive overrelaxatio are iferior to the FFT, block cyclic reductio, ad multigrid for large eough. But they remai of iterest because they are buildig blocks for some of the faster methods ad because they apply to larger classes of problems tha the faster methods. All of these algorithms ca be implemeted i parallel; see the lectures o PARALLEL-HOMEPAGE for details. It is iterestig that, depedig o

13 Iterative Methods for Liear Systems 279 the parallel machie, multigrid may o loger be fastest. This is because o a parallel machie the time required for separate processors to commuicate data to oe aother may be as costly as the floatig poit operatios, ad other algorithms may commuicate less tha multigrid Basic Iterative Methods I this sectio we will talk about the most basic iterative methods: Jacobi's, Gauss-Seidel, successive overrelaxatio (SOR(w)), Chebyshev acceleratio with symmetric successive overrelaxatio (SSOR(w)). These methods are also discussed ad their implemetatios are provided at NETLIB/templates. Give X o,, these methods geerate a sequece x m covergig to the solutio A~ l b of Ax = 6, where x m+1 is cheap to compute from x m. DEFINITION 6.3. A splittig of A is a decompositio A = M K, with M osigular. A splittig yields a iterative method as follows: Ax = M X K x b implies MX = Kx + b or x M~ l K x + M~ l b Rx + c. So we ca take X m+1 Rx m + c as our iterative method. Let us see whe it coverges. LEMMA 6.4. Let \\ \\ be ay operator orm (\\R\\ max x o Rx / x ). If 11R11 < 1, the x m+1 = Rx m + c coverges for ay X o.. Proof. Subtract x = R x+c from x m+1 = Rx m +c to get x m+1 x = R(x m x}. Thus x m+1 x\\ \\R\\ \\x m x\\ \\R\\ m+l \\X o x\\, which coverges to 0 sice \\R\\ < 1. Our ultimate covergece criterio will deped o the followig property of R. max A, where the max- DEFINITION 6.4. The spectral radius of R is p(r) imum is take over all eigevalues A of R. LEMMA 6.5. For all operator orms p(r) \\R\\. For all R ad for all > 0 there is a operator orm \\ * such that \\R\\ * p(r) + c. The orm \\ * depeds o both R ad. Proof. To show p(r) \\R\\ for ay operator orm, let x be a eigevector for A, where p(r} = \X\ ad so.r = max y o