Relaxaton Methods for Iteratve Soluton to Lnear Systems of Equatons Gerald Recktenwald Portland State Unversty Mechancal Engneerng Department gerry@pdx.edu
Overvew Techncal topcs Basc Concepts Statonary Methods a.k.a. Relaxaton methods Gauss Sedel Jacob SOR Context Relaxaton methods are not compettve as stand-alone teratve solvers. Relaxaton methods are used as smoothers n multgrd. Relaxaton methods provde the foundaton for consderng more sophstcated teratve solvers. Relaxaton Soluton to Lnear Systems page 1
Drect and Iteratve Methods Solve Ax = b where A s n n matrx, x and b are n element column vectors. Gaussan elmnaton wth backward substtuton s a Drect Method A Drect Method obtans the soluton n a fnte number of steps Soluton s equvalent to x = A 1 b Iteratve Methods obtan a sequence of approxmatons to the soluton x k k = 0, 1, 2,... such that (b Ax k ) 0 as k. Relaxaton Soluton to Lnear Systems page 2
Resdual The resdual r = b Ax s zero when x s the soluton to Ax = b. For an teratve method the resdual at teraton k s r k = b Ax k k = 0, 1, 2,... If an teratve methods converges, then r k 0 as k. Relaxaton Soluton to Lnear Systems page 3
Relaxaton Methods (1) Overvew Smple to program Converges slowly for large systems of equatons (large n) Not a useful stand alone soluton method Key ngredent to multgrd methods Examples Jacob Gauss-Sedel SOR Relaxaton Soluton to Lnear Systems page 4
Relaxaton Methods (2) Basc Idea Update elements of the soluton vector one element at a tme Solve a nodal equaton by assumng other nodal values are known N Apply the fnte volume method to get a S φ S a W φ W + a P φ P a E φ E a N φ N = b W P E If k s the teraton counter, we solve for φ k+1 P S φ k+1 P = 1 [ ] b + a S φ k S a + a W φ k W + a Eφ k E + a Nφ k N P Values of φ W, φ E, φ N and φ S are updated by applyng the teratve formula to the cells centered at those neghbor nodes. Relaxaton Soluton to Lnear Systems page 5
Matrx Notaton (1) On a structured mesh the nteror nodes can be numbered sequentally wth a mappng lke J=n y +2 =n x n y = I 1 + (J 2)n x 3 =n x +1 where n x s the number of cells n the x drecton and x s the value of the th dependent feld varable whch s located at the cells wth ndces I and J. The compass pont notaton y 2 =1 =2 =n x J=1 I=1 2 3 I=n x +2 x a S φ S a W φ W + a P φ P a E φ E a N φ N = b becomes a S, x n x a W, x 1 + a P, x a E, x +1 a W, x +n x = b Relaxaton Soluton to Lnear Systems page 6
Matrx Notaton (2) The natural orderng of nodes on a two-dmensonal structured mesh yelds a coeffcent matrx wth fve dagonals. Coeffcents on the dagonals correspond to a P coeffcent of φ P x and the four neghbor coeffcents a S, a W, a E, and a N. A = a S a W a P a E a N Relaxaton Soluton to Lnear Systems page 7
Matrx Notaton (3) In an unstructured mesh, the coeffcent matrx s also sparse, but the non-zero elements of the coeffcent matrx do not le along dagonals. Unstructured Mesh Non-zeros n coeffcent matrx 0 10 20 30 40 50 60 70 80 To smplfy the presentaton, we ll use a structured mesh. 90 0 10 20 30 40 50 60 70 80 90 nz = 361 Relaxaton Soluton to Lnear Systems page 8
Relaxaton Methods Use relaxaton methods to solve Ax = b Relaxaton methods do not work for all A, but they do work for standard fnte-dfference and fnte-volume dscretzatons of the Posson equaton. Jacob Gauss-Sedel SOR Successve Over-relaxaton Relaxaton Soluton to Lnear Systems page 9
Jacob Iteraton (1) A dscrete model of the Posson equaton yelds a system of equatons that can be wrtten Ax = b. The equaton for the th nodal value s n a j x j = b j=1 where n s the total number of nodes n the doman. Extract the dagonal term from the sum a x + n a j x j = b j=1 j The precedng equaton wll not be satsfed untl, n the lmt as the number of teratons ncreased to. Relaxaton Soluton to Lnear Systems page 10
Solve for the next guess x. Jacob Iteraton (2) x = 1 ( b a n ) a j x j j=1 j ( ) In terms of the compass-pont notaton for the two-dmensonal fnte-volume method, the Equaton ( ) s φ (k+1) P = 1 [ b + a S φ (k) S a P + a W φ (k) W + a Eφ (k) E ] + a Nφ (k) N Equaton ( ) s not really the soluton for x untl all the other x j are known. In other words, the value of x from Equaton ( ) wll be correct only when the system of equatons Ax = b s solved. If matrx A has favorable propertes, the value of x from Equaton ( ) wll be closer to the soluton than the prevous guess at x. Relaxaton Soluton to Lnear Systems page 11
Jacob Iteraton (3) Let k be the teraton counter. The Jacob teraton formula s x (k+1) = 1 ( b a n j=1 j ) a j x (k) j = 1,..., n Ths formula mples that two copes of the x vector are mantaned n memory: one for teraton k and one for teraton k + 1. for k=1:maxt % -- For nodes away from the boundares for =(nx+1):n-nx xnew() = (b() + as()*x(-nx) + aw()*x(-1)... + an()*x(+nx) + ae()*x(+1) ) / ap(); end x = xnew; end Relaxaton Soluton to Lnear Systems page 12
Jacob Iteraton (4) The Jacob teratve formula can be wrtten as a matrx equaton. Let D = dag(a 11,..., a nn ) and B = D A then x (k+1) = 1 ( b a n j=1 j ) a j x (k) j s equvalent to x (k+1) = D 1 ( b + Bx (k) ) (1) where x (k+1) and x (k) are the soluton vectors at teraton k + 1 and k, respectvely. Relaxaton Soluton to Lnear Systems page 13
Jacob Iteraton (5) The matrx form of the teraton can be rewrtten x (k+1) = Hx (k) + d (2) where H = D 1 B d = D 1 b The matrx formulaton s prmarly useful n the analyss of teratve methods and n toy mplementatons n Matlab. The egenvalues of H determne the convergence rate of the teratons. Relaxaton Soluton to Lnear Systems page 14
Stoppng Crtera (1) Recall that the goal s to solve Ax = b where A s n n matrx, x and b are n element column vectors. Iteratve Methods obtan a sequence of approxmatons to the soluton x k k = 0, 1, 2,... such that b Ax k 0 as k. The resdual r = b Ax s zero when x s the soluton to Ax = b. Relaxaton Soluton to Lnear Systems page 15
Stoppng Crtera (2) For an teratve method the resdual at teraton k s r k = b Ax k k = 0, 1, 2,... If an teratve methods converges, then r k 0 as k. Snce r s a vector, what does t mean that r k 0? Use a convenent vector norm to test whether r s small enough. Relaxaton Soluton to Lnear Systems page 16
Jacob Iteraton (3) Soluton to a toy problem 2 T x + 2 T 2 y = 0 2 on 0 x L, 0 y W T = 0 on three boundares Soluton 4.5 4 3.5 unform q on fourth boundary 3 2 1.5 1 0.5 0 0 0.5 1 1.5 Soluton s obtaned by Matlab code demojacob Relaxaton Soluton to Lnear Systems page 17
Jacob Iteraton (4) Convergence of Jacob teratons slow as mesh s refned. 10 0 8 x 8 16 x 16 32 x 32 10 1 r 1 / r 0 1 10 2 10 3 10 4 0 200 400 600 800 1000 1200 1400 Iteraton Relaxaton Soluton to Lnear Systems page 18
Gauss-Sedel Iteraton (1) In the Jacob method x (k+1) s obtaned from the frozen values of x (k). The Gauss-Sedel method uses a smlar formula for updatng x (k+1), but the new value of s used as soon as t s avalable x (k+1) values from teraton k y values from teraton k+1 x Relaxaton Soluton to Lnear Systems page 19
Gauss-Sedel Iteraton (2) Replace nodal values as sweep progresses through the doman for k=1:maxt end % -- For nodes away from the boundares for =(nx+1):n-nx x() = (b() + as()*x(-nx) + aw()*x(-1)... + an()*x(+nx) + ae()*x(+1) ) / ap(); end Relaxaton Soluton to Lnear Systems page 20
Gauss-Sedel Iteraton (3) Consder the sweep through the nodal values n order of ncreasng : x (k+1) 1 = 1 ( b 1 a 11 n j=2 ) a 1j x (k) j x (k+1) 2 = 1 ( b 2 a 21 x (k+1) 1 a 22 n j=3 ) a 1j x (k) j x (k+1) 3 = 1 ( b 3 a 31 x (k+1) 1 a 32 x (k+1) 2 a 33 n j=4 ) a 1j x (k) j. x (k+1) = 1 ( b a 1 j=1 a j x (k+1) n j=+1 ) a 1j x (k) j (3) Relaxaton Soluton to Lnear Systems page 21
Gauss-Sedel Iteraton (4) Coeffcents that multply new values A x = b new values old values = Coeffcents that multply old values Agan, let and defne D = dag(a 11,..., a nn ) L = lower trangular part of A U = upper trangular part of A Then Gauss-Sedel teratons correspond to the splttng of A A = D L U Note that A LU,.e.,, L and U are not the usual factors of A. Relaxaton Soluton to Lnear Systems page 22
Equaton (3) can be rewrtten as Gauss-Sedel Iteraton (5) a x (k+1) = b 1 j=1 a j x (k+1) n j=+1 a 1j x (k) j or or a x (k+1) 1 j=1 a j x (k+1) = b n j=+1 a 1j x (k) j Dx (k+1) Lx (k+1) = b + Ux (k) = (D L)x (k+1) = Ux (k) + b or x (k+1) = (D L) 1 (Ux (k) + b) (4) Relaxaton Soluton to Lnear Systems page 23
Gauss-Sedel Iteraton (6) Note: The order n whch the nodes are processed n the Gauss-Sedel teratons wll affect the path toward the soluton. In other words, for two dfferent orderngs of the nodes, the ntermedate terates of x wl be dfferent. The Jacob teraton does not depend on the order n whch the nodes are numbered. Much of the classcal theory of relaxaton methods assumes that the nodes and numbered n natural order. Relaxaton Soluton to Lnear Systems page 24
SOR: Successve Over-Relaxaton (1) Let ˆx (k+1) be the updated value of x from the Gauss-Sedel formula, vz. ˆx (k+1) = 1 ( b a j< a j x (k+1) j j> ) a j x (k) j (5) Instead of takng ˆx (k+1) as the value of x at the k + 1 step, use x (k+1) = ωˆx (k+1) + (1 ω)x (k) (6) where ω s a scalar weghtng factor. The basc step n Equaton (5) s called relaxaton. ω = 1 for Gauss-Sedel ω < 1 causes the teratons to more slowly move toward the soluton. Ths s called under-relaxaton. ω > 1 causes the teratons to accelerate,.e. to be more aggressve. Ths s called over-relaxaton. Relaxaton Soluton to Lnear Systems page 25
Relaxaton as an update We can rewrte Equaton (6) as x (k+1) = x (k) + ω ( ˆx (k+1) ) x (k) (7) or where x (k+1) x (k+1) = x (k) + ω x (k+1) (8) = ˆx (k+1) x (k) x (k+1) s the update to x n teraton k. x (k+1) ω x (k+1) s the drecton n whch the soluton s changng. s a magnfed (ω > 1) or reduced (ω < 1) change to the soluton. Relaxaton Soluton to Lnear Systems page 26
SOR: Successve Over-Relaxaton (2) Substtute Equaton (6) nto Equaton (5) x (k+1) = (1 ω)x k + ω a ( b j< a j x (k+1) j j> ) a j x (k) j (9) Multply through by a and rearrange to get a x (k+1) + ω j< a j x (k+1) j = (1 ω)a x k ω j> a j x (k) j + ωb Use the splttng A = D L U and the precedng equaton becomes Dx (k+1) ωlx (k+1) = (1 ω)dx (k) + ωux (k) + ωb (D ωl)x (k+1) = [ (1 ω)dx (k) + ωu ] x (k) + ωb x (k+1) = (D ωl) 1[ (1 ω)dx (k) + ωu ] x (k) + ω(d ωl) 1 b Relaxaton Soluton to Lnear Systems page 27
SOR: Successve Over-Relaxaton (3) The update equaton s x (k+1) = (D ωl) 1[ (1 ω)dx (k) + ωu ] x (k) + ω(d ωl) 1 b Thus where x (k+1) = Hx (k) + d H = (D ωl) 1[ (1 ω)dx (k) + ωu ] d = ω(d ωl) 1 b The speed of convergence depends on the egenvalues of H. Relaxaton Soluton to Lnear Systems page 28
On a coarse 8 8 mesh Compare Relaxaton Convergence 10 4 Jacob Gauss Sedel SOR 10 3 r 1 10 2 10 1 10 0 0 20 40 60 80 100 120 Iteraton Relaxaton Soluton to Lnear Systems page 29
On a slghtly fner 16 16 mesh Compare Relaxaton Convergence 10 5 Jacob Gauss Sedel SOR 10 4 r 1 10 3 10 2 10 1 0 50 100 150 200 250 300 350 400 Iteraton Relaxaton Soluton to Lnear Systems page 30
Summary of Convergence for Relaxaton SOR s better than Gauss-Sedel Gauss-Sedel s better than Jacob All methods converge more slowly as the number of unknowns ncreases = Relaxaton methods don t work on fne meshes Multgrd explots ths dea Use relaxaton on multple levels Transfer soluton from fne to coarse levels and back agan Use multgrd as an nner teratve solver to accelerate solutons on fne meshes Relaxaton Soluton to Lnear Systems page 31