STAT 309: MATHEMATICAL COMPUTATIONS I FALL 08 LECTURE 7. sor method remnder: n coordnatewse form, Jacob method s = [ b a x (k) a and Gauss Sedel method s = [ b a = = remnder: n matrx form, Jacob method s and Gauss Sedel s a =+ =+ D = b (L + U)x (k) (D + L) = b Ux (k) a x (k) a x (k) another general adage n numercal computatons s: don t dscard prevous nformaton, try to use t too applyng ths to Gauss Sedel, we could try to use both and x (k) for =,..., to obtan ths yelds the method of successve over relaxaton (sor) ths s gven by the teraton = ω [b a a x (k) + ( ω)x (k) (.) a = =+ the quantty ω s called the relaxaton parameter f ω =, then the sor method reduces to the Gauss Sedel method,.e., = [ b a a x (k) a = the name over relaxaton comes from choosng ω > n matrx form, the teraton can be wrtten as =+ D = ω(b L Ux (k) ) + ( ω)dx (k) whch can be rearranged to obtan or = (D + ωl) = ωb + [( ω)d ωux (k) ( ) [( ) ( ) ω D + L ω D U x (k) + ω D + L b (.) Date: December, 08, verson.. Comments, bug reports: lekheng@galton.uchcago.edu.
the teraton matrx s B ω = ( ) [( ) ω D + L ω D U snce B = B GS, f we pck some ω such that ρ(b ω ) < ρ(b GS ), we would mprove the convergence of Gauss Sedel so sor s at least as fast as Gauss Sedel and often faster n fact, for certan types of matrces, one can pck ω so that ρ(b ω ) s mnmzed for example, f A s () a nonsngular matrx, () ts Jacob teraton matrx B J has only real egenvalues, and () A may be permuted nto the form [ D B A = Π Π B D where Π, Π are permutaton matrces and D, D are dagonal matrces, then the optmal elaxaton parameter s gven by ω opt = + + ρ(b J ) and note that f Ax = b, then ρ(b ωopt ) = + ρ(b J ) + + ρ(b J ) Dx = ω(b Lx Ux) + ( ω)dx and so ( ) [( ) ( ) x = ω D + L ω D U x + ω D + L b (.3) subtractng (.3) from (.), we get note that e (k+) = B ω e (k) ( ) [( det B ω = det ω D + L det = det ( det ω D + L) ω n = n = a = ( ω) n [( ω ( ω) n n = a ω n ) ω ) D U D U therefore n = λ = ( ω) n where λ,..., λ n are the egenvalues of B ω, wth λ λ n hence we get λ n ( ω) n snce we must also have λ = ρ(b ω ) < for convergence t follows that a necessary condton for convergence of sor s 0 < ω <
f A s symmetrc postve defnte, then the condton 0 < ω < s also suffcent a result of Ostrowsk mples that for such an A, ρ(b ω ) < ff 0 < ω < suppose A R n n s a symmetrc matrx, then U = L T and f we set M ω = ω ( ) ( ) ω ω ω D + L D D + LT and defne our teraton as = x (k) M ω (Ax (k) b) ths s called the method of symmetrc successve over relaxaton (ssor) there are yet other varants of sor such as: block sor or block ssor when the matrx has a block structure; lkewse, we may also ntroduce block Jacob or block Gauss Sedel applyng the sor extrapolaton to Jacob method to get = ω [b a x (k) a = =+ a x (k) + ( ω)x (k) (.4) to preserve parallelsm; ths s often called or note the dfference between (.4) and (.) and note that when ω = n (.4), then the or method reduces to Jacob method one may even defne a nonlnear verson of sor for teratons of the form = f(x (k) ) where f s some nonlnear functon f : R n R n : sor = ( ω)x (k) sor + ωf(x (k) sor). Rchardson method unlke the splttng methods n the prevous lecture, the teratve methods here do not requre splttng A nto a sum of two matrces but they are a bt lke sor n that there s a scalar parameter nvolved at each step ths scalar parameter can ether be fxed throughout (e.g., Rchardson) or can vary from one nteraton to the next (e.g., steepest descent, Chebyshev) or there can even be two scalar parameters at each step (e.g., conugate gradent) the smplest one s known as the Rchardson method, where the teraton s smply = (I αa)x (k) + αb (.) = x (k) + α(b Ax (k) ) = x (k) + αr (k) where r (k) := b Ax (k) s the resdual at the kth step note that f x = A b, then we trvally have x = (I αa)x + αb (.) as usual we defne the error e (k) = x x (k), then subtractng (.) from (.) yelds e (k+) = B α e (k) where the teraton matrx s B α = I αa we want to choose the parameter α > 0 a pror so as to mnmze ρ(b α ) suppose A s symmetrc postve defnte, wth egenvalues µ µ µ n > 0 snce B α = I αa, we have λ = αµ for =,..., n 3
f we want α so that ρ(b α ) s mnmzed,.e., mn α max λ (α) = mn max αµ n α = mn max( αµ, αµ n ), n α the optmal parameter α s attaned when and snce these must dffer by a sgn, α µ = α µ n α µ n = ( α µ ), whch yelds α = µ + µ n note that when αµ =, the teraton dverges for some choce of x (0) hence the method converges for 0 < α < µ however ths teraton s senstve to perturbaton and therefore bad numercally for example, f µ = 0 and µ n = 0 4, then the optmal α s /(0 + 0 4 ), but ths s close to a value of α for whch the teraton dverges, α = /0 also, note that and smlarly, therefore λ (α ) = µ + µ n µ = µ n µ µ + µ n = κ(a) + κ(a) 0, λ n (α ) = µ µ n µ + µ n = µ /µ n µ /µ n + = κ(a) κ(a) + 0 ρ(b α ) = κ(a) κ(a) + and we see that the convergence rate depends on κ(a) 3. rate of convergence and spectral radus we kept referrng to the spectral radus of the teraton matrx ρ(b) as the rate of convergence but f you thnk about t, snce e (k+) B e (k), t should be B that controls the convergence rate of course f the teraton matrx B s symmetrc, then t makes no dfference snce B = ρ(b) but n general they are not equal, n fact s possble for ρ(b) = 0 and B 0 the reason s because we look at the average rate of convergence for any teraton matrx B C n n (not necessarly symmetrc) and any consstent norm (not necessarly submultplcatve), after k steps, we get the average reducton n error per step after k steps s then ( ) /k e (k) e (0) e (k) B k e (0) (3.) 4
and by (3.) s bounded by B k /k the spectral radus then drops out when we take lmts lm k Bk /k = ρ(b), whch holds for any consstent norm (.e., satsfes Ax A x for any A C n n and any x C n ) 5