Fixed-Point Iterations, Krylov Spaces, and Krylov Methods

Transcription

1 Fxed-Pont Iteratons, Krylov Spaces, and Krylov Methods Fxed-Pont Iteratons Solve nonsngular lnear syste: Ax = b (soluton ˆx = A b) Solve an approxate, but spler syste: Mx = b x = M b Iprove the soluton usng the resdual: r = b Ax (teratve refneent) Error, e = xˆ x, satsfes Ae = b Ax = r Don t copute exact error, nstead solve Mz = r and set x = x + z Iterate: = b Ax = b A( x + z ) = r Az z M = r (solve Mz = r ) x x z + Methods: Jacob teraton (dagonal), Gauss-Sedel (upper trangular), any others such as (S)SOR,...

2 Fxed-Pont Iteratons Convergence of such teratons? Let A= M N (atrx splttng). Then Mz = b Ax Mz = b Mx + Nx Mx = Nx + b + (fxed-pont teraton x = M Nx + M b) + Note that the fxed-pont s the soluton (proof?) Error: e = xˆ x = M Nxˆ+ M b M Nx M b = M N x x = M Ne = ( ˆ ) ( M N) e Resdual: r = ( NM ) r and ( ) ( M r M N M r ) = (proof?) To analyze convergence we need to ntroduce/revew a nuber of concepts 3 Rate of Convergence Let ˆx be the soluton of Ax = b, and we have terates x, x, x, { x } converges (q-)lnearly to ˆx f there are N and c [,) such that for N: x xˆ c x xˆ, + { x } converges (q-)superlnearly to ˆx f there are N and a sequence { c } that converges to such that for N: x xˆ c x xˆ + { x } converges to ˆx wth (q-)order at least p f there are p >, c, and N such that N: x x ˆ c x x ˆ p (quadratc f p =, cubc + f p = 3, and so on) { x } converges to ˆx wth j-step (q-)order at least p f there are a fxed nteger j, p >, c, and N, such that N: x xˆ c x xˆ p + j 4

3 Nors A nor on a vector space V s any functon f : V such that. f ( x) and f ( x) = x =,. f ( x) = f ( x), 3. f ( x + y) f ( x) + f ( y), where x V and. n n n n Iportant vector spaces n ths course:,, and, (atrces). Note that the set of all -by-n atrces (real or coplex) s a vector space. Many atrx nors possess the subultplcatve or consstency property: n f AB f A f B for all A and B (or real atrces). ( ) ( ) ( ) Note that strctly speang ths s a property of a faly of nors, because n general each f s defned on a dfferent vector space. 5 Nors We can defne a atrx nor usng a vector nor (an nduced atrx nor): Ax A = ax = ax Ax x x x = Induced nors are always consstent (satsfy consstency property). Two nors. and. are equvalent f there exst postve, real constants a β and b such that x : a x x b x β The constants depend on the two nors but not on x. All nors on a fnte densonal vector space are equvalent. 6 3

4 Nors Soe useful nors on n n,, n p p = x p = n, n : p-nors: x, especally p =,,, where x ax = x. Induced atrx p-nors are: A A A n = ax a (ax absolute colun su) j ( A) = j = σ (ax sngular value harder to copute than others) ax = n ax a (ax absolute row su) j= j Matrx Frobenus nor: n A = a F j, = j (slar to vector -nor for a atrx) All these nors are consstent (satsfy the subultplcatve property) 7 Egenvalues and Egenvectors Let Ax = λx and ya = λy (for sae λ). We call the colun vector x a (rght) egenvector, the row vector y a left egenvector, and λ an egenvalue, the trple together s called an egentrple, and,x λ,y a (rght) egenpar or left egenpar. ( λ ) and ( ) The set of all egenvalues of A, Λ ( A), s called the spectru of A. If the atrx A s dagonalzable (has a coplete set of egenvectors) we have A= VΛV AV = VΛ, where V s a atrx wth the rght egenvectors as coluns and Λ s a dagonal atrx wth the egenvalues as coeffcents. A slar decoposton can be gven for the left egenvectors. 8 4

5 Spectral Radus The spectral radus ρ ( A) s defned as ρ( A) ax { λ : λ Λ( A) } Theore: For all A and ε > a consstent nor. So, f ρ ( A) <, then a consstent nor. Tae ε ( ρ( A) ) Defne A = and apply theore above. = A (coplex conjugate transpose). T If A s Hertan ( A If A s noral ( AA = A ), then ( ) ρ A = A. = A A),then ρ ( A) = A. =. exsts such that A ρ( A) exsts such that A <. + ε. 9 Fxed-Pont Iteratons Under what condtons does e and x xˆ (convergence) for arbtrary e? Theore: ( ) e ff ( M N) e = M N e for arbtrary. Proof: Let G = M N and the atrx nor. be nduced by a vector nor.. Assue G Then G G and Ge G e for any e.. Assue Ge for all e. Consder the dentty atrx I = η η η n. GI= G η Gη Gη n ; so G snce GI= G. Alternatvely, consder G η G η G η n ( note that G η ) 5

6 Nors Note that we can generalze the result for the one-nor to all nors by usng the equvalence of nors on fnte densonal vector spaces. Slarly, the results are readly generalzed for nconsstent atrx nors (wth AB > A B possble), by usng the equvalence of nors on fnte densonal spaces. Fxed-Pont Iteratons Theore: G ff ( G) ρ <. Proof:. G ρ ( G) <. For each egenvalue λ of G there exsts at least one egenvector v s.t. Gv = λv. Then Gv v = λ = λ v and Gv. So, λ v λ <. Snce ths holds for each egenvalue, ρ ( G). ( G) G < ust hold. ρ <. There exsts a consstent nor. s.t. G <. Hence, G G. Therefore G G. So e ( x xˆ ) for arbtrary e ff ρ ( M N) <. 6

7 Krylov Spaces Gven x, set r = b Ax. For =,,, z M = r, x = x + z, + r = b Ax = r Az. + + Note that x x = z = M r and hence x x = z + z + + z. + + Ths ples = + + ( ) + + ( ) x x M r M N M r M N M r. So, correcton x x s gven by polynoal S + () t = + t + t + + t : x x = ( M N ) M r = S ( M N ) M r. + = Note also e = ( M N) e and ( ) ( ) r = NM r = M M N M r. 3 Slarty Transforaton Let A have egenpars (, v ) Av = λ v λ : Defne the slarty transforaton: BAB The atrx BAB ( ) has the sae egenvalues, λ, as A and egenvectors Bv : BAB Bv = BAv = λ Bv Show that ( M N) has the sae egenvalues as ( ) NM. 4 7

8 Polynoals and Spaces Man coputatonal cost s n the ultplcaton by A and solvng for M. So, we can try to generate better polynoals (faster convergence) at sae cost. { } Also correcton span,( ),, + ( ) We call a space K ( B, y) span { y, By, B y,, B y } x x M r M N M r M N M r the Krylov (sub)space of denson assocated wth B and y. So, x x K ( M N, M r ) e ( M N ) e K ( M = N, e + ) and M ( r M N ) M r K ( M N, M r + ) =. Therefore, alternatvely we can copute better approxate solutons fro the sae space (faster convergence) at sae cost. 5 Krylov Spaces So, a Krylov space s a space of polynoals n a atrx tes a vector. These spaces nhert the any portant approxaton propertes that polynoals on the real lne or n the coplex plane possess. For splcty let the atrx B be dagonalzable, B = VΛV. Then B = VΛV VΛV = VΛV and generally B = VΛ V. So, the polynoal p () t = + t + + t appled to B gves ( ) = ( + Λ+ Λ + + Λ ) ( ) = ( Λ) = dag ( ( λ ),, ( λ )) p B V I V and hence n p B Vp V V p p V So, the polynoal s appled to the egenvalues ndvdually. Ths allows us to approxate solutons to lnear systes, egenvalue probles, and ore general probles usng polynoal approxaton. 6 8

9 Approxaton by Matrx Polynoals Let B = VΛV, let Λ( B) Ω. If p () t for all t Ω, then p ( B ) B. t ζ Let y = Vζ. Then p ( B) y = v p ( λ) ζ v B y = λ Furtherore, let ε and λ λ > δ (for soe egenvalue λ ) j ε, t Ω and t λ > δ, If p () t =, t = λ, p B y vζ. then ( ) If we can construct such polynoals for odest we have an effcent lnear solver or egensolver. 7 Krylov Spaces We have M N = M ( M A) = I M A. So, ( ) ( x x I M A M r K M A, M r + + ) =, and x M Nx M b xˆ= I M A xˆ+ M b M Axˆ= M b. = +, + wth fxed-pont ( ) So, we solve the precondtoned proble M Ax = M b. Precondtonng as to prove the convergence of Rchardson s teraton x = ( I A ) x + b + However, wth A = M A, M = I, and b = M b our teraton becoes Rchardson s teraton, and we have Krylov spaces and polynoals based on A. Hence, for splcty we consder A and b as an explctly precondtoned atrx and vector, and wor wth Krylov spaces n A and r (ost of the te). 8 9

10 Approxatons fro Krylov Spaces Rchardson for Ax = b sees update z K ( I A, r ) = K ( A, r ) How to defne an teraton that fnds better approxaton n sae space? Gven x and r = b Ax fnd z K ( A, r ) and set x = x + z. There are several possbltes. Two partcularly portant ones are. Fnd z K ( A, r ). Fnd z K ( A, r ) such that r = r Az s nal. such that e = xˆ ( x + z ) s nal. The second one sees hard, but s possble n practce for specal nors. Further possbltes for optal solutons exst, and for non-hertan atrces certan non-optal solutons turn out to have advantages as well. 9 Inner Products Many ethods to select z fro the Krylov space are related to projectons. We call f : S S an nner product over the real vector space S, f for all vectors xyz,, and scalars,. f (, xx) and f (, xx) = x=. f ( xz, ) = fxz (, ) 3. f ( x + y, z) = f( x, z) + f( y, z) 4. f (, xz) = fzx (, ) For a coplex nner product, f : S S, over a coplex vector space S we have nstead of property (4): f (, xz) = fzx (, ). Inner products are often wrtten as, xy, (, ) xy, or xy,, etc.. We say x and y are orthogonal (w.r.t -IP), x y f xy, =.

11 Inner products and Nors Each nner product defnes, or nduces, a nor: x = x, x. (proof?) Many nors are nduced by nner products, but not all. Those nors that are have addtonal nce propertes (that we ll dscuss soon). An nner product and ts nduced nor satsfy: xy, x y (CS neq) A nor nduced by an nner product satsfes the parallelogra equalty: x + y + x y = x + y ( ) In ths case we can fnd the nner product fro the nor as well: Real case: xy, = ( x+ y x y ) 4 Coplex case: Re xy, = ( x+ y x y ) I xy, = x+ y x y 4 4, ( ) Mnu Resdual Solutons Frst, we consder nzng the -nor of the resdual: Fnd z K ( A, r ) such that r = r Az s nal. n The vector -nor s nduced by the Eucldean nner product yx= yx. = It aes sense to nze the resdual, because the error s n general unnown, and we can only drectly nze specal nors of the error (those that don t requre the error ). Moreover, e A = r A r, so the nor of the error s bounded by a constant tes the nor of the resdual. Fnally, note that r = e Ae (show ths s a nor f A regular) AA Theore: z s the nzer ff r = b A( x + z ) K ( A, Ar ). Note that b A( x + z ) = r Az.

12 Mnu Resdual Solutons Proof: Let f () z = r Az. Then zˆ K ( A, r ): r Azˆ n, ples ẑ s a statonary pont of f () z : For any unt vector (, ) f () zˆ p K A r we have =. p f ( zˆ+ εp) f( zˆ) So, l =, whch gves ε, ε ε r Azˆ ε Ap r Azˆ l = ε ε εp A ( r Azˆ) ε( r Azˆ) Ap + ε p A Ap l = ε ε p A ( r Azˆ) ( r Azˆ) Ap = for any unt vector p K ( A, r ). So, ( r Azˆ) Ap = ( r Azˆ) Ap for any unt p K ( A, r ) (why?) Snce Ap K ( A, Ar ) we have ( r Az ˆ) K ( A, Ar ) AK ( A, r ). 3 Mnu Resdual Solutons Mnzng a nor (cont. functon) s n general coplcated. However, the orthogonalty condtons lead naturally to a lnear syste of equatons. Let { w, w,, w } for a bass for K ( A, r ) and { Aw } for (, ) = K A Ar. Let z = W ζ (for soe unnown ζ ). Now the orthogonalty condtons Aw ( r AW ζ) yeld the lnear equatons ( waaw ) ζ = war. = j j j In atrx for: WAAWζ = WAr noral equatons (accuracy probles) LS proble: n r K ζ where K = AW ζ More accurate to solve LS proble usng QR-decoposton. Solvng for ζ requres only the soluton of an syste ndependent of n. 4

13 Mnu Resdual Solutons We have seen that n r K ζ K ( r K ζ) ζ Copute K = Q R (QR-decoposton) where n n n s.t. QQ = I and uppertrangular Q If ran( K ) ( ) =, then R s nonsngular and range(q ) = range( K ) Now Q ( r K ζ) gves Q r Q Q R ζ = R ζ = Q r (easy solve) Note that K ζ = Q Q r r K ζ = r Q Q r K ζ s the orthogonal projecton of r onto range ( K ) (, ) = K A Ar. r r K ζ R = s the orthogonal projecton of (, ). r onto ( K A Ar ) QQ and I Q Q are orthogonal projectors. P s projector f P = P, orthogonal projector f R(P) N(P) P = P. 5 Mnu Resdual Solutons Iteraton-wse the proble s solved n four steps:. Extend the Krylov spaces K ( A, r ) and K ( A, Ar ) by addng the respectve + next vectors Ar and A r (only atvec). Update orthogonal bass for K ( A, Ar ) : QR-decop. of K 3. Update projected atrx and projecton of r (orthog) onto K ( A, Ar ) 4. Solve the projected proble, e.g. Rζ = Q r. Note that ths proble s only or ( + ) rrespectve of the sze of the lnear syste. These steps vary soewhat for dfferent ethods. We would le to carry out these steps effcently. The GCR ethod (Generalzed Conjugate Resduals) llustrates these steps well. (Esenstat, Elan, and Schulz 983) 6 3

14 Mnu Resdual Solutons: GCR GCR: Ax = b Choose x (e.g. x = ) and tolerance ε; set r = b Ax ; = whle r ε do = + r adds search vector to K ( A, r ) u = r ; c = Au Ar extends K ( A, Ar ) for j =,, do (start QR decoposton) u = u u c c Orthogonalze c aganst prevous c j j j and c = c c c c update u such that Au = c antaned j j end do u = u / c ; c = c / c Noralze; (end QR decoposton) x = x + uc r Project new c out of resdual and update r = r cc r soluton accordngly; note r c for j j end do What happens f c r 7 Mnu Resdual Solutons: GCR Fro the algorth we see that f cr for j = : j j u = νr span{ r } = K ( A, r ), ν s a noralzaton constant c = νar span{ Ar } K ( A, Ar ). r = r c = r νar K ( A, r ) also Ar span{ r, r } By nducton (and the stateents above) we can show u = r β u span{ r,, r } K ( A, r ) = j< j j c = Ar β c span{ Ar,, Ar } K ( A, Ar ) = and also that j< j j c = Au, c c, c,, c, and cc = (last by constructon) r = r c span{ r, Ar, Ar,, Ar } = K ( A, r ) + These relatons hold even f { r, Ar,, A r } are dependent for soe. In that case all the spaces have a axu denson of

15 Mnu Resdual Solutons: GCR Theore: Let cr for =,,,. Then b Ax for GCR s nal. We use our earler theore on the nu resdual. We have (prevous slde) x = x + u β = x + z and z K ( A, r ) as requred. = Now we only need to show that r K ( A, Ar ). Note that our assupton ples r for =,,,. By constructon we have r = r ccr cr = cr cccr = Assue that for j = we have r c, c,, c (nduc. hypo.). j j Fro r = r c c r we have r c. j j j j j j j For < j, cr = cr j j = (fro orthogonalty c and nducton hypothess). Ths proves the requred orthogonalty result snce the c span K ( A, Ar ). 9 Mnu Resdual Solutons: GCR Recaptulaton of GCR after t.s: r = n{ r Az z = U ζ} u K ( A, r ) (, ), r K ( A, r ) for =,, & = for r +., c K A Ar Ths ples that K ( A, r ) = span{ r,, r} for =,, (f cr =/ ). + Let U = [ u u u ] and C = [ c c c ] Then AU = C, C C = I, and range( U ) = K ( A, r ). For GCR the projected syste s the atrx for the noral equatons (but coputed plctly): U A AU = ( AU ) ( AU ) = C C = I. The projected rght hand sde s C r, and so we have ζ = C r. z = U ζ s gven by condton C ( r AU ζ) =, whch gves ζ = C r. Ths gves z = U C r, x x z = +, and r = r C C r. 3 5

16 Mnu Resdual Solutons: GCR Note that C C and I C C are both orthogonal projectors. C C s a projecton snce C C C C = C C. C C s an orthogonal projecton snce C C s Hertan. I C C s a projecton snce I C C s Hertan. ( I C C )( I C C ) = I C C C C + C C = I C C. I C C s an orthogonal projecton snce 3 Model Probles Dscretze ( ) ( ) pu qu + ru + su + tu = f. x x y y x y D (,j+) C (-,j) V (,j) (+,j) A (,j-) B Integrate equalty over box V. Use Gauss dvergence theore to get pu ( ) ( ) x pu + qu dxdy = n ds V x x y y V qu y And approxate the lne ntegral nuercally. 3 6

17 Model Probles Now we approxate the boundary ntegral pu x nds V qu y. We approxate the ntegrals over each sde of box V usng the dpont rule and we approxate the dervatves usng central dfferences. C Δy pu n dy p ( U U ) and so on for the other sdes B x /, j, j, j Δ x + + We approxate the ntegrals over ru, su, tu, and f usng the area of the box x y and the value at the dpont of the box, where we use central dfferences for dervatves. So, u ( U U,, )/( Δ x x + j j ), and so on. For varous exaples we wll also do ths whle strong convecton relatve to the esh sze aes central dfferences a poor choce (as t gves nterestng systes). 33 Model probles Ths gves the dscrete equatons Δy p U U p U U Δx Δx q U U p U U ( ) ( ) +, j +, j j,, j j,, j j, + ( j, + j, ) j, ( j, j, ) Δy ( Δy /) r ( U U ) ( Δx /) s ( U U ) + + j, +, j, j j, j, + j, + ΔxΔyt U = ΔxΔyf j, j, j, Often we dvde ths result agan by ΔΔ. x y 34 7

18 Mnu Resdual Solutons: GMRES An alternatve s to generate teraton-wse an orthogonal bass for K ( A, r ) The Arnold algorth (teraton) goes as follows: v = r r ; Let for =, v Av ; = + for j =, h = v v j, j + ; v = v h v ; + + j, j end h = v ; v = v h ; +, , end Note/show the followng results: AV = V H (Arnold recurrence) V V = I (orthogonal), H = V AV (upper Hessenberg) Mnu Resdual Solutons: GMRES { } Usng AV V H + as follows. Let z = V ζ, and nze r AV ζ over all -vectors ζ. Note that ths s an n least squares proble (as before). =, we solve n r Az z K ( A, r ) Now substtute r = V η r and + AV = V H. Ths gves + ( ) V η r V H ζ = V η r H ζ = η r H ζ The latter s a sall ( ) + least squares proble we can solve by standard dense lnear algebra technques (e.g. usng LAPACK) We can explot the structure of H and the least squares proble to. do ths effcently,. copute the resdual nor wthout coputng the resdual 36 8

19 GMRES By constructon H has the followng structure H = h, h, h,3 h, h, h, h, h,3 h, h, h 3, h 3,3 h 4,3 h, h, h, h, h +, (Upper Hessenberg) Cheapest QR decop. s by Gvens rotatons to zero lower dagonal. G H H = c s s c I = & & & & & h 3, h 3, GMRES Next step we copute: G H G H H = c s s c I & & & & & & & h 3, h 3,3 h 3, h 4,3 h 4, = & & & & & & & & & h 4,3 h,3 After Gvens rotatons: G H G H H = Q H + H = r, r, r, r 3,3 r, = R 6-7 9/4/7 / :5 AM

20 GMRES Theore: An unreduced ( + ) % Hessenberg atrx s nonsngular. (unreduced eans no zeros on subdagonal) Proof:? GMRES So the least squares proble y = arg n e ær æ H y : y Š can be solved by ultplyng H y l e ær æ fro left by R H Q : y = R Q H e ær æ In practce: Stepwse copute G H (G H G H H ) and G H (G H G H e ær æ ) In Arnold step, update H wth new colun; then carry out prevous Gvens rotatons on new colun. Copute new Gvens rotaton and update H and rght hand sde (of sall least squares proble): G H (G H G H e ær æ ) 8-9 9/4/7 / :5 AM

21 GMRES The least squares syste now loos le R y = Q H + e ær æ. We ay assue has no zeros on dagonal (see later) R Snce botto row of R s zero we can only solve for (Q H + e ær æ ) (frst coeff.s) Ths s exactly what we do by solvng R y = Q H e ær æ Note LS resdual nor equals the nor of the actual resdual: ær æ = q + H e ær æ ( q + snce t changes wth ): ær AV yæ = V + e ær æ V + H y = e ær æ H y Ths way we can ontor convergence wthout actually coputng updates to soluton and resdual (cheap). GMRES GMRES: Ax = b CHOOSE x (E.G. x = ) AND tol r = b Ax ; = ; v = r /ær æ ; WHILE ær æ > tol = + ; ṽ + = Av ; FOR j = :, h j, = v j H ṽ + ; ṽ + = ṽ + h j, v ; END h +, = æṽ + æ ; v + = ṽ + /h +, ; UPDATE QR-DECOMP.: H = Q + R ær æ = q + H e ær æ END y = R Q H e ær æ ; x = x + V y ; r = r V + H y = V + I Q Q H e ær æ ; (or sply r = b Ax ) 3-3 9/4/7 / :5 AM

22 Mnu Resdual Solutons: GMRES GMRES: Ax = b Choose x, tolerance ε; set r = b Ax ; v = r r, =. whle r ε do = + v Av ; = + for j =, h = v v ; j, j + v = v h v ; + + j, j end h = v ; v = v h ; +, , Solve LS n η r H ζ ζ ( = r ) by constructon (actually we update the soluton rather than solve fro scratch see later) end x = x + V ζ ; r = r V H ζ = V η r H ζ r b Ax or sply = ( ) Convergence Restarted GCR Test proble on unt square: Interor: grd ponts = Boundary u = for x = and y = u = elsewhere ( u) Resdual Nor vs Nuber of Iteratons log r full GCR GCR() -8 GCR(5) GCR() GCR(5) GCR() Iteraton count.5 3 x

23 Convergence restarted GMRES Test proble on unt square: grd ponts Interor: ( u) = Boundary u = for x = and y = u = elsewhere log r GMRES() full GMRES GMRES(5) GMRES() GMRES() GMRES(5) log r GMRES() full GMRES GMRES(5) GMRES() GMRES() GMRES(5) x 4 Iteraton count Iteraton count 39 GMRES vs GCR GMRES() x unnowns te (s) teratons log( r / b ) full rgcr() x unnowns te (s) teratons log( r / b ) full

24 Conjugate Gradents () Hertan atrces: Error nzaton n the A-nor We are solvng Ax = bwth ntal guess x d r = b Ax and xˆ s the soluton to Ax = b. The error at teraton s = xˆ (x + z ), where z K (A, r ) s the th update to the ntal guess. Theore: Let Abe Hertan, then the vector z c K (A, r ) satsfes z = arg n{æxˆ (x + z)æ A : z c K (A,r )} ff r h r Az satsfes r Ω K (A, r ). The ost portant algorth of ths class s the Conjugate Gradent Algorth. Conjugate Gradents () Proof: z = arg n{æxˆ (x + z)æ A : z K (A, r )} w (xˆ x ) z Ω A K (A, r ) We now. K (A,r )=span r, r,, r Ths gves r Ω A (xˆ x z ) for =,, g A(xˆ x z ), r for =,, b Ax Az, r for =,, r Az, r for =,, r,r for =,, r ΩK (A, r ) g g g g 9/4/7 / :8 AM

25 Conjugate Gradents (3) So we can nze the error by choosng the update such that the new resdual s orthogonal to all prevous resduals. Hence, the nae orthogonal resdual ethods. (note the coparson between Orthon() and Steepest Descent) We can generate an orthogonal bass for the Krylov subspace usng the Lanczos teraton, the 3-ter recurrence verson of the Arnold-teraton. Conjugate Gradents (4) Lanczos teraton: CHOOSE q ; = ; q = ; FOR =,, DO q + = Aq ; = Aq,q ; q + = q + q ; q + = q + q ; = æq +æ ; q + = q +/ ; END Show q + = q + q sets q +Ωq. (one arguent s the syetry of the Hessenberg atrx for Arnold, gve another) Ths algorth generates the recurrence relaton: T AQ = Q T + q + e, where Q = [q q q ], T =. 9/4/7 / :8 AM

26 Conjugate Gradents (5) Use Lanczos orthonoral bass for nzng A-nor of error. z = arg n{æxˆ (x + z)æ A : z K (A, r )} ff r h r Az satsfes r Ω K (A, r ). q = r /ær æ ; Lanczos ethod: AQ = Q T + q + e T Solve r AQ y Ω Q w Q H (ær æ q AQ y )= w Q H (ær æ q AQ y )= w ær æ e Q H AQ y =. Notce. range(q )=span{r, r,, r } AQ = Q T + q + e T u Q H AQ = T So we reduced our proble to solvng ær æ e T y = : y = T e ær æ Conjugate Gradents (6) In order to update step-by-step we use sae trc as n MINRES: Let H T = L D L ; then y = L H D L e ær æ, where L s unt lower b-dagonal wth lower dagonal coeff.s, l, l,, l (ndex gves the colun) Change of varables: and : P = Q L H ŷ = D L e ær æ Q y = P ŷ Each teraton only the last coponent of ŷ changes. Fro P L H = Q we get a recurrence for p : p + l p = q (p = q ) So every new step we copute a new q +, we update the decoposton of T and fro that ŷ + and p +. x = x + p ŷ, (where s th cop of vector ) r = r Ap ŷ, = q + ŷ, ŷ, ŷ 9/4/7 / :8 AM

27 Conjugate Gradents (7) Ths leads to the coupled two-ter recurrence for of CG CG algorth: Ax = b x t r = b Ax p = r ; = CHOOSE ; WHILE END ær æ > tol = + ; = r,r p,ap ; x = x + p ; r = r Ap ; = r,r r,r ; p = r p ; DO Conjugate Gradents log r - -4 GMRES CG # teratons (atvecs) 9/4/7 / :8 AM