Scientific Computing II

Transcription

1 Scientific Computing II Conjugate Gadient Methods Michael Bade Summe 2014 Conjugate Gadient Methods, Summe

2 Families of Iteative Solves elaxation methods: Jacobi-, Gauss-Seidel-Relaxation,... Ove-Relaxation-Methods Kylov methods: Steepest Descent, Conjugate Gadient,... GMRES,... Multilevel/Multigid methods, Domain Decomposition,... Conjugate Gadient Methods, Summe

3 Remembe: The Residual Equation fo Ax = b, we defined the esidual as: and the eo: e i) := x x i) leads to the esidual equation: i) = b Ax i) Ae i) = i) elaxation methods: solve a modified easie) SLE: B ê i) = i) whee B A multigid methods: coase-gid coection on esidual equation A H e i) H = i) H and x i+1) := x i) + IH h ei) H Conjugate Gadient Methods, Summe

4 Pat I Quadatic Foms and Steepest Descent Quadatic Foms Diection of Steepest Descent Steepest Descent Conjugate Gadient Methods, Summe

5 Quadatic Foms A quadatic fom is a scala, quadatic function of a vecto of the fom: f x) = 1 2 x T Ax b T x + c, whee A = A T y 2 x x y Conjugate Gadient Methods, Summe

6 Quadatic Foms 2) The gadient of a quadatic fom is defined as f x) = Ax b f x) = x 1 f x). x n f x) f x) = 0 Ax b = 0 Ax = b Ax = b equivalent to a minimisation poblem pope minimum only if A positive definite Conjugate Gadient Methods, Summe

7 Diection of Steepest Descent gadient f x): diection of steepest ascent f x) = Ax b = with esidual = b Ax) esidual : diection of steepest descent 4 y x Conjugate Gadient Methods, Summe

8 Solving SLE via Minimum Seach basic idea to find minimum: move into diection of steepest descent most simple scheme: x i+1) = x i) + α i) α constant Richadson iteation often consideed as a elaxation method) bette choice of α: move to lowest point in that diection Steepest Descent Conjugate Gadient Methods, Summe

9 Steepest Descent find an optimal α task: line seach along the line x 1) = x 0) + α 0) choose α such that f x 1) ) is minimal: use chain ule: α f x 1) ) = 0 α f x 1) ) = f x 1) ) T α x 1) = f x 1) ) T 0) emembe f x 1) ) = 1), thus: 1)) T 0)! = 0 hence, f x 1) ) = 1) should be othogonal to 0) Conjugate Gadient Methods, Summe

10 Steepest Descent find α 2) 1)) T 0) = b Ax 1)) T 0) ) T b Ax 0) + α 0) ) 0) b Ax 0)) T 0) α A 0)) T 0) 0)) T 0) α 0)) T A 0) = 0 = 0 = 0 = 0 Solve fo α: ) 0) T 0) α = ) 0) T A 0) Conjugate Gadient Methods, Summe

11 Steepest Descent Algoithm 4 1. i) = b Ax i) ) i) T i) 2. α i = ) i) T A i) 3. x i+1) = x i) + α i i) y 2 x Obsevations: slow convegence sim. to Jacobi elaxation) ) i e i) κ 1 e A 0) κ+1 A fo positive definite A: κ = λ max /λ min lagest/smallest eigenvalues of A) many steps in the same diection -6 Conjugate Gadient Methods, Summe

12 Pat II Conjugate Gadients Conjugate Diections A-Othogonality Conjugate Gadients A Miacle Occus... CG Algoithm Conjugate Gadient Methods, Summe

13 Conjugate Diections obsevation: Steepest Descent takes epeated steps in the same diection obvious idea: ty to do only one step in each diection possible appoach: choose othogonal seach diections d 0) d 1) d 2)... notice: eos othogonal to pevious diections: e 1) d 0), e 2) d 1) d 0),... Conjugate Gadient Methods, Summe

14 Conjugate Diections 2) compute α fom d 0)) T e 1) = d 0)) T e 0) αd 0)) = 0 equies popagation of the eo e 1) = x x 1) x 1) = x 0) + α i d 0) x x 1) = x x 0) α i d 0) e 1) = e 0) α i d 0) fomula fo α: ) d 0) T e 0) α = ) d 0) T d 0) but: we don t know e 0) Conjugate Gadient Methods, Summe

15 A-Othogonality make the seach diections A-othogonal: d i)) T Ad j) = 0 again: eos A-othogonal to pevious diections: e i+1)) T Ad i)! = 0 equiv. to minimisation in seach diection d i) : f x i+1)))t α f x i+1)) = α x i+1) = 0 i+1)) T d i) d i)) T Ae i+1) = 0 = 0 Conjugate Gadient Methods, Summe

16 A-Conjugate Diections emembe the fomula fo conjugate diections: ) d 0) T e 0) α = ) d 0) T d 0) same computation, but with A-othogonality: ) d i) T Ae i) α i = ) d i) T Ad i) ) d i) T i) = ) d i) T Ad i) fo the i-th iteation) these α i can be computed! still to do: find A-othogonal seach diections Conjugate Gadient Methods, Summe

17 A-Conjugate Diections 2) classical appoach to find othogonal diections conjugate Gam-Schmidt pocess: fom linealy independent vectos u 0), u 1),..., u i 1) constuct othogonal diections d 0), d 1),..., d i 1) i 1 d i) = u i) + β ik d k) k=0 β ik = ui) ) T Ad k) d k) ) T Ad k) needs to keep all old seach vectos in memoy On 3 ) computational complexity infeasible Conjugate Gadient Methods, Summe

18 Conjugate Gadients use esiduals i.e., u i) := i) ) to constuct conjugate diections: i 1 d i) = i) + β ik d k) k=0 new diection d i) should be A-othogonal to all d j) : 0 =! d i)) T Ad j) = i)) T i 1 Ad j) + β ik d k) ) T Ad j) all diections d k) fo k = 0,..., i 1) ae aleady A-othogonal and j < i), hence: 0 = i)) T Ad j) + β ij d j) ) ) T Ad j) i) T Ad j) β ij = ) d j) T Ad j) k=0 Conjugate Gadient Methods, Summe

19 Conjugate Gadients Status 1. conjugate diections and computation of α i : α i = d i) ) T i) d i) ) T Ad i) x i+1) = x i) + α i d i) 2. use esiduals to compute seach diections: i 1 d i) = i) + β ik d k) k=0 ) i) T Ad k) β ik = ) d k) T Ad k) still infeasible, as we need to stoe all vectos d k) Conjugate Gadient Methods, Summe

20 A Miacle Occus Pat 1 Two small contibutions: 1. popagation of the eo e i) = x x i) x i+1) = x i) + α i d i) x x i+1) = x x i) α i d i) e i+1) = e i) α i d i) we have used this once, aleady) 2. popagation of esiduals i+1) = Ae i+1) = A e i) α i d i)) i+1) = i) α i Ad i) Conjugate Gadient Methods, Summe

21 A Miacle Occus Pat 2 Othogonality of the esiduals: seach diections ae A-othogonal only one step in each diections hence: eo is A-othogonal to pevious seach diections: d i) ) T Ae j) = 0, fo i < j esiduals ae othogonal to pevious seach diections: d i) ) T j) = 0, fo i < j seach diections ae built fom esiduals: span { d 0),..., d i 1)} = span { 0),..., i 1)} hence: esiduals ae othogonal i) ) T j) = 0, i < j Conjugate Gadient Methods, Summe

22 A Miacle Occus Pat 3 combine othogonality and ecuence fo esiduals: i) ) T j+1) α j i) ) T Ad j) = i)) T j) α j i) ) T Ad j) = i)) T j) i)) T j+1) i)) T j) = 0, if i j: i) ) ) 1 T Ad j) α i i) T i), i = j = ) 1 α i 1 i) T i), i = j othewise. Conjugate Gadient Methods, Summe

23 A Miacle Occus Pat 4 computation of β ik fo k = 0,..., i 1): ) i) T Ad k) β ik = ) = d k) T Ad k) thus: seach diections ) i) T i) α ), if i = k + 1 i 1 d i 1) T Ad i 1) 0, if i > k + 1 i 1 d i) = i) + β ik d k) = i) + β i,i 1 d i 1) k=0 ) i) T i) β i := β i,i 1 = α ) i 1 d i 1) T Ad i 1) educes to a simple iteative scheme fo β i Conjugate Gadient Methods, Summe

24 A Miacle Occus Pat 5 build seach diections emembe: α i = d i) ) T i) d i) ) T Ad i) thus: α i d i) ) T Ad i) = d i) ) T i) d i+1) = i+1) + β i d i) β i+1 = ) i+1) T i+1) α ) i d i) T Ad i) β i+1 = i+1) ) T i+1) d i) ) T i) = i+1) ) T i+1) i) ) T i) last step: d i) ) T i) = i) + β i 1 d i 1)) T i) = i)) T i) + β i 1 d i 1) ) T i) = i)) T i) esidual i) othogonal to pevious seach diection d i 1) ) Conjugate Gadient Methods, Summe

25 Conjugate Gadients Algoithm Stat with d 0) = 0) = b Ax 0) While i) > ɛ iteate ove: 1. α i = i) ) T i) d i) ) T Ad i) 2. x i+1) = x i) + α i d i) 3. i+1) = i) α i Ad i) 4. β i+1 = i+1) ) T i+1) i) ) T i) 5. d i+1) = i+1) + β i+1 d i) Conjugate Gadient Methods, Summe

26 Pat III Peconditioning CG Convegence Peconditioning CG with Change-of-Basis Peconditioning CG with Matix Peconditione Peconditiones Examples ILU and Incomplete Cholesky Conjugate Gadient Methods, Summe

27 Conjugate Gadients Convegence Convegence Analysis: uses Kylow subspace: span { 0), A 0), A 2 0),..., A i 1 0)} Kylow subspace method Convegence Results: in pinciple: diect method n steps) howeve: othogonality lost due to ound-off eos exact solution not found) in pactice: iteative scheme ) i e i) A κ 1 e 0) 2, κ + 1 A κ = λmax/λ min Conjugate Gadient Methods, Summe

28 Peconditioning convegence depends on matix A idea: modify linea system Ax = b M 1 Ax = M 1 b, then: convegence depends on matix M 1 A optimal peconditione: M 1 = A 1 : in pactice: A 1 Ax = A 1 b x = A 1 b. avoid explicit computation of M 1 A find an M simila to A, compute effect of M 1 i.e., appoximate solution of SLE) o: find an M 1 simila to A 1 Conjugate Gadient Methods, Summe

29 CG and Peconditioning just eplace A by M 1 A in the algoithm?? poblem: M 1 A not necessaily symmetic even if M and A both ae) we will ty an altenative fist: symmetic peconditioning Ax = b L T ALˆx = L T b, x = Lˆx Remembe: fo Finite Element discetization, this coesponds to a change of basis functions! equies some e-computations in the CG algoithm see following slides) Conjugate Gadient Methods, Summe

30 Change-of-Basis Peconditioning peconditioned system of equations: Ax = b L } T {{ AL } )ˆx = }{{} L T b, x = Lˆx =:  =: ˆb computation of esidual: ˆ = ˆb  ˆx = LT b L T ALˆx = L T b Ax) = L T computation of α fo new system: i) i) i) i) i) i) ˆ )Tˆ ˆ )Tˆ ˆ )Tˆ α i := ˆd i) )T  ˆd = ˆd ) i) i) T LT AL ˆd = d ) i) i) T A d i) d i) whee we defined Lˆd i) =: update of solution: ˆx i+1) = ˆx i) i) + α i ˆd x i+1) = Lˆx i+1) = Lˆx i) + Lα i ˆd i) = x i) + α i d i) Conjugate Gadient Methods, Summe

31 Change-of-Basis Peconditioning 2) update esiduals ˆ: ˆ i+1) = ˆ i) α i ˆd i) = ˆ i) α i L T i) AL ˆd = ˆ i) α i L T A d i) computation of β i : β i+1 = i+1) i+1) ˆ )Tˆ ˆ i) )Tˆ i) update of seach diections: ˆd i+1) = ˆ i+1) i) + β i ˆd d i+1) = Lˆd i+1) = Lˆ i+1) i) + Lβ i ˆd = Lˆ i+1) i) + β i d Conjugate Gadient Methods, Summe

32 CG with Change-of-Basis Peconditioning Stat with ˆ 0) = L T b Ax 0) ) and d 0) = L ˆ 0) ; While ˆ i) > ɛ iteate ove: i) i) ˆ )Tˆ 1. α i = d ) i) T A d i) 2. x i+1) = x i) + α i d i) 3. ˆ i+1) = ˆ i) α i L T i) A d 4. β i+1 = i+1) i+1) ˆ )Tˆ ˆ i) )Tˆ i) 5. d i+1) = L ˆ i+1) + β i d i) Conjugate Gadient Methods, Summe

33 Hieachical Basis Peconditioning Some specifics fo the CG implementation: L tansfoms coefficient vecto fom hieachical basis to nodal basis, fo example ˆx = L x o d = L ˆd L T tansfoms the vecto of basis functions fom nodal basis to hieachical basis cmp. FEM), thus ˆ = L T effect of L and L T can be computed in ON) opeations HB-CG fo the Poisson poblem: in 1D: convegence afte a log N iteations! in this case: L T AL diagonal matix with log N diffeent eigenvalues) in 2D and 3D vey fast convegence! futhe impoved by additional diagonal peconditioning so-called hieachical geneating systems change to a multigid basis) achieve multigid-like pefomance Conjugate Gadient Methods, Summe

34 CG with Hieachical Geneating Systems Recall: system of linea equations A GS v GS = b GS given as A h A h P2h h A h P h 4h Rh 2hA h A 2h A 2h P4h 2h v h b h v 2h = Rh 2hb h Rh 4hA h R2h 4hA 2h A 4h v 4h Rh 4hb h Peconditioning fo CG? system A GS v GS = b GS is singula subspace of solutions v GS minima of the quadatic fom!) howeve: any of the solutions will do! convegence esult: i e i) A ˆκGS 1 e 0) 2, ˆκ GS = ˆκGS + 1) ˆλ max /ˆλ min A whee ˆκ GS is the atio of lagest vs. smallest non-zeo eigenvalue fo Poisson eq.: ˆκ GS independent of h multigid convegence Conjugate Gadient Methods, Summe

35 CG and Peconditioning evisited) peconditioning: eplace A by M 1 A poblem: M 1 A not necessaily symmetic compae symmetic peconditioning Ax = b L T ALˆx = Lb, x = Lˆx wokaound: find E T E = M Cholesky fact.), then Ax = b E T AE 1ˆx = E T b, ˆx = Ex what if E cannot be computed efficiently)? neithe M no M 1 might be known explicitly!) E, E T, E 1 can be eliminated fom algoithm again equies some e-computations): set ˆd = Ed and use ˆ = E T, ˆx = Ex, E 1 E T = M 1 Conjugate Gadient Methods, Summe

36 CG with Peconditione Stat: 0) = b Ax 0) ; d 0) = M 1 0) 1. α i = i) ) T M 1 i) d i) ) T Ad i) 2. x i+1) = x i) + α i d i) 3. i+1) = i) α i Ad i) 4. β i+1 = i+1) ) T M 1 i+1) i) ) T M 1 i) 5. d i+1) = M 1 i+1) + β i+1 d i) fo detailed deivation, see Shewchuck) Conjugate Gadient Methods, Summe

37 Implementation Peconditioning steps: M 1 i), M 1 i+1) M 1 known then multiply M 1 i) M known? Then solve My = i) to obtain y = M 1 i) neithe M, no M 1 ae known explicitly: algoithm to solve My = i) is sufficient! any appoximate solve fo Ae = i) algoithm to compute M 1 is sufficient! compute spase) appoximate invese SPAI) Examples: Multigid, Jacobi, ILU, SPAI,... Conjugate Gadient Methods, Summe

38 Peconditiones fo CG Examples find M A and compute effect of M 1 : Jacobi peconditioning: M := D A Symmetic) Gauss-Seidel peconditioning: M := L A o M = D A + L A )D 1 A D A + L A )T ), etc. just compute effect of M 1 : any appoximate solve might do incl. multigid methods incomplete LU-decomposition ILU) should be symmetic incomplete Cholesky factoization use a multigid method as peconditione?) wothwhile only) in situations whee multigid does not wok well) as stand-alone solve find an M 1 simila to A 1 spase appoximate invese SPAI) ties to minimise I MA F, whee M is a matix with given) spase non-zeo patten Conjugate Gadient Methods, Summe

39 Peconditiones ILU and Incomplete Cholesky Recall LU decomposition and Cholesky factoization: LU decomposition: given A, find lowe/uppe tiangula matices L and U such that A = LU Cholesky factoization: given A = A T, find lowe tiangula matix L such that A = LL T symmetic peconditioning! vaiants with explicit diagonal matix D: A = LD 1 U o A = LD 1 L T, whee L = D + L and R = D + R with stict lowe/uppe tiangula L, R but: fo spase A, L and U may be non-spase Idea: disegad all fill-in duing factoization Conjugate Gadient Methods, Summe

40 Cholesky Factoization L 11 L 21 L 22 L 31 L 32 L 33 D 1 11 D 1 22 D 1 33 L T 11 L T 21 L T 31 A 11 A T 21 A T 31 L T 22 L T 32 = A 21 A 22 A T 32 L T 33 A 31 A 32 A 33 Deive the factoization algoithm:! assume that A 11 = L 11 D 1 11 LT 11 is aleady factoized let L 21 be a 1 k submatix, i.e., to compute next ow of L: L 21 D 1 11 LT! 11 = A 21 D 1 11 LT 11 uppe tiangula matix solve tiangula system fo L 21 by convention L 22 = D 22, which is computed fom: L 21 D 1 11 LT 21+L 22 D 1 22 L T 22! = A 22 L 22 = D 22 := A 22 L 21 D 1 11 LT 21 Conjugate Gadient Methods, Summe

41 Incomplete Cholesky Factoization Algoithm: A LD 1 L T ) initialize D := 0, L := 0 fo i = 1,..., n: 1. fo k = 1,..., i 1: if i, k) S then set L ik := A ik L ij D 1 2. set L ii = D ii := A ii L ij D 1 jj L ij j<i note: sums j<k and only conside non-zeo elements S j<i uses given patten S of non-zeo elements in the factoization fequent choice: use non-zeos of A fo S) Cholesky factoization computed in On) opeations fo spase matices with c n non-zeos) fequently used fo peconditioning j<k jj L kj Conjugate Gadient Methods, Summe

42 Liteatue/Refeences Conjugate Gadients: Shewchuk: An Intoduction to the Conjugate Gadient Method Without the Agonizing Pain. Hackbusch: Iteative Solution of Lage Spase Systems of Equations, Spinge M. Giebel: Multilevelmethoden als Iteationsvefahen ï 1 2 be Ezeugendensystemen, Teubne Skipten zu Numeik, 1994 M. Giebel: Multilevel algoithms consideed as iteative methods on semidefinite systems, SIAM Int. J. Sci. Stat. Comput. 153), Conjugate Gadient Methods, Summe