Gadient Methods Outlines Pat I: Quadatic Foms and Steepest Descent Pat II: Gadients Pat III: Summe tem 2012
Pat I: Quadatic Foms and Steepest Descent Outlines Pat I: Quadatic Foms and Steepest Descent Pat II: Gadients Pat III: 1 Quadatic Foms 2 Diection of Steepest Descent 3 Steepest Descent
Pat II: Gadients 4 Diections 5 A-Othogonality Outlines Pat I: Quadatic Foms and Steepest Descent Pat II: Gadients Pat III: 6 Gadients 7 A Miacle Occus... 8 CG Algoithm
Pat III: 9 CG Convegence 10 Outlines Pat I: Quadatic Foms and Steepest Descent Pat II: Gadients Pat III: 11 CG with Change-of-Basis 12 CG with Matix Peconditione 13 Peconditiones Examples
Families of Iteative Solves elaxation methods: Jacobi-, Gauss-Seidel-Relaxation,... Ove-Relaxation-Methods Kylov methods: Steepest Descent, Gadient,... GMRES,... Multilevel/Multigid methods, Domain Decomposition,... Outlines Pat I: Quadatic Foms and Steepest Descent Pat II: Gadients Pat III:
Remembe: The Residual Equation fo Ax = b, we defined the esidual as: = b Ax and the eo: e := x x leads to the esidual equation: Outlines Pat I: Quadatic Foms and Steepest Descent Pat II: Gadients Pat III: Ae = elaxation methods: solve a modified easie) SLE: B ê = whee B A multigid methods: coase-gid coection on esidual equation
Quadatic Foms Diection of Steepest Descent Pat I Steepest Descent Quadatic Foms and Steepest Descent
Quadatic Foms A quadatic fom is a scala, quadatic function of a vecto of the fom: f x) = 1 2 x T Ax b T x + c, whee A = A T Quadatic Foms Diection of Steepest Descent Steepest Descent 4 160 2 y x 120-2 0 0 2 4 6 80 40-2 0-4 -2 0 x 2 4 6-6 -4-2 0 y 2 4-4 -6
Quadatic Foms 2) The gadient of a quadatic fom is defined as x 1 f x) f x) =. x n f x) Quadatic Foms Diection of Steepest Descent Steepest Descent f x) = Ax b f x) = 0 Ax b = 0 Ax = b Ax = b equivalent to a minimisation poblem pope minimum only if A positive definite
Diection of Steepest Descent gadient f x): diection of steepest ascent f x) = Ax b = with esidual = b Ax) esidual : diection of steepest descent Quadatic Foms Diection of Steepest Descent Steepest Descent 4 y 2-2 0 2 4 0 x 6-2 -4-6
Solving SLE via Minimum Seach Quadatic Foms basic idea to find minimum: move into diection of steepest descent most simple scheme: Diection of Steepest Descent Steepest Descent x i+1) = x + α α constant Richadson iteation often consideed as a elaxation method) bette choice of α: move to lowest point in that diection Steepest Descent
Steepest Descent find an optimal α task: line seach along the line x 1) = x 0) + α 0) choose α such that f x 1) ) is minimal: Quadatic Foms Diection of Steepest Descent Steepest Descent α f x 1) ) = 0 use chain ule: α f x 1) ) = f x 1) ) T α x 1) = f x 1) ) T 0) emembe f x 1) ) = 1), thus: 1)) T 0)! = 0
Steepest Descent find α 2) ) 1) T 0) = b Ax 1)) T 0) b Ax 0) + α 0) ) ) T 0) ) b Ax 0) T 0) α A 0)) T 0) ) 0) T 0) α 0)) T A 0) = 0 = 0 = 0 = 0 Quadatic Foms Diection of Steepest Descent Steepest Descent Solve fo α: ) 0) T 0) α = 0) ) T A 0)
Steepest Descent Algoithm 1 = b Ax ) T 2 α i = ) T A 3 x i+1) = x + α i 4 y 2 x -2 0 2 4 0-2 -4 6 Quadatic Foms Diection of Steepest Descent Steepest Descent Obsevations: slow convegence sim. to Jacobi elaxation) e A κ 1 i e 0) κ+1) A fo positive definite A: κ = λ max /λ min lagest/smallest eigenvalues of A) many steps in the same diection -6
Diections A-Othogonality Pat II Gadients Gadients A Miacle Occus... CG Algoithm
Diections Diections obsevation: Steepest Descent takes epeated steps in the same diection obvious idea: ty to do only one step in each diection possible appoach: choose othogonal seach diections d 0) d 1) d 2)... notice: eos othogonal to pevious diections: e 1) d 0), e 2) d 1) d 0),... A-Othogonality Gadients A Miacle Occus... CG Algoithm
Diections 2) compute α fom ) d 0) T e 1) = d 0)) T e 0) αd 0)) = 0 equies popagation of the eo e 1) = x x 1) x 1) = x 0) + α i d 0) x x 1) = x x 0) α i d 0) e 1) = e 0) α i d 0) Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm fomula fo α: α = d 0) ) T e 0) d 0) ) T d 0) but: we don t know e 0)
A-Othogonality make the seach diections A-othogonal: d ) T Ad j) = 0 again: eos A-othogonal to pevious diections: e i+1) ) T Ad! = 0 Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm equiv. to minimisation in seach diection d : α f x i+1)) = f x i+1)))t α x i+1) = 0 i+1)) T d = 0 d ) T Ae i+1) = 0
A- Diections emembe the fomula fo conjugate diections: ) d 0) T e 0) α = ) d 0) T d 0) same computation, but with A-othogonality: Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm α i = d ) T Ae d ) T Ad = d ) T d ) T Ad fo the i-th iteation) these α i can be computed! still to do: find A-othogonal seach diections
A- Diections 2) classical appoach to find othogonal diections conjugate Gam-Schmidt pocess: fom linealy independent vectos u 0), u 1),..., u i 1) constuct othogonal diections d 0), d 1),..., d i 1) i 1 d = u + β ik d k) k=0 β ik = u ) T Ad k) d k) ) T Ad k) Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm needs to keep all old seach vectos in memoy On 3 ) computational complexity infeasible
Gadients use esiduals i.e., u := 0) ) to constuct conjugate diections: i 1 d = + β ik d k) k=0 new diection d should be A-othogonal to all d j) : Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm 0 =! d ) T Ad j) = ) i 1 T Ad j) + β ) ik d k) T Ad j) k=0 all diections d k) fo k = 0,..., i 1) ae aleady A-othogonal and j < i), hence: 0 = ) T Ad j) +β ) ) ij d j) T T Ad j) Ad j) β ij = ) d j) T Ad j)
Gadients Status 1 conjugate diections: α i = ) d T ) d T Ad x i+1) = x + α i d Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm 2 use esiduals to compute seach diections: i 1 d = + β ik d k) k=0 ) T Ad k) β ik = ) d k) T Ad k) still infeasible, as we need to stoe all vectos d k)
Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm
A Miacle Occus Pat 1 Two small contibutions: 1 popagation of the eo e = x x x i+1) = x + α i d x x i+1) = x x α i d e i+1) = e α i d Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm we have used this once, aleady) 2 popagation of esiduals i+1) = Ae i+1) = A e α i d ) i+1) = α i Ad
A Miacle Occus Pat 2 Othogonality of the esiduals: seach diections ae A-othogonal only one step in each diections hence: eo is A-othogonal to pevious seach diections: d ) T Ae j) = 0, fo i < j esiduals ae othogonal to pevious seach diections: d ) T j) = 0, fo i < j seach diections ae built fom esiduals: span { d 0),..., d i 1)} = span { 0),..., i 1)} hence: esiduals ae othogonal Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm ) T j) = 0, i < j
A Miacle Occus Pat 3 combine othogonality and ecuence fo esiduals: ) T j+1) = ) T j) α ) j T Ad j) α ) j T Ad j) = ) T j) ) T j+1) Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm ) T j) = 0, if i j: ) ) 1 T Ad j) α i T, i = j = ) 1 α i 1 T, i = j + 1 0 othewise.
A Miacle Occus Pat 4 computation of β ik fo k = 0,..., i 1): ) T Ad k) β ik = ) d k) T Ad k) = thus: seach diections ) T α i 1 d i 1) ) T Ad i 1), i = k + 1 0 i > k + 1 i 1 d = + β ik d k) = + β i,i 1 d i 1) k=0 ) T β i := β i,i 1 = α ) i 1 d i 1) T Ad i 1) Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm
A Miacle Occus Pat 5 build seach diections d i+1) = i+1) + β i d ) i+1) T i+1) β i+1 = α ) i d T Ad emembe: α i = d ) T d ) T Ad thus: α i d ) T Ad = d ) T Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm i+1) ) T i+1) i+1) ) T i+1) β i+1 = d ) T = ) T last step: d ) T = + β i 1 d i 1)) T = ) T + β i 1 d i 1) ) T = ) T esidual othogonal to pevious seach diection d i 1) )
Gadients Algoithm Stat: d 0) = 0) = b Ax 0) 1 α i = ) T d ) T Ad 2 x i+1) = x + α i d Diections A-Othogonality Gadients A Miacle Occus... CG Algoithm 3 i+1) = α i Ad 4 β i+1 = i+1) ) T i+1) ) T 5 d i+1) = i+1) + β i+1 d
CG Convegence Pat III CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples
Gadients Convegence Convegence Analysis: uses Kylow subspace: span { 0), A 0), A 2 0),..., A i 1 0)} Kylow subspace method CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples Convegence Results: in pinciple: diect method n steps) in pactice: iteative scheme ) i e κ 1 A 2 e 0) κ + 1 A, κ = λmax/λ min
convegence depends on matix A idea: modify linea system Ax = b M 1 Ax = M 1 b, then: convegence depends on matix M 1 A optimal peconditione: M 1 = A 1 : CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples A 1 Ax = A 1 b x = A 1 b. in pactice: avoid explicit computation of M 1 A find an M simila to A, compute effect of M 1 i.e., appoximate solution of SLE) o: find an M 1 simila to A 1
CG and just eplace A by M 1 A in the algoithm?? poblem: M 1 A not necessaily symmetic even if M and A both ae) we will ty an altenative fist: symmetic peconditioning CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples Ax = b LAL T ˆx = Lb, x = L T ˆx Note: fo Finite Element discetization, this coesponds to a change of basis functions! equies some e-computations in the CG algoithm see following slides)
Change-of-Basis peconditioned system of equations: Ax = b L} T {{ AL} )ˆx = }{{} L T b, x = Lˆx =:  =: ˆb computation of esidual: ˆ = ˆb  ˆx = LT b L T ALˆx = L T b Ax) = L T computation of α fo new system: ˆ ) Tˆ ˆ ) Tˆ ˆ ) Tˆ α i := ˆd )T =  ˆd ˆd ) = T LT AL ˆd d ) T A d CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples whee we defined Lˆd =: update of solution: d ˆx i+1) = ˆx + α i ˆd x i+1) = Lˆx i+1) = Lˆx + Lα i ˆd = x + α i d
Change-of-Basis 2) update esiduals ˆ: ˆ i+1) = ˆ α i ˆd = ˆ α i L T AL = ˆ α i L T A d computation of β i : ˆ ) i+1) i+1) Tˆ β i+1 = ˆ ) Tˆ ˆd CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples update of seach diections: ˆd i+1) = ˆ i+1) + β i ˆd d i+1) = Lˆd i+1) = Lˆ i+1) + Lβ i ˆd = Lˆ i+1) + β i d
CG with Change-of-Basis Stat: ˆ 0) = L T b Ax 0) ); d 0) = L ˆ 0) 1 α i = ˆ ) Tˆ d ) T A d CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples 2 x i+1) = x + α i d 3 ˆ i+1) = ˆ α i L T A d 4 β i+1 = ˆ i+1) ) Tˆ i+1) ˆ ) Tˆ 5 d i+1) = L ˆ i+1) + β i d
Hieachical Basis Some specifics fo the CG implementation: L tansfoms vecto fom nodal basis to hieachical basis, fo example ˆ = L L T tansfoms the vecto of basis functions fom nodal basis to hieachical basis cmp. FEM) effect of L and L T can be computed in ON) opeations HB-CG fo the Poisson poblem: in 1D: convegence afte a log N iteations! in this case: L T AL diagonal matix with log N diffeent eigenvalues) in 2D and 3D vey fast convegence! futhe impoved by additional diagonal peconditioning so-called hieachical geneating systems change to a multigid basis) achieve multigid-like pefomance CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples
CG and evisited) peconditioning: eplace A by M 1 A poblem: M 1 A not necessaily symmetic compae symmetic peconditioning Ax = b L T ALˆx = Lb, x = Lˆx wokaound: find E T E = M Cholesky fact.), then CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples Ax = b E T AE 1ˆx = E T b, ˆx = Ex what if E cannot be computed efficiently)? neithe M no M 1 might be known explicitly!) E, E T, E 1 can be eliminated fom algoithm equies some e-computations)
CG with Peconditione Stat: 0) = b Ax 0) ; d 0) = M 1 0) 1 α i = ) T M 1 d ) T Ad 2 x i+1) = x + α i d CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples 3 i+1) = α i Ad 4 β i+1 = i+1) ) T M 1 i+1) ) T M 1 5 d i+1) = M 1 i+1) + β i+1 d
Implementation CG Convegence steps: M 1, M 1 i+1) M 1 known then multiply M 1 M known, then solve My = to obtain y = M 1 neithe M, no M 1 ae known explicitly: algoithm to solve My = is sufficient! any appoximate solve fo Ae = algoithm to compute M 1 is sufficient! compute spase) appoximate invese SPAI) Examples: Multigid, Jacobi, ILU, SPAI,... CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples
Peconditiones fo CG Examples find M A and compute effect of M 1 : Jacobi peconditioning: M := D A Gauss-Seidel peconditioning: M := L A, etc. just compute effect of M 1 : any appoximate solve might do incl. multigid methods incomplete LU-decomposition ILU) use a multigid method as peconditione?) wothwhile only) in situations whee multigid does not wok well) as stand-alone solve find an M 1 simila to A 1 spase appoximate invese SPAI) ties to minimise I MA F, whee M is a matix with given) spase non-zeo patten CG Convegence CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples
Liteatue CG Convegence Geneal: Gande, Hebicek: Solving Poblems in Computing Using Maple and MATLAB. Golub, Otega: Computing and Diffeential Equations. Dongaa, et. al.: Numeical linea algeba fo high-pefomance computes. CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples
Liteatue 2) CG Convegence Multigid: Biggs, Henson, McComick: A Multigid Tutoial 2nd ed.). Gadients: Shewchuk: An Intoduction to the Gadient Method Without the Agonizing Pain. CG with Change-of-Basis CG with Matix Peconditione Peconditiones Examples