Conjugate Gradient (CG) Method

Transcription

1 Optimization II

2 Conugate Gaient CG Metho Anothe metho oes not equie explicit secon eivatives, an oes not even stoe appoximation to Hessian matix CG geneates sequence of conugate seach iections, implicitly accumulating infomation about Hessian matix Fo quaatic obective function, CG is theoetically exact afte at most n iteations, whee n is the imension of poblem Numeical Methos Wen-Chieh Lin 2

3 Conugate Diections x 1 x f x x x 1 Pic a set of iections: 0, 1,, n-1 ae exactly one step along each iection Solution is foun within n steps wo poblems: how o we etemine these iections? how to we etemine the step size along each iections? Numeical Methos Wen-Chieh Lin 3

4 How o we etemine the optimal step size? Suppose we now the CG iections fo now, we want to now the optimal step size in each CG iection? Numeical Methos Wen-Chieh Lin 4

5 Conugacy A-othogonality If we tae the optimal step size along each conugate iection f f x 1 0 x 1 0 x 1 1 wo vectos v an u ae conugate o A- othogonal if va u=0 0 0 Ae 1 x Ae x 1 Numeical Methos Wen-Chieh Lin A x * x 1 1

6 A-othogonality vs. Othogonality A-othogonal vectos othogonal vectos Numeical Methos Wen-Chieh Lin 6

7 Optimal Step Size 0 Ae 1 e 1 must be A - othogonal to Use this conition, can you eive α? Numeical Methos Wen-Chieh Lin 7

8 Algoithm Suppose we can come up with a set of A- othogonal iections { } 1. ae x x A 1 Numeical Methos Wen-Chieh Lin 8

9 Why oes it wo? We nee to pove that x* can be foun in n steps if we tae α step size along at each step n 1 e0 i i i0 n 1 Ae0 i0 i s ae linealy inepenent if s ae conugate A i Ae A 0 See poof in next slie Ae 0 A e A A ecall Ae A Ae Numeical Methos Wen-Chieh Lin 9

10 Poof that x x e x 1 1 x * x x * e 1 e e e e 1 e e e e 0 e 1 1 e Numeical Methos Wen-Chieh Lin 10

11 A-othogonality cont. 0 Ae 1 e e Numeical Methos Wen-Chieh Lin 11

12 Eo tem vanishes afte n steps e e n1 0 Afte n steps, e n = 0 1 n 1 0 he eo at step is equal to 1 e 0 + total path of pevious -1 steps tavele 2 total path of next n- steps that ae going to be tavele Numeical Methos Wen-Chieh Lin 12

13 Seach Diections We now how to etemine the optimal step size along each iection We still nee to figue out what seach iections ae What o we now about 0, 1,, n-1? hey ae A-othogonal to each othe: i A = 0 i is A-othogonal to e i+1 Numeical Methos Wen-Chieh Lin 13

14 Gam-Schmit Conugation Suppose we have a set of linealy inepenent vectos u 0, u 1,, u n-1 o constuct i, tae u i an subtact out any components that ae not A-othogonal to the pevious vectos: 1 = u 1 - u + Numeical Methos Wen-Chieh Lin 14

15 Gam-Schmit Conugation he seach iections can be epesente as 1 u i i an 0 u 0 i0 Use the same tic to get i of the summation A u A A 0 u A A u A A by A-othogonality of vectos Numeical Methos Wen-Chieh Lin 15

16 Dawbacs of Gam-Schmit Conugation All the ol seach vectos must be ept in memoy to constuct each new one! 1 u i i an 0 u 0 i0 Remey conugate gaients Numeical Methos Wen-Chieh Lin 16

17 Conugate Gaients If we pic a set of u s intelligently, we might be able to save both time an space It tuns out that esiuals s is an excellent choice fo u s esiual is othogonal to the pevious seach iection esiuals wo fo steepest escent Numeical Methos Wen-Chieh Lin 17

18 Conugate Gaients cont. Gam-Schmit Conugation 1 u i i an 0 u 0 i0 In conugate gaient metho, seach iections ae constucte by conugation of the esiuals setting u Numeical Methos Wen-Chieh Lin 18

19 Numeical Methos Wen-Chieh Lin 19 Conugate Gaients cont. ae an subtact out any components that ae not A-othogonal to the pevious vectos 1 0 i i i i i i 1 0 A A A 0 A A by A-othogonality of vectos A A

20 Conugate Gaients cont. is A-othogonal to all the pevious seach iections except fo -1 1 A A 1 0, if if 1 1 We on t nee to stoe pevious seach iections now! Poof: A 0 when 1 Numeical Methos Wen-Chieh Lin 20

21 Numeical Methos Wen-Chieh Lin 21 Poof: Othogonality Poof: is othogonal to all the pevious seach iections 0, 1,, -1 Fom hee, we can pove 1 n i i i e 1 if 0 n i i i A Ae if 0 i i if 0 see page of Shewchu s pape

22 Geometic Intepetation 0 if 0 Ae 1 u 1 i0 i i Numeical Methos Wen-Chieh Lin 22

23 Numeical Methos Wen-Chieh Lin 23 othewise A Poof: A-Othogonality is A-othogonal to all the pevious seach iections except fo Ae e A A 1 A

24 Numeical Methos Wen-Chieh Lin 24 Conugate Gaients Recall in slie 19, , A A A A A fom last slie A Ae Fom slie 9:

25 Numeical Methos Wen-Chieh Lin 25 Conugate Gaients Put it all togethe Ax b A 1 A x x

26 Each new esiual is othogonal to all the pevious esiual an seach iections Each new seach iection is constucte fom the esiual to be A-othogonal to all pevious esiuals an seach iections Numeical Methos Wen-Chieh Lin 26

27 Example: Minimize fx,y=1.5*x^2+2*x*y+3*y^2-2*x+8*y 3 2 A b 8 ugategaient/ Numeical Methos Wen-Chieh Lin 27