A rapidly convergent descent method for minimization

Transcription

1 A rapdly convergent descent method for mnmzaton By R. Fletcher and M. J. D. Powell A powerful teratve descent method for fndng a local mnmum of a functon of several varables s descrbed. A number of theorems are proved to show that t always converges and that t converges rapdly. Numercal tests on a varety of functons confrm these theorems. The method has been used to solve a system of one hundred non-lnear smultaneous equatons. 1. Introducton We are concerned n ths paper wth the general problem of fndng an unrestrcted local mnmum of a functon J[x u x 2, x n ) of several varables x x, x 2,., x n. We suppose that the functon of nterest can be calculated at all ponts. It s convenent to group functons nto two man classes accordng to whether the gradent vector g, = Wbx, s defned analytcally at each pont or must be estmated from the dfferences of values of/. The method descrbed n ths paper s applcable to the case of a defned gradent. For the other case a useful method and general dscusson are gven by Rosenbrock (196). Methods usng the gradent nclude the classcal method of steepest descents (Courant, 1943; Curry, 1944; and Householder, 1953), Levenberg's modfcaton of damped steepest descents (1944), a somewhat smlar varaton due to Booth (1957), the conjugate gradent method of Hestenes and Stefel (1952), smlar methods of Martn and Tee (1961), the "Partan" method of Shah, Buehler and Kempthorne (1961), and a method due to Powell (1962). In ths paper we descrbe a powerful method wth rapd convergence whch s based upon a procedure descrbed by Davdon (1959). Davdon's work has been lttle publczed, but n our opnon consttutes a consderable advance over current alternatves. We have made both a smplfcaton by whch certan orthogonalty condtons whch are mportant to the rate of attanng the soluton are preserved, and also an mprovement n the crteron of convergence. Because, near the mnmum, the second-order terms n the Taylor seres expanson domnate, the only methods whch wll converge quckly for a general functon are those whch wll guarantee tofnd the mnmum of a general quadratc speedly. Only the latter four methods of the last paragraph do ths, and the procedures of Hestenes and Stefel and of Martn and Tee are not applcable to a general functon. Of course the generalzed Newton-Raphson method (Householder, 1953) has fast convergence eventually, but t requres second dervatves of the functon to be evaluated, and frequently fals to converge from a poor approxmaton to the mnmum. The method descrbed has quadratc convergence and s superor to "Partan" and to Powell's method, both n that t makes use of nformaton determned by prevous teratons and also n that each teraton s quck and smple to carry out. 163 Furthermore, t yelds the curvature of the functon at the mnmum, so excellent tests for convergence and estmates of varance can be made. The method s gven an elegant theoretcal bass, and proofs of stablty and of the rate of convergence are ncluded. The results of numercal tests wth a varety of functons are also gven. These confrm that the method s probably the most powerful general procedure for fndng a local mnmum whch s known at the present tme. 2. taton It s convenent to descrbe the method n terms of the Drac bra-ket notaton (Drac, 1958) appled to real vectors. In ths notaton the column vector (x u x 2., x n ) s wrtten as x>. The row vector wth these same elements s denoted by <x. The scalar product of <x and j> s wrtten (x\y} and we may note that <x\y~) s= 2x,-y, = j,x; s= <j/ x>. / The constructon.x><j>, however, denotes a lnear operator wth matrx elements D } = x,y>j so that.x><j> #.F><*. A general lnear operator or matrx wll be denoted by a captal letter n bold type. It then follows that say H\xy s a column vector, (x\h s a row vector and (x\h\yy s a scalar. We reserve / to denote the functon of nterest, x> to denote ts arguments and g> to denote ts gradent. Hence the standard quadratc form n n dmensons / = /o + a,x, = 1 becomes n ths notaton and also \g> = = I j= 1 G\x>. 3. The method If we consder the quadratc form (1) then, gven the matrx G u = 'b 2 fl'bx ( 'bxj, we can calculate the dsplacement between the pont \x} and the mnmum \x o y as x o >- x>=-g-' g>. (3) In ths method the matrx G~' s not evaluated drectly; (1) (2)

2 Descent method for nstead a matrx H s used whch may ntally be chosen to be any postve defnte symmetrc matrx. Ths matrx s modfed after the th teraton usng the nformaton ganed by movng down the drecton \sfy = - J5TV> (4) n accordance wth (3). The modfcaton s such that y, the step to the mnmum down the lne s effectvely an egenvector of the matrx H +l G. Ths ensures that as the procedure converges H tends to G~ l evaluated at the mnmum. It s convenent to take the unt matrx ntally for H so that the frst drecton s down the lne of steepest descent. Let the current pont be x'> wth gradent g'> and matrx H'. The teraton can then be stated as follows. Set s<> = - H^g'y. Obtan a.' such that f(\x'} + a' s'>) s a mnmum wth respect to A along \x'} + A s'> and a' >. We wll prove that a' can always be chosen to be postve. Set a ( > = a-y>. (5) Set x' + '> = x' Evaluate /( x /+I» and orthogonal to a'y, that s notng that > s Set Set where and y\h'\yy Set / = + 1 and repeat. There are two obvous and very useful ways of termnatng the procedure, and they arse because ^'> tends to the correcton to \x*y. One s to stop when the predcted absolute dstance from the mnmum (s^1)* s less than a prescrbed amount, and the other s to fnsh when every component of s'> s less than a prescrbed accuracy. Two addtonal safeguards have been found necessary n automatc computer programs. The frst s to work through at least n (the number of varables) teratons, and the second s to apply the tests to CT'> as well as to \s'y. The method of obtanng the mnmum along a lne s not central to the theory. The suggested procedure gven n the Appendx, whch uses cubc nterpolaton, s based on that gven by Davdon, and has been found satsfactory. We shall now show that the process s stable, and demonstrate that f/( x» s the quadratc form (1) then (6) (7) 164 mnmzaton the procedure termnates n n teratons. We shall also explan the theoretcal justfcaton for the manner n whch the matrx H s modfed. 4. Stablty It s usual for descent methods to be stable because one ensures that the functon to be mnmzed s decreased by each step. It wll be shown n ths Secton that the drecton of search \s'}, defned by equaton (4), s downhll, so «' can always be chosen to be postve. Because g'> s the drecton of steepest ascent, the drecton \s'} wll be downhll f and only f s postve. We wsh the drecton of search to be downhll for all possble \g'y so we must prove that H' s postve defnte. Because H has been chosen to be postve defnte an nductve argument wll be used. In the proof t s assumed that H l s postve defnte and consequently that a' s postve. It s proved that, for any x>, (x\h +x \xy >. We may defne \py = (H''Y\xy and?>= (#')* /> as the square root of a postve defnte matrx exsts. From (7) <y\h'\yy on account of Schwartz's nequalty. But <a' /> = <a' g'+ 1 > from (6) from (4) and (5) Hence <x ^l+l x> > for all non-trval x>. Therefore H + s postve defnte and the procedure s stable. 5. Quadratc convergence In ths Secton t s assumed that / s the quadratc form (1) and that /has a well defned mnmum. It s proved that n ths case the method fnds the mnmum n n teratons. The method of proof s to show that o- >, CT'>,..., o*> are lnearly ndependent egenvectors of H k + > G wth egenvalue unty. Therefore t wll follow that H"G s the unt matrx. By defnton from (2) = G\a>y. (8)

3 Also from (8) The equatons Descent method for mnmzaton Ths result can be proved from the orthogonalty condtons (1) because these mply that S'GS = A, where S s the matrx of vectors <x'>, and A s a dagonal matrx - H'\y'> by usng (7) wth elements <a' (? <7'>. = //' y> + = k>. (9) <o \G\o>> = <<j<k (1) and H k G\a>> = a'> < / < k (11) wll now be consdered. It s clear from (9) that they are true f k 1. It wll be proved that f they are true for k they are true for k + 1. From (2) = \a> Hence by defnton G = Therefore G- =SA 1 S / and as A s a dagonal matrx ths reduces to Therefore from the defnton of A' and equaton (8), equaton (15) s proved. The form of the term B' can be deduced because equaton (9) must be vald. For a quadratc we must have Therefore from (1) and (6) Hence from (11) = < < k. > = so from (4) and (5) -a'<a' G a*> =. Therefore Also from (8), (11) and (13) = < < k. (12) (13) Therefore usng the above result and equatons (7), (11) and (13) y = H k G\a l y <<k. (14) Equatons (9), (13) and (14) prove the nducton. Equaton (1) proves that the vectors a >, ICT 1 ),..., <r"~'> are lnearly ndependent and therefore H" = G~ l. That the mnmum s found by n teratons s proved by equaton (12). g"> must be orthogonal to CT >, CT'>,..., O"-'> whch s only possble f g"> s dentcally zero. 6. Improvng the matrx H The matrx H' s modfed by addng to t two terms A' and B'. A' s the factor whch makes H tend to G~ l n the sense that for a quadratc G-< = n-i A 1. (15) /=o 165 Therefore as A'G\&y = a'> the equaton B'G\a l y = - H'G a'> = - //' /> must be satsfed. Ths mples that the smplest form for B> s. = _ H'\yy <z\ and as B' s to be symmetrc ths gves. = _ <y\h'\y'y' Although Davdon's method nvolves these relatons, some of the other deas used by hm can cause H not to tend to G~' even n the quadratc case. The effect n the non-quadratc case would depend upon the functon n queston but mght well lead to slower convergence. 7. Numercal results comparson wth other procedures As a comparson wth other methods we use the functon gven by Rosenbrock ftxy, x 2 ) = 1(x 2 - x]) 2 + (1 - x,) 2 startng at ( 1-2, 1-). Ths functon s dffcult to mnmze on account of ts havng a steep sded valley followng the curve x\ = x 2. Eghteen teratons were requred to reach the mnmum, each one requrng the mnmum to be calculated n only one drecton. Table 1 shows how ths procedure compares wth the classcal steepest descent method and Powell's method, one of the procedures wth quadratc convergence. The table takes nto account that the latter method requres mnma to be found n three drectons for each teraton. It wll be seen that ths method s consderably more effcent than that of Powell, both of these beng far more effcent than steepest descents.

4 Table 1 A comparson n two dmensons Descent method for mnmzaton Table 2 A quadratc functon EQUIVALENT n STEEPEST DESCENTS /(*!. X2) ' POWELL'S METHOD fx u x 2 ) x x lo" 9 OUR METHOD f{x\,x 2 ) x 1-8 ITERATION X f H \ > A l : I o-' 5 1 A smlar comparson was made wth the functon gven by Powell:,, x 2, x 3, x 4 ) = (x, + 1x 2 ) 2 + 5(x 3 x 4 ) 2 (x 2-1(x, - x 4 )< startng at (3, 1,,. 1). In sx teratons the method reduced /from 215 to 2-5 x 1~ 8. Powell's method took the equvalent of seven teratons to reach 9 x 1~ 3, whereas steepest descents only reached 6-36 n seven teratons. The method also brought out the sngularty of G at the mnmum of/, the elements of H becomng ncreasngly large. To compare ths varaton of Davdon's method wth hs orgnal method the smple quadratc /(*1, *2> = A~ 2 *1*2 + 2x1 was used. The complete progress of the method descrbed s gven n Table 2, showng that t does termnate n two teratons and that H does converge to G~ { whch for ths functon s It wll be notced also, as proved, that C"" 1 = "LA!. In Davdon's method, although a value of/of smlar order of magntude had been reached n two teratons, Hhad only reached /-95-47\ \-47-48A Ths was due to one of the alternatves allowed by Davdon. Also hs procedure for termnatng the process was unsatsfactory, and the computaton had to be stopped manually. A non-quadratc test n three dmensons was also 166 made, by usng a functon wth a steep sded helcal valley. Ths functon /(x,, x 2, x 3 ) = 1{[x 3 - lo(x,, x 2 )] 2 + [r(x {, x 2 ) - I] 2 } + x\ where 2TT6(X U X 2 ) = arctan (x 2 /x,), x t > = 77 + arctan (x 2 /x), x t < and / (*!, x 2 ) = (x 2 + x 2 )* has a helcal valley n the x 3 drecton wth ptch 1 and radus 1. It s only consdered for TT/2 <2TT<3 77/2 that s 2-5<x 3 <7-5. It has a mnmum at the pont (1,, ). Both methods were started from ( 1,, ) and H set to the unt matrx. The method gven n ths paper converged n eghteen teratons, whereas Davdon's method requred only ten. However, on account of the more complcated nature of Davdon's teratons, the mnmum often beng sought along more than one drecton n a sngle teraton, the tme taken by the two procedures was almost dentcal. The progress of ths method on the functon s gven n Table Numercal results functons of a large number of varables Tests were also made to fnd out whether the method s sutable for fndng the mnmum of a functon of a large number of varables. In these tests the Stretch computer was used to solve non-lnear smultaneous equatons n up to a hundred varables. The equatons were n S Ajj sn ocj + Bjj cos a,- = E, = 1, 2,..., n

5 n Table 3 A functon wth a steep-sded helcal valley * o- 5 X o- 5 so that the functon to be mnmzed was Descent method for mnmzaton / 2-5 x IO x X x IO- 4 3 x IO- 6 7 x IO- 8 /= {E, - {A lt sn ccj + B,j cos a,)} 2. = j The matrx elements of A and B were generated as random ntegers between 1 and +1, and the values of the varables <x h = 1, 2,..., n were generated randomly between n and -n. For these values the rght-hand sdes of the equatons, E h were worked out. The method of ths paper was appled to fnd optmum values of «,- startng from (a, -fo-ls,) where the 8,'s were also generated as random numbers between IT and 7T. In each run the crteron for convergence was that every a should be found to accuracy 1. The method was entrely successful. Table 4 shows that the number of tmes/and ts dervatves had to be calculated was approxmately lnear n the number of varables. The total tme taken for all the runs was ffteen mnutes, ten mnutes of whch was spent on the fnal case. That a dfferent mnmum was found on fve occasons was not surprsng because f A = B t may be shown that there are up to 2" real solutons to the equatons such that a, < n. Ths abundance of mnma emphaszes the power of the method because n every case t converged to a reasonable soluton. The progress of these tests s nterestng. For the frst n teratons the changes n the functon were smlar to those experenced wth the method of steepest descents, that s a substantal change occurred ntally due to descendng nto a nearby valley, after whch convergence was slow. However, after n teratons had been completed 167 Table 4 Applcaton to a functon of many varables n NO. OF TIMES /EVALUATED WHETHER EXPECTED MINIMUM FOUND a good approxmaton to the fnal matrx H had been accumulated, after whch the functon was decreased substantally at each teraton. For example n the hundred-varable tral the functon to be mnmzed was decreased from to n the frst ten teratons, and to after one hundred teratons. After 12 teratons t was down to 1342, and after 14 to 147. The functon was reduced to -44 by 16 teratons, and the mnmum was found on the 162 nd. The second ffty-varable tral was even more strkng. Ten teratons reduced the functon from 2538 to 4264, ffty teratons reduced t to 3526, and a further ten teratons reduced t to 27. The concluson to be drawn from ths behavour s that for many applcatons of the method a substantal number of the teratons requred wll be spent on settng up the nverse of the matrx of second dervatves. Therefore, f a good postve defnte approxmaton to H can be calculated ntally, as s the case when the method s beng appled to solvng smultaneous equatons, then ths approxmaton should be chosen for H. 9. Concluson The numercal examples show clearly that the type of method gven by Davdon s consderably superor to other methods prevously avalable. The smplfcatons we have made enable programs to be wrtten more easly, and they do not seem to mpar the speed of convergence. It s obvously practcable to apply ths method to fnd a local mnmum of a general functon of a large number of varables whose frst dervatves can be evaluated quckly, even f only poor ntal approxmatons to a soluton are known.

6 Descent method for mnmzaton 1. Acknowledgement where w = (z 2 <,g x \s l y (g y \s'y)* One of us, R.F., s ndebted to Dr. C. M. Reeves for hs constant help and encouragement, and also to the D.S.I.R. for the provson of a research studentshp. Appendx The mnmum on a lne A smple algorthm s gven for estmatng the parameter a.'. A pont y> s chosen on \x 1 } + A j'> wth A >. Let f x, \g x y,f y and \g y y denote the values of the functon and gradent at the ponts \x'} and />. Then an estmate of a' can be formed by nterpolatng cubcally, usng the functon values f x and f y and the components of the gradents along 5'>. Ths s gven by a' "A References <gyw> + W Z = 1 - <gy\s'> - 2W and z = ^ (f x - f y ) + <g x \s>> + <g, * >. A sutable choce of the pont \y'y s gven by h / ' \' = MINIMUM OF (l, /o s the predcted lower bound of/( x», for example zero n least-squares calculatons. Ths value of -q ensures that the choce of /> s reasonable. It s necessary to check that /( *'> + a' s'» s less than both f x and f y. If t s not, the nterpolaton must be repeated over a smaller range. Davdon suggests one should ensure that the mnmum s located between \x! ) and y> by testng the sgn of <^ ^'> and comparng f x and f y before nterpolatng. The reader s referred to Davdon's report for more extensve detals of ths stage. BOOTH, A. D. (1957). Numercal Methods, London: Butterworths. COURANT, R. (1943). "Varatonal methods for the soluton of problems of equlbrum and vbratons," Bull. Amer. Math. Soc, Vol. 49, p. 1. CURRY, H. D. (1944). "The method of steepest descent for non-lnear mnmzaton problems;" Qu. App. Maths., Vol. 2, p DAVIDON, W. C. (1959). "Varable metrc method for mnmzaton," A.E.C. Research and Development Report, ANL-599 (Rev.). DIRAC, P. A. M. (1958). The Prncples of Quantum Mechancs, Oxford: O.U.P. HESTENES, M. R., and STIEFEL, E. (1952). "Methods of conjugate gradents for solvng lnear systems," /. Res. N.B.S., Vol. 49, p. 49. HOUSEHOLDER, A. S. (1953). Prncples of Numercal Analyss, New York: McGraw-Hll. LEVENBERG, K. (1944).."A method for the soluton of certan non-lnear problems n least squares," Qu. App. Maths., Vol. 2 p MARTIN, D. W., and TEE, G. J. (1961). "Iteratve methods for lnear equatons wth symmetrc postve defnte matrx," The Computer Journal, Vol. 4, p POWELL, M. J. D. (1962). "An teratve method for fndng statonary values of a functon of several varables," The Computer Journal, Vol. 5, p ROSENBROCK, H. H. (I96). "An automatc method for fndng the greatest or least value of a functon," The Computer Journal, Vol. 3, p SHAH, B. V., BUEHLER, R. J., and KEMPTHORNE, O. (1961). "The method of parallel tangents (Partan) for fndng an optmum," Offce of Naval Research Report, NR (. 2). Book Revew (.contnued from p. 143) here. It seems entrely wrong that modern source languages should be domnated by the sequental regmes of early machne codes, and any move towards conventonal mathematcal forms s to be welcomed. Agan, the approach here seems rather tentatve and some major benefts are lost. I thnk t s preferable to make sequental codng subordnate to defntons rather than the other way round: here les the key to the very mportant problem of ntegratng the translator wth a realstc operatng system. Amongst the other problems tackled are the handlng of complex varables,- recurrence relatons, and drect transfer of control to parts of'the program not smlarly accessble n ALGOL. It may be regarded as a trbute to ALGOL that an attempt has been made to graft such a system onto the same tree. At the same tme t- must accept a measure of responsblty for the fact that the above deas were not more fully developed and n use three years ago. J. K. ILIFFE. 168