Steepest descent method implementation on unconstrained optimization problem using C++ program

Transcription

1 IOP Conerence Series: Materials Science and Engineering PAPER OPEN ACCESS Steepest descent method implementation on unconstrained optimization problem using C++ program o cite this article: H Napitupulu et al 8 IOP Con. Ser.: Mater. Sci. Eng. Related content - Native Language Integrated Queries with CppLINQ in C++ V Vassilev - he ininite-ranged spin glass with m- component spins J R L de Almeida, R C Jones, J M Kosterlitz et al. - Stability conditions o generalised Ising spin glass models E J S Lage and J R L de Almeida View the article online or updates and enhancements. his content was downloaded rom IP address on 8/9/8 at :

2 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// Steepest descent method implementation on unconstrained optimization problem using C++ program H Napitupulu, Suono, I Bin Mohd, Y Hidayat, S Supian Department o Mathematics, Faculty o Mathematics and Natural Sciences, Universitas Padjadjaran, Indonesia Institute or Mathematical Research (INSPEM, Universiti Putera Malaysia, Selangor, Malaysia Department o Statistics, Faculty o Mathematics and Natural Sciences, Universitas Padjadjaran, Indonesia * napitupuluherlina@gmail.com Abstract. Steepest Descent is nown as the simplest gradient method. Recently, many researches are done to obtain the appropriate step size in order to reduce the objective unction value progressively. In this paper, the properties o steepest descent method rom literatures are reviewed together with advantages and disadvantages o each step size procedure. he development o steepest descent method due to its step size procedure is discussed. In order to test the perormance o each step size, we run a steepest descent procedure in C++ program. We implemented it to unconstrained optimization test problem with two variables, then we compare the numerical results o each step size procedure. Based on the numerical eperiment, we conclude the general computational eatures and weanesses o each procedure in each case o problem.. Introduction Consider the ollowing nonlinear unconstrained minimization problem ind * R n such that ( * = min ( R n ( where : R n R. Steepest descent method is the simplest gradient method or solving unconstrained optimization which is designed by Cauchy in 87. his gradient method search along the negative gradient unction, can ensure a reduction o the objective unction as long as the current iterate point is not a stationary point ([]. Many researches have been done which is concern in inding the better and more appropriate step size procedure which aecting to the signiicance o minimizing the objective unctions. In this paper we give some reviews the properties o steepest descent method, its development, and compare the numerical results o steepest descent step size rom some literatures on solving global optimization problem. We run a steepest descent in C++ program to test the perormance o tested step size procedure, then compare the numerical results obtained o program eecution. Based on the numerical eperiment, the eatures as well as the weanesses o each step size procedure in each case o problem are concluded. Content rom this wor may be used under the terms o the Creative Commons Attribution. licence. Any urther distribution o this wor must maintain attribution to the author(s and the title o the wor, journal citation and DOI. Published under licence by Ltd

3 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x///. Steepest Descent Method Steepest descent method is the simple minimizing gradient method or solving nonlinear equations since it is based on the linear approimation o aylor series ( + δ ˆ ( + δ = ( + ( δ ( he term ( δ is the directional derivatives o at. he step δ is a descent direction i the directional derivative is negative, which guarantees that the unction could be reduced along this direction. Furthermore, the step δ is need to be chosen to mae the linear directional derivative ( δ as negative as possible, which give the maimum reduction o but no consuming too much time in mae a choice. Assume that a unction ( is continuous in the neighborhood o point, d = g is the steepest descent direction at point and change δ in given by δ = αd. A small positive constant α is called step size, which its selection mae an eect to the reduction value o (. By solving one dimensional global optimization problem min φ ( α = ( + αd ( α> the maimum reduction in ( can be obtained. he procedure o steepest descent iteration is perormed by ormula = + + αd ( Starting with an initial point, a direction d = g and the step size α that minimizes ( + αd can be determined so that the net point = + α d is obtained. he procedure in ( is repeated until the convergence is achieved or the value α d is suiciently small. he algorithm o steepest descent is presented as ollows. ALGORIHM SEEPES DESCEN given : starting point domain set : = repeat. Compute steepest descent direction d = g. Update step size α = min φ ( α = ( + αd α>. Update : = +α d + until convergence or stopping criterion is satisied. Steepest Descent Step Size he step size in steepest descent method plays the important role in obtaining the minimizing the objective unction. he step size o each iteration α, must be eectively chosen to give a signiicant reduction o unction value, while the eiciently timing o choosing it also be optimized. he early step size procedure o steepest descent is considered as step size which obtained by eact line search, { α g } α = arg min ( + ( ( α. But i the steepest descent direction is used with eact line search step size would bring to the zigzag behavior which mae the convergence is very slowly. Aaie ([] analyzing that by this search

4 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// procedure the converges is linearly and badly aected by ill-conditioning, by shown that the two asymptotic directions are in a two dimensional sub space spanned by two eigen vector o the Hessian matri. Greenstadt ([] studied the eiciency o steepest descent method and shown the bound o the ratio between the reductions obtained by Cauchy step and by the Newton s method, and the rate o convergence o steepest descent was proved by Forsythe ([]. Besides using eact line search, step size α could be computed by type o ineact line search methods, or instance Armijo condition ([5], Goldstein condition ([6], Wole condition ([7], Powell ([8], Dennis and Schnabel ([9], Fletcher ([], Potra and Shi ([], Lemaréchal ([], Moré and huente ([] and the others. Steepest descent method with mentioned conditions is always convergent theoretically, which is will not terminate unless the stationary point is ound. However, step size computed by those ineact line search procedure also appearing the bad zigzag behavior as well as in eact line search procedure. Considering the weaness o eisting step sizes, Barzilai and Borwein ([] proposed the the step size along negative gradient direction rom a two point approimation to the secant equation rom quasi-newton s method so called BB method. I the dimension is two, Barzilai and Borwein established R-superlinear convergence result or the method and their analyses indicated that the convergence rate is aster. For the general n dimensional strictly conve quadratic unction, Raydan ([] proved that the two-point step size gradient method is globally convergent, and the convergence rate is R linear (Dai and Liao ([5]. For the non-quadratic case, Raydan ([] incorporated the globalization scheme o the two point step size gradient method using the traditional technique o nonmonotone line search by Grippo et al ([6]. he resulted algorithm is competitive and sometimes preerable to several amous conjugate gradient algorithm or large scale unconstrained optimization. Due it simplicity and numerical eiciency the two-point step size gradient method has received many studies. hey have been successully applied to obtaining the local minimizers o large scale real problems ([7]. he algorithm o Raydan ([5] was urther generalized by Birgin et al ([8] or the minimization o dierentiable unction on closed conve sets, yielding an eicient projected gradient method. Eicient projected algorithm based on BB -lie methods have also been designed (Seraini et al ([9] and Dai and Fletcher([] or special quadratic programs arising rom training support vector machine, that has a singly linear constraint in addition to bo constraints. he BB method has also received much attention in inding sparse approimation solution to large undetermined linear systems o equations rom signal/ image processing and statics or eample in Wright et al ([]. However Fletcher ([] shown that or some problems o non-quadratic unction this method may very slow. here are many eisting step sizes modiication, some o them are easy to be applied but some are satisied the complicated algorithm. Following are some step size procedure which is simple to be applied to steepest descent method. able. List o Some Recent Simple Step Size No Step Size Remar g g Step size method by Cauchy (87 α = g which is computed by eact line search H g (C step size. Given s >, β, σ (,, α ma { s, s, s,...} = β β his procedure o step size using ineact line search so called Armijo s line search such that (A step size. ( + α d ( + σα g d Given β, σ (,, ˆ α = his procedure is so called Bactracing line search (B step size, this ineact line α = βαˆ search is use just the suicient decrease

5 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// 5 such that ( + α d ( + σα g d s y α = (BB and y s α = (BB where s y s = y = g g g g α α = ( d ( + α + α g g ( condition to terminate the line search procedure. hese two point step sizes are method o non-line search procedure which is namely ater the inventors, Barzilai and Borwein s Formula. hese ormula reduce the value o objective unction and gradient and improved the convergence rom linear to R superlinear. his step size is so called elimination line search (EL step size, which is estimate the step size without compute the Hessian.. Numerical Eperiment In this section, several step size procedure o the steepest descent method shown in able are tested computationally. he programming is using Visual C++ 6., which is applied to testing unction o global optimization given in the ollowing.. Beale Function min (, (.5 (.5 (.65 = s. t..5,.5, = (,. Bohachevsy I Function min (, = +. cos( π. cos( π +.7 s. t. 5, 5, = (.,.. Bohachevsy II Function min (, = +. cos( π cos( π +. s. t. 5, 5, = (.,.. Booth Function min (, ( 7 ( 5 = s. t.,, = (, 5. Matyas Function min (,.6 = ( +.8 s. t., 6. Rosenbroc Simpliied Function min (, =.5 +, = (, ( ( s. t., =.6,.9, (

6 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// 7. Si Hump Bac Camel Function min (,. 6 = + + s. t.,, = (, 8. Rastrigin Function min (, = + cos(8 cos(8 s. t.,, = (, 9. hree Hump Bac Camel Function 6 min (,.5 = s. t.,, = (,. reccani Function min (, = s. t.,, = (,. wo Dimensional Function c=.5 min (, = ( +.5 sin ( π + (.5sin ( π s. t.,. wo Dimensional Function c=. min (, = ( +. sin ( π + (.5 sin ( π, = (.5,.5 s. t.,. wo Dimensional Function c=.5 min (, = ( +.5sin ( π + (.5sin ( π, = (, s. t., =,. Goldstein and Price Function min (, ( = ( , ( ( ( ( + ( s. t.,, = (, By reerring to [] or Armijo procedure (A, s : =, β : =.75, σ : =.8, and or Bactracing procedure σ : =.. he initial step size or Barzilai-Borwein step size (BB and Barzilai- Borwein step size (BB is., initial step size or elimination line search is α : =.. he comparison o Numerical result will be based on: time eecution, number o total iteration, total percentage o unction, gradient and Hessian evaluation and the most decreased value o objective unction obtained. Stopping criteria are α d 6 or d 6. Following are the abbreviations used in o numerical results o able. 5

7 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// Abbreviation in column one: C = Cauchy step size A = Armijo step size B = Bactracing step size BB = Barzilai & Borwein step size BB = Barzilai & Borwein step size EL = Elimination line search step size Abbreviation in row one: Iter = percentage o the total number o iteration until termination ime = percentage o the total running time until the termination in second Ft = percentage o the total number o unction evaluation until termination Gt = percentage o the total number o gradient evaluation until termination Ht = percentage o the total number o Hessian evaluation until termination able. Numerical Results No. Step ime Iter Ft Gt Ht. C (.99995, e A (.9999, e B (.9999, e BB (.99985, e BB (.99987, e EL (.9999, e Step C (.658, A (.6965e-6, B (.95e-6, BB ( e-7,.6e e BB (9.7959e-7, EL ( e-7, Step C (.68,.757e A (.58e-6, 9.9e e B (.898e-, e e BB (8.8e-7,.956e-7.5e BB (.7968e-7, -5.9e-8.599e EL (-5.56e-7,.77e e Step Iter Ft Gt Ht ime C (.99999,. 7.95e A (.99999, e B (-.868e-, BB ( , 6.97e-..8 BB ( ,.56e-.77. EL ( ,.87e Step C (.9e-5,.9e-5.86e A (.76e-5,.76e-5.568e B (.76e-5,.76e-5.568e BB (.9e-,.9e e BB (.889e-,.889e-.9699e

8 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// EL (.867e-,.867e-.796e Step ime Iter Ft Gt Ht C (-.76, A (-.698, B (-.76, BB (.76, BB (-.76, EL (-.76, Step C (, 8.878e A (.99999, e B (,..889e BB ( , e BB (.99999, e EL ( , e Step C (.67, A (.67, B (.8988, BB (.67, BB (.67, EL (.67, Step C (-.7755, A (-.7755, B (-.7755, BB (-.7755, BB (-.7755, EL (-.7755, Step C (.89e-8,.95e-7.558e-.7.. A (7.6e-9, e e B (-,.85e-5.5e BB (-6.678e-8,.65e-6.888e-.. BB (-6.678e-8,.65e-6.888e-.. EL (.5576e-7,.78967e e Step C (.55, e A (.55, e B (.8658,.7.75e BB (.59, e BB (.56, e EL (.8657,.7.5e Step C (.769, A (.596, e B (.87779, e BB (.87779, e BB (.87779, e EL (.87779, e Step ime ime Iter Iter Ft Ft Gt Gt Ht Ht 7

9 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x///. C ail ail ail ail ail ail ail A (.55,.8.879e B (.977, BB (.97, BB (.975, EL (.979, Step C ( , A ( , B (-9.98e-6, BB (-,- ail ail ail ail ail ail BB (-,- ail ail ail ail ail ail EL (-,- ail ail ail ail ail ail From the numerical results o able we summarize the number o successul (ns, number o local minimizer obtained (nl, and number o ailure (n presented in able. able. Summary o Numerical Results Step Size ns nl n C 7 6 A B 9 5 BB BB 9 EL 9 From the numerical result obtained, the A method and BB method are the method with the most successul among others or the case o two dimensional global optimization given. A method never ailed in solving the testing problems but have times go into another local minimizer. he C method is the least in obtaining the minimizer successully with one ailure. he B, BB and EL method reach the same level in successully obtain the minimizer. he B method reach without any ail, but BB and EL have ail. From this numerical eperiment we can conclude that A is better than others based on the number o minimizer obtained in the testing eample given in given initial condition. 5. Conclusion he advantages and disadvantages o steepest descent method are, this method is sensitive to the initial point; this method has descent property and it is a logical starting procedure or all gradient based methods. Since = α ( as ( * =, so + approaches the minimizer rather slowly, in act in a zigzag way. It is not economical to do linear searches thoroughly. All that is necessary to do is to obtain reduction in unction values at successive iterates/ According to the numerical eperiment o the given test unction, we can conclude that in the case o two dimensional unconstrained global optimization problem given, the A and BB is considered as the better method among others and C method is good enough, or the case o testing eample given with given initial condition. Acnowledgments Authors wishing to the Rector, Director o DRPMI, and Dean o FMIPA, Universitas Padjadjaran, which provides a grant program o the Academic Leadership Grant (ALG and a grant program o Competence Research o Lecturer o Unpad (Riset Kompetensi Dosen Unpad/RKDU. 8

10 IORA-ICOR 7 IOP Con. Series: Materials Science and Engineering (8 doi:.88/ x/// Reerences [] Yuan Y 8 Step-sizes or the gradient method In hird International Congress o Chinese Mathematicians. Part, AMS/IP Stud. Adv. Math.,, Pt Amer. Math. Soc., Providence, RI [] Aaie H 959 On a successive transormation o probability distribution and its application to the analysis o the optimum gradient method Ann, Inst. Statist. Math. oyo pp -7 [] Greenstadt J 967 On the relative eiciencies o gradient methods Math. Comp. pp 6-67 [] Forshyte G E 968 On the Asymptotic directions o the s-dimensional optimum gradient method Numerische Mathemati pp [5] Armijo L 966 Minimization o unction having Lipschitz continuous irst partial derivatives Paciic J. o Mathematics 6 pp - [6] Goldstein A A 965 On steepest descent. SIAM Journal on Control, vol., 7-5. [7] Wole P 968 Convergence conditions or ascent methods SIAM Review pp 6-5 [8] Powell M J D 976 Some global convergence properties o a variable-metric algorithm or minimization without eact line searches SIAM-AMS Proceedings, Philadelphia, 9 pp 5-7 [9] Dennis J E and Schnabel R B 98 Numerical Methods or Unconstrained Optimization and Nonlinear Equations Prentice-Hall Englewood Clis New Jersey [] Barzilai J and Borwein J 988 wo-point step size gradient methods IMA J. o Numer. Anal. 8-8 [] Potra F A and Shi Y 995 Eicient line search algorithm or unconstrained optimization JOA 85, no. pp [] Lemaréchal C 98 A view o line search in A. Auslander, W. Oettli and J. Stoer (Eds.. Optimization and Optimal Control Springer Verlag, pp [] Moré J and huente D J On line search algorithms with guaranteed suicient decrease. Mathematics and Computer Science Division Preprint MCS-P Argonne National Laboratory Argonne. [] Raydan M 99 On the Barzilai and Borwein choice o steplength or the gradient method IMA J. Numer. Anal. pp -6 [5] Dai Y H, and Liao L Z.. R-linear convergence o the Barzilai and Borwein gradient method. IMA. J. Numer. Anal. 6, - [6] Grippo L, Lampariello F, and Lucidi S 986. A nonmonotone line search technique or Newton s Method SIAM J. Numer. Anal. pp77-76 [7] Luengo F and Raydan M Gradient method with dynamical retards or large scale Optimization problems Electronic ransactions on Numer. Anal. vol 6 Kentstate University [8] Birgin E G, Floudas C A, Martínez J M Math. Programming; Heidelberg 5 Iss. pp 9-6 [9] Seraini G, Zanghirati and Zanni L 5 Gradient projection methods or quadratic programs and application in training support vector machine. Optim. Meth. Sotware pp 7-7 [] Dai Y H and Fletcher 6 New algorithms or singly linearly constrained quadratic subject to lower and upper bound Math Program Ser A. 6, pp - [] Wright J S, Nowa R D, and Figueiredo M A 9 Sparse Reconstruction by separable approimation. IEEE ransaction on Signal processing. 57, 79-9 [] Fletcher R. On the Barzilai-Borwein Method Numer. Anal. Report NA/ [] Wen G K, Mamat M, Mohd B I, and. Dasril Y A novel Step Size Selection Procedures or Steepest Descent Method Appl. Math. Sci. 6 No. 5, pp