A Novel of Step Size Selection Procedures. for Steepest Descent Method

Transcription

1 Applied Mathematical Sciences, Vol. 6, 0, no. 5, A Novel of Step Size Selection Procedures for Steepest Descent Method Goh Khang Wen, Mustafa Mamat, Ismail bin Mohd, 3 Yosza Dasril Department of Physical and Mathematical Science Faculty of Science, Universiti unu Abdul Rahman, Malaysia. Department of Mathematics Universiti Malaysia erengganu, Kuala erengganu, Malaysia. 3 Department of Industrial Electronics Faculty Electronics and Computer Engineering, Universiti enial Malaysia Melaa (UeM, Malaysia. Abstract: It is well nown that the classical steepest descent method which uses the exact line search procedure in determining the step size is converge very slowly to the solution. By the way, several effective inexact line search procedures have been proposed to overcome the weaness. Beside that, Barzilai and Borwein (BB have proposed two surprising non-line search procedures in determining the step size for steepest descent method which has been proved to be R-superlinearly convergent for convex quadratic in two-dimensional space. However, in order to give the greatest possible reduction to the objective function along the search direction, we have introduced a Newton-lie exact line search procedure. First in this paper, we will elaborate several well nown step size selection procedures in more detail. hen, a numerical performance comparison of all selected step size selection procedures has been done. he results showed the Newton-lie exact line search has performed better than others well nown procedures.

2 508 Goh Khang Wen et al. Introduction he classical steepest descent method which design by Cauchy (847 can be considered among the most important procedures for minimization of real-valued n function defined on R. he steepest descent step size appeared in x x λ d + = +, (. which the step size λ is obtained using exact line search [4] λ = arg min f ( x λ > 0 + λd, (. λ min{ λ f ( x + λd = 0} 0 = λ > (.3 respectively where d f ( x =. However, for solving some complicated optimization problems, it is difficult to compute the step size λ using (. and (.3 in practical computational and some time even impossible to compute it [5,3,4]. herefore, there are several inexact line search condition have been introduced, such as Armijo condition [], Goldstein condition [7], or Wolfe condition [] in determined the step size for steepest descent method. It is easy to show that the steepest descent method with those condition, is always convergent and theoretically the method will only terminate after a stationary point is found. According to the previous study of the inexact line search condition implemented to steepest descent method [,9,3], we found that Armijo line search and Bactracing line search [0] are easier to apply and more effective compared to others. Line search procedure is a useful and efficient technique for solving unconstrained optimization problems, especially for solving small and middle scale problems, such as determines a step size for steepest descent method. However, some researcher mentioned that if line search procedure is constructed every iteration of the approximation, it will lead a significant amount of computational cost. herefore, in order to reduce the evaluations of objective functions and gradient, several researchers have tried to avoid using line search in their designed algorithms. In year 988, Barzilai and Borwein (BB have derived two-point step size for the steepest descent approximating the secant equation [3]. hey have proposed two new step size formulae to compute the step size λ for steepest descent method which have been proved to be R-superlinearly convergent for convex quadratic in two dimension space.

3 Steepest descent method 509 In year 989, JM method [8] which improve the BB methods by taes the average value of both BB methods. Even though there are several inexact line searches and non-line search procedures have been proposed, but in the previous studies on line search, we have found that only the exact line search can gives the greatest possible reduction to the objective function along the search direction. All the exact line search procedures which must satisfy the condition (. are always computed by using (.3. But the previous researchers have shown that the weaness of the (.3 in computational for determining the step size λ [5,3,4]. herefore, according to Newton-Raphson method, we have introduced an alternative Newton-lie exact line search [5,6] and we have proved that it is practically computational available in our preliminary study of the method. his paper is organized as follows. In Section, we elaborate several mentioned procedure for determining the step size λ for steepest descent method. In Section 3, we describe the Newton-lie exact line search procedure and present the algorithm of steepest descent method. Section 4 contains numerical results of the testing examples for implementing the methods discuss in this paper. he conclusion which ends this paper is discussed in Section 5.. Step Size How to select the suitable step size for steepest descent method? In the early studies of this method, there are a varieties of research have been done in determining a suitable step size for the algorithm in solving optimization problems. Several well nown procedures which mentioned in previous section, have been selected and will be discussed in this Section.. Cauchy s step size (exact line search he classical and oldest steepest descent step size λ which was designed by Cauchy, is computed as g g λ =, (. g Ag where g = f x and A = f (. he equation (. can be proved as follows. ( x

4 50 Goh Khang Wen et al Consider the quadratic model problem given by Since g = f ( x, we have n min f ( x = x H x b x, x R. (.. g = H x b. (.. he step size is computed by using the exact line search in (3., to give f ( x λ g = H ( x λ g b = 0 (..3 By introducing (.. into (..3, we obtain g λ H g = 0. (..4 Since step size λ R, (..4 can be written as (..5 g g = λ g H g from which the equation (. can be obtained. Even the formula of the step size (. can be proved by using quadratic model problem, but since there are no any suggested formula in solving complicated optimization problem and it is very hard to compute the step size λ using (.3, we still using (. as Cauchy s step size formula in solving complicated optimization problems as listed in Section 4... Armijo Line Search (inexact line search Zhen (006 notices that among several well nown inexact line search procedures which introduced by previous researchers, Armijo line search [] is one of the most useful and easy to be implemented in computational. he Armijo line search rule is described as follows. Given s > 0, β (0,, σ (0, and λ = max{ s, sβ, sβ, K }

5 Steepest descent method 5 such that f ( x + λ d f ( x σλ g d (. where d g f ( x = =..3 Bactracing Line Search (inexact line search Neculai [9] and Burachi [] say that bactracing line search is more simple and effective than other inexact line search methods [0]. Actually bactracing line search is a method obtained from modification of Armijo line search, and its procedure is as follows. Procedure Bactracing :! his procedure computes the step size λ using Bactracing line search. Set t =.. while f ( x + td > f ( x + σtg d, do. t = t /. 3. λ = t..4 Barzilai and Borwein (BB method Barzilai and Borwein [3] derive two-point step size for the steepest descent routine by approximating the secant equation underlying quasi-newton method, in which they have determined two step sizes λ by minimizing λ g and it s symmetry λ g with respect to λ, where = x x and g = g g, from which we obtain g λ = (.3 g g and λ = (.4 g.4 Mohd and Jaafar (JM method

6 5 Goh Khang Wen et al Mohd and Jaafar [8] have improved the BB methods in determining step size λ for steepest descent by taing the average value of both BB methods with λ is given by = g g + g g λ. (.5 3. Newton-lie exact line search All the exact line search procedure which must satisfied the condition of (. is always computed using (.3. But the previous researchers have showed the weaness of the (.3 in computational for determining the step size λ [5,3,4]. herefore, according to Newton-Raphson method, in this section an alternative Newton-lie exact line search is suggested and we have proved it is available in practical computational in our preliminary study of the method. hus, instead of using (.3 to satisfied the condition of (., we use the approximation of Newton-Raphson s idea to obtained the step size λ through the routine ϕ ( λ λ + = λ, ϕ ( λ (3. where respectively. and he procedure of this alternative exact line search can be described as follows. procedure compute., :! his procedure computes λ. i = 0. λi + ϕ λ ( = λi ϕ ( λ 3. while λ i+ λ i ε, do 3. i = i + ϕ ( λ 3. λi + = λi ϕ ( λ 4. λ = λ i return.

7 Steepest descent method Algorithm of Steepest Descent Method + Procedure ASDM( m, n Z,, :.! his procedure compute implements the algorithm of steepest descent method where m is the! maximum number of iteration, n is the number of variable, x 0 is the initial point, ε is a tolerance.. = 0. d = f ( x 3. converge = false 4. while d ε and m and not converge do 4.. Use any step size procedure to compute the step size λ. 4.. x = x + + λ d 4.3. if x + x ε, then x : = x +! he minimizer converge = true else = d = f ( x 5. return. 4. Numerical Example In this section, we report a comparison results between those selected procedures in Section and the Newton-lie exact line search applied into steepest descent method. 4. esting Problems. f ( x, x = x + 3x, x 0 = (5,5. f ( x, x = 4x + x x x, x 0 = (, 3. six hump came bac function 6 4 x, x 4x.x + xx 4x f ( x = + x, x = (.6, hree hump came bac function 6 4 x 0 f ( x, x = x.05x + xx + x, x = (.6, Booth function f ( x, x 0 = (0.4,.6, x = ( x + x 7 + (x + x 5

8 54 Goh Khang Wen et al 4 x x x 0 6. f ( x, x = + 0.x +, x = (, f ( x, x = (x x x + 6( x x + x, x 0 = (, 8. Goldstein-Price function f ( x, x = g( x h( x g ( x = [ + ( x + x + (9 4x + 3x 4x + 6x x + 3 ], x 0 h ( x = [30 + (x 3x (8 3x + x + 48x 36x x + 7 ], x = (,. x, x 0.5x + 0.5( cos(x 9. f ( x = + x, x 0 = (, 0. he two dimension function.. f ( x x, x = [ x + csin(4πx x] + [ x 0.5sin(π ], = 0. f ( x x, x = [ x + csin(4πx x] + [ x 0.5sin(π ], = 0. 5 f ( x x, x = [ x + csin(4πx x] + [ x 0.5sin(π ], = Numerical Results 0 c, x = (6, c, x 0 = (0,0 0 c, x = (, Each step size procedure which has been discussed in previous section has been implemented into steepest descent method and the algorithms have been programmed into visual C++ language. Our testing problems and the initial points used are shown in Section 4.. For each problem, the limiting number of iteration is set to 00,000 6 and the tolerance ε = 0. For Armijo line search procedure, we use s =, β = 0. 75, σ = 0.38 suggested by Zhen [4] and bactracing line search procedure, we use σ = proposed by Neculai [9]. Beside that, Vrahatis [] have noticed that, a small step size has to be chosen in order to avoid oscillation and to guarantee the steepest descent convergence. herefore, we use the initial step size λ = for our Newton-lie exact line search procedure. P Cauchy Armijo Bactracing BB BB JM Newton F F F F F F F F F F F 0 5 F 0 F able : Iteration Number in Solving Problems

9 Steepest descent method 55 he number of iteration which used to solve each problems by all the selected step size procedures are shown in able, where BB, BB, JM and Newton stand for (.3, (.4, (.5 and Newton-lie exact line search procedure respectively. he bolded numbers in the able represent the least of iteration number among all the procedures for each problem. he symbol F stand for failure in which mean that the algorithm still cannot find the solution when the iteration number reach at 00,000 or the solution determined by the algorithm does not satisfied the minimization properties f x + = f ( x + λ d f ( x f ( x. (4. ( 0 5. Conclusion Stephen and Areila [0] notice that although Newton s method needs less iteration than steepest descent method for obtaining the solution, but labour cost of Newton s method is higher than steepest descent since the Newton s method requires more memory space and expensive in calculating the inverse of the Hessian matrix. Since in our Newton-lie exact line search, the first and second derivatives with respect to λ are defined as and, therefore it does not have the weaness which mentioned above but it does having the fast converge advantage of Newton s method. According to the comparison results shown in Section 4., we can say that the step size obtained by using the Newton-lie exact line search is more effective compared to other step size procedures and even there are three problems it does not obtain the solution with the least iteration, but those number of iteration is not much more than the least one as seeing in able. From the numerical results, we also found that the well nown BB s methods are easy to failure in solving complicated optimization problems. Acnowledgement. he authors are grateful acnowledge financial support from the Government of Malaysia and Universiti Malaysia erengganu through Fundamental Research Grant Scheme (Vot 5900.

10 56 Goh Khang Wen et al References []. Armijo L., (966. Minimization functions having Lipschitz continuous first partial derivative, Pacific J. Math 6, -3. []. Burachi, R., Drummond, L. M. G., Iusem, A. N. and Svaiter, B. F. (995. Full convergence of the steepest descent method with inexact line searches, Optimization [3]. Barzilai J. and Borwein J.M., (988. wo point step size gradient method. IMA J. Numer. Anal., 8, [4]. Curry H.B., (944. he method of steepest descent for nonlinear minimization problems, Quart. Appl. Math.,, [5]. Goh Khang Wen and Ismail Bin Mohd, (006 Analyze steepest descent using maple, Computer Science and Mathematics Symposium 006 (CSMS006, Kolej Universiti Sains dan enologi Malaysia. [6]. Goh, K. W. & Ismail, B. M. (007. An Alternative Newton-lie Exact Line Search for Steepest Descent Method. Proceedings of 3 rd International Conference on Research and Education in Mathematics 007, Universiti Putra Malaysia, Applied Mathematics and Mathematics Education, page -7. [7]. Goldstein A. A., (965. On steepest descent, SIAM Journal of Control, 3, [8]. Ismail Bin Mohd and Azmi Bin Jaafar, (990. A modification of wo-point step size gradient method for unconstrained optimization, Sains Malaysiana 9(4, 97-0.

11 Steepest descent method 57 [9]. Neculai Andrei, (004. A new descent gradient method for unconstrained optimization, Research report, Research Institute for Informatics, Romania. [0]. Stephen G. Nash and Ariela Sofer, (996. Linear and nonlinear programming, George Mason University, he McGraw-Hill Companies, Inc.. []. Vrahatis M.N., Androulais G.S., Lambrinos J.N., Magoulas G.D., (000. A class of gradient unconstrained minimization algorithms with adaptive stepsize, Journal of Computational and Applied Mathematics, 4, []. Wolfe P., (969. Convergence conditions for ascent methods, SIAM Rev. : [3]. Zhen-Jun Shi and Jie Shen, (006a. On step-size estimation of line search methods, Applied Mathematics and Computation, 73, [4]. Zhen-Jun Shi and Jie Shen, (006b. Convergence of descent method with new line search, Journal Applied Mathematics and Computing, 0(-,

12 58 Goh Khang Wen et al Received: ed: January, 0