Hybridization of accelerated gradient descent method

Transcription

1 ORIGINAL PAPER Hybridization of accelerated gradient descent method Milena Petrović 1 Vladimir Rakočević 2,3 Nataša Kontrec 1 Stefan Panić 1 Dejan Ilić 3 Received: 7 December 2016 / Accepted: 11 December 2017 Springer Science+Business Media, LLC, part of Springer Nature 2017 Abstract We present a gradient descent algorithm with a line search procedure for solving unconstrained optimization problems which is defined as a result of applying Picard-Mann hybrid iterative process on accelerated gradient descent SM method described in Stanimirović and Miladinović (Numer. Algor. 54, , 2010). Using merged features of both analyzed models, we show that new accelerated gradient descent model converges linearly and faster then the starting SM method which is confirmed trough displayed numerical test results. Three main properties are tested: number of iterations, CPU time and number of function evaluations. The efficiency of the proposed iteration is examined for the several values of the correction parameter introduced in Khan (2013). Milena Petrović milena.petrovic@pr.ac.rs Vladimir Rakočević vrakoc@sbb.rs Nataša Kontrec natasa.kontrec@pr.ac.rs Stefan Panić stefanpnc@yahoo.com Dejan Ilić ilicde@ptt.rs 1 Faculty of Sciences and Mathematics, University of Priština, Lole Ribara 29, Kosovska Mitrovica, Serbia 2 Serbian Academy of Sciences and Arts, Kneza Mihaila 35, Belgrade, Serbia 3 Faculty of Sciences and Mathematics, University of Niš, Višegradska 33, Niš, Serbia

2 Keywords Line search Gradient descent methods Newton method Convergence rate Mathematics Subject Classification (2010) 65K05 90C30 90C53 1 Introduction and preliminaries The goal we are dealing with is to find an efficient algorithm for solving the unconstrained optimization problem min f(x), x R n, (1.1) where R n is the set of n-tuples with elements in the set of real numbers R and f : R n R is an objective function. For the function f : R n R we suppose that it is uniformly convex and twice continuously differentiable. The very common form of iterative processes for unconstrained optimization with multivariable functions is described with general iteration x k+1 = x k + t k d k. (1.2) Here, x k+1 is the next iterative point, x k is the current iterative point, variable t k > 0 is an iterative step length, and d k is an iterative search direction. The ways of defining descending search direction and the value of the step-size parameter determine the effectiveness and the efficiency of considered gradient descent method. In [26], an accelerated gradient descent scheme, so-called the SM method, is presented and described with the following iteration x k+1 = x k γk 1 t k g k, (1.3) where g k and the γ k present, respectively, the gradient and the scalar approximation of the Hessian of the objective function f at the k-th iterative point. For the search direction d k in iterative processes defined by (1.2), it is usually expected to satisfy the descent condition: gk T d k < 0. (1.4) There are interesting choices for defining the search direction vectors proposed in [12, 27]. In [26], the authors simply define vector direction as d k = g k and so defined vector does satisfy condition (1.4). Another equally important element in iteration (1.2) is the value of the step length t k. A very common way for computing parameter t k is by applying one of the procedures from the class of exact or inexact line search algorithms [3, 7, 8, 11, 15, 21, 22, 25, 28]. In the SM iteration (1.3), step length parameter is determined by the Armijos s Backtracking inexact line search technique. Since the SM method has the features of quasi-newton and modified Newton methods, the estimation to the inverse Hessian matrix of the objective function in SM iteration is given by the scalar diagonal matrix γk 1 I where scalar γ k = γ(x k,x k 1 ) R, denoted as the acceleration parameter,isderived usingthetaylorexpansionappliedonthesm scheme. It is well-known that the good convergence speed of the Newton method is closely related to the properties of the Hessian. In order to avoid computing the Hessian, the basic

3 aim of the quasi-newton methods and the modified Newton methods is to find appropriate Hessian s approximation (or approximation of its inverse) which will preserve rapid convergence of a certain method. In [26], acceleration properties of parameter γ k are confirmed. Even so, better characteristics of the proposed SM iteration in comparison to the classical gradient descent method, (GD), and the accelerated gradient descent model, (AGD), introduced in [1] are numerically certified. Similar approaches in defining acceleration parameter for several gradient descent iterative processes which resulted with the good performance characteristics are given in [18, 20]. In the next section, we describe the determination of the acceleration parameter of the iterative process introduced in this paper. On the other hand, in [10] a Picard-Mann-Ishikawa hybrid iterative process is defined with next three relations x 1 = x R, x k+1 = Ty k, (1.5) y k = (1 α k )x k + α k Tx k,k N which can be summed in aggregated form as x k+1 = T [(1 α k )x k + α k Tx k ], k N. (1.6) In (1.5), T : C C is a mapping defined on nonempty convex subset C of a normed space E, x k and y k are the sequences determined by the iterations (1.5). The sequence of positive numbers {α k } (0, 1) from (1.5) is denoted in this paper as the correction parameter. The author in [10] for numerical experiments uses a chosen set of constant values for this parameter (α = α k (0, 1) k) and shows that the process (1.5) converges faster than the Picard, Mann and Ishikawa iterative processes from [9, 13, 23]. These three mentioned schemes are defined with the next sets of relations, respectively: { u1 = u C, u k+1 = Tu k,k N, { v1 = v C, v k+1 = (1 α k )v k + α k Tv k,k N, z 1 = z C, z k+1 = (1 α k )z k + α k Tyk, y k = (1 β k )z k + β k Tz k, k N, where v k,z k and y k are the sequences defined by the proposed expressions, {α k }, {β k } (0, 1) are the sequences of positive numbers, which in the Ishikawa process [9] satisfy the next conditions 0 α k β k 1, k 0, lim k β k = 0, k=1 α k β k =. An interesting approach in defining the step length parameter and vector of the search direction described in [26] as well as thehybridizationofsome knowniterative processes that resulted with better speed feature proposed in [9, 13, 23] are used

4 as a motivation for research presented in this paper. In the second section, a hybrid correction concept applied on the SM method is described. In this section, we also define the HSM method as a result of the hybridization of the SM iteration and present the HSM algorithm. In the third section, we prove the convergence property of the presented HSM model for the set of uniformly convex functions. Right after, in the fourth section, we also confirm the convergence of the hybrid model for the set of strictly convex quadratic functions. In the fifth section, the results of numerical experiments and comparisons are displayed. 2 Hybrid correction of the SM method In order to define a hybrid form of the SM method, for which we expect to have better or at least similar characteristics, assume that mapping T is defined by SM iteration, i.e. Ty k = y k γk 1 t k g k. Then, using (1.5), we are able to derive the next set of relations: x 1 = x R, x k+1 = Ty k = y k γk 1 t k g k, y k = (1 α k )x k +α k Tx k = (1 α k )x k +α k (x k γk 1 t k g k ) = x k α k γk 1 t k g k,k N. (2.1) By replacing the third equation from (2.1) into the second one, we obtain x k+1 = x k (α k + 1)t k γk 1 g k. (2.2) Previous iteration presents hybridization of the SM method, and in this paper, we shortly denote this scheme as the HSM process. The HSM, just like the SM iteration, is the accelerated gradient descent scheme with accelerated parameter γ k.more closely, γ k is the value which generates the scalar diagonal approximation of the Hessian of function f. We determine, further on, accelerated parameter γ k using the Taylor s series in two successive iterative points. With that, we calculate the iterative step length parameter t k using the inexact backtracking line search technique, where for the vector direction we take gradient descent vector, d k = g k : Algorithm 1 The backtracking line search starting from t = 1 Require: Objective function f(x), the direction d k of the search at the point x k and numbers 0 <σ <0.5andβ (0, 1). 1: t = 1. 2: While f(x k + td k )>f(x k ) + σtg T k d k,taket := tβ. 3: Return t k = t. In order to complete defining the HSM iterative process, we need to compute the value of the acceleration parameter γ k. More accurate, approximation of the Hessian of the objective function f is given by the diagonal matrix 2 f(ξ)= γ k+1 I,

5 in which scalar γ k+1 = γ(x k+1,x k ) we compute by applying Taylor series on function f at the point x k+1 using the features of (2.2). In other words, the next state holds f(x k+1 ) = f(x k ) (α k +1)t k gk T γ k 1 g k (α k+1) 2 tk 2 1 (γk g k ) T 2 f(ξ)γk 1 g k. (2.3) In the previous equation, variable ξ [x k,x k+1 ] is defined as ξ = x k + β(x k+1 x k ) = x k β(α k + 1)t k γk 1 g k, 0 β 1. (2.4) Since we assume that the point x k is close enough to the point x k+1,wetakeβ = 1 in (2.4) and this results to ξ = x k+1. Now, we can justifiably do the next replacement 2 f(ξ)= γ k+1 I. Returning the last two estimations into the (2.3), we obtain f(x k+1 ) f(x k ) (α k + 1)t k γk 1 g k (α k + 1) 2 tk 2 γ k+1γk 2 g k 2. (2.5) Directly from (2.5) we derive the value of the acceleration parameter in the (k+1) th iteration [ γ k f(xk+1 ) f(x k ) ] + (α k + 1)t k g k 2 γ k+1 = 2γ k (α k + 1) 2 tk 2 g k 2. (2.6) We will show that the acceleration parameter γ k has to be a positive value. This state arises from the Second-Order Necessary Conditions (the gradient of the objective function at the point of the local minimum equals to zero and the Hessian is positive semidefinite) and from the Second-Order Sufficient Conditions (if the gradient of the objective function equals to zero at some point x k and the Hessian is positive definite then the objective function has a local minimum in x k )[27]. If in some iteration a negative value for the acceleration parameter, γ k+1 < 0, is calculated, then we take γ k+1 = 1. This choice is valid because if G k is not a positive definite matrix, then using γ k+1 = 1 the next iterative point x k+2 is calculated using the classical gradient descent iteration: x k+2 = x k+1 t k+1 g k+1. Before we show the algorithm of defined HSM scheme, let us remark that for the correction parameter α k, as proposed in [10], we take a constant value from (0, 1) for each iteration. This implies that the value (α k + 1) α (1, 2). For the end of this section, we display the algorithm of the HSM process: 3 Convergence of the HSM scheme for uniformly convex functions In this section, we prove that accelerated gradient descent HSM iteration is well defined, and for that purpose, under some posed conditions, we confirm its convergence property for uniformly convex functions. We star this section with definition of the uniformly convex space originally given by Clarkson. Therewith, we state the definition of uniformly convex function [4, 29]. Definition 3.1 (Clarkson, 1936., [5]) Banach space X is uniformly convex if ɛ, 0 < ɛ 2existsδ(ɛ) such that from the next two relations x = y =1 x y ɛ

6 Algorithm 2 The HSM algorithm defined by (2.2)and(2.6) Require: Function f(x), α (1, 2), initial point x 0 dom(f). 1: Set k = 0 and calculate f(x 0 ), g 0 = f(x 0 ),setγ 0 = 1. 2: Check the test criteria; if stopping criteria are fulfilled then stop the algorithm; otherwise, go to the next step. 3: Applying Algorithm 1: compute the value of step size t k (0, 1] taking d k = γk 1 g k. 4: Determine x k+1 = x k αt k γk 1 g k,f(x k+1 ) and g k+1 = f(x k+1 ). 5: Compute γ k+1, approximation of the Hessian of function f at the point x k+1 using (2.6). 6: If γ k+1 < 0takeγ k+1 = 1. 7: k := k + 1, go to the step 2. 8: Return x k+1 and f(x k+1 ). where x,y X, follows the next estimation x + y 1 δ(ɛ). 2 Definition 3.2 A uniformly convex function with modulus φ, is a function f that for all x,y in the domain and t [0, 1] satisfies f(tx+ (1 t)y) tf (x) + (1 t)f(y) t(1 t)φ( x y )) where φ is a non-negative function that vanishes only at zero. Now we cite some known statements. First, we mention the proposition proved in [17, 24] and right after the relevant lemma proved in [26]. The relations from these statements will be required in proving Lemma 3.2 and Theorem 3.1. Proposition 3.1 Let function f : R n R be twice continuously differentiable and uniformly convex on R n. Then the next two statements are true: 1) The function f has a lower bound on L 0 ={x R n f(x) f(x 0 )}, where x 0 R n is available; 2) The gradient g is Lipschitz continuous in an open convex set B which contains L 0, i.e. there exists L>0such that g(x) g(y) L x y, x,y B. Lemma 3.1 Under the assumptions of Proposition 3.1, there exist numbers m, M R such that 0 <m 1 M, (3.1)

7 and function f(x)has an unique minimizer x and fulfills next inequalities: m y 2 y T 2 f(x)y M y 2, x,y R n ; (3.2) 1 2 m x x 2 f(x) f(x ) 1 2 M x x 2, x R n ; (3.3) m x y 2 (g(x) g(y)) T (x y) M x y 2, x,y R n. (3.4) The following lemma refers to defined iterative scheme (2.2). It gives the value of iterative decreasing of analyzed function when the HSM method is applied. Lemma 3.2 Suppose f is a twice continuously differentiable and uniformly convex function defined on R n and the sequence {x k } is defined by the Algorithm 2. Then, the next holds f(x k ) f(x k+1 ) μ g k 2, (3.5) where { } σ(αk + 1) σ(1 σ) μ = min, β. (3.6) M L Proof Let L>0 be the Lipschitz constant.we analyze the next two cases which refer to the value of the iterative step size: t k < 1andt k = 1. Consider the exit condition of Backtracking Algorithm 1 the next inequality is valid f(x k ) f(x k+1 ) σt k g T k d k, k N. (3.7) For t k < 1, the same way as described in [25], we get β(1 σ) gk T t k > d k L d k 2. Replacing d k = (α k + 1)γk 1 g k we get t k > β(1 σ)γ k L(α k + 1). (3.8) Substituting the previous evaluation into the (3.7) lead us to the next conclusion f(x k ) f(x k+1 ) σt k g T k d k > σ(1 σ)βγ k gt k (α k + 1)g k L(α k + 1) γ k σ(1 σ)β = g k 2. L To analyze the case t k = 1, we first consider the number M defined in Lemma 3.1 and the estimation γ k <Mthat arises from the inequalities (3.2), (3.3) and the fact

8 that γ k is k-th approximation of a Hessian of the objective function f. Knowing this, it is easy to derive the following f(x k ) f(x k+1 ) σt k gk T d k = σgk T d k = σgk T (( α k + 1)γk 1 )g k = σ(α k + 1) g k 2 γ k > σ(α k + 1) g k 2. M From the last two inequalities we derive the final conclusion { } σ(αk + 1) σ(1 σ) f(x k ) f(x k+1 ) min, β g k 2 (3.9) M L which completes the proof. The next theorem confirms that the HSM method is linearly convergent process. Theorem 3.1 For the twice continuously differentiable and uniformly convex function f defined on R n, and the sequence {x k } generated by Algorithm 2 the following is valid lim g k =0, (3.10) k and the sequence {x k } converges to x at least linearly. Proof Function f is bounded below and it decreases. Because of these facts the next expression is valid lim (f(x k) f(x k+1 )) = 0, k Applying estimation (3.5) from Lemma 3.2 we obtain lim g k =0. k Now we can put x = y in the inequality (3.4). Afterwards, we apply the mean value Theorem and the Cauchy-Schwartz inequality [17, 24]. The resulting inequality is then m x x g(x) M x x, x R n. (3.11) Using the estimations (3.3), (3.5), (3.6)and(3.11)weget μ g k 2 μm 2 x k x 2 2μ m2 M (f (x k) f(x )). (3.12) Previous inequations lead us to conclusion that lim x k x =0, i.e. the sequence k {x k } converges to minimizer x. Now we prove that the convergence of the HSM method is linear under the assumption M 2, where M is a number defined by (3.1) in Lemma 3.1. Accordingto(3.12) we need to confirm that ρ = 2μ m2 M < 1. (3.13)

9 In the case μ = σ(α k + 1)/M we have ρ 2 = 2μ m2 M = 2σ(α k + 1)m 2 M 2 2σ 2 m2 M < 1. 2 In the other case, μ = σ(1 σ)β/l, we also obtain the same estimation ρ 2 = 2μ m2 σ(1 σ) m 2 = 2β M L M < m2 ML < 1, and since m L previous inequality is valid. Finally,takingthe result oftheorem4.1 from [25] we achieve the final estimation 2(f x k x (x0 ) f(x )) m 1 ρ 2 k. 4 Convergence of the HSM method for strictly convex quadratic functions Further in this section we discover under which condition, considering the smallest and the largest eigenvalues of the matrix A,theHSM Algorithm 2 can be applied on the strictly convex quadratic functions given by f(x) = 1 2 xt Ax b T x, (4.1) where A is a real n n symmetric positive definite matrix and b R n. Motivation to examine a class of strictly convex quadratic functions presented by (4.1) arises from the fact that convergence analysis of many gradient methods is untypical and generally presents a difficult process. From the same reason the authors in [6, 14]also examined the convergence of the relevant gradient methods for the case of strictly convex quadratics. By applying the HSM method on this subset of functions we indicate additional condition which connects the smallest and the largest eigenvalues of the matrix A from (4.1). This condition is described by the next lemma. Lemma 4.1 When the Algorithm 2 of gradient descent HSM method is applied on the strictly convex quadratic function f given by (4.1), wherea R n n presents a symmetric positive definite matrix, then the next estimation holds λ 1 γ k+1 t k+1 4 λ n σ, (4.2) under the assumption that λ 1 and λ n are, respectively, the smallest and the largest eigenvalues of A.

10 Proof Using the expression (4.1) we can evaluate the difference between function values at the current and the previous point: f(x k+1 ) f(x k ) = 1 2 xt k+1 Ax k+1 b T x k xt k Ax k + b T x k. (4.3) After applying the expression (2.2)weget: f(x k+1 ) f(x k ) = 1 ( ) T ( ) x k (α k + 1)t k γk 1 g k A x k (α k + 1)t k γk 1 g k 2 b T ( x k (α k + 1)t k γ 1 k g k ) 1 2 xt k Ax k + b T x k = 1 2 (α k + 1)t k γ 1 k g T k Ax k 1 2 (α k + 1)t k γ 1 x T k Ag k (α k + 1) 2 tk 2 γ k 2 g T k Ag k + (α k + 1)t k γk 1 b T g k. Since the gradient of the function (4.1)isg k = Ax k b and b T g k = g T k b, we can conclude that: f(x k+1 ) f(x k ) = 1 ( ) 2 (α k + 1)t k γk 1 b T g k x T k Ag k (α k + 1) 2 tk 2 γ k 2 g T k Ag k ) = (α k + 1)t k γk (b 1 T x T k A g k (α k + 1) 2 tk 2 γ k 2 g T k Ag k = (α k + 1)t k γk 1 g T k g k (α k + 1) 2 tk 2 γ k 2 g T k Ag k. (4.4) After replacing (4.4) into(2.6), the scalar value of approximation of the Hessian in (k + 1) th iteration, parameter γ k+1, becomes (α k + 1)t k g T k γ k+1 = 2γ g k (α k + 1) 2 tk 2γ k 1 g T k Ag k + (α k + 1)t k g T k g k k (α k + 1) 2 tk 2gT k g k = gt k Ag k g T k g, k which confirms that γ k+1 is the Rayleigh quotient of the real symmetric matrix A at the vector g k, and that is why the next holds: λ 1 γ k+1 λ n, k N. (4.5) On the other hand, we have that 0 t k+1 1 which according to the previous inequalities prove the left-hand side in (4.2). To prove the right-hand side in (4.2), we use the estimation (3.8) which lead us to the next inequality: γ k+1 < L(α k+1 + 1) t k+1 β (1 σ ). (4.6) Since g(x) = Ax b and the matrix A is symmetric, the next holds: g(x) g(y) = Ax Ay = A(x y) A x y =λ n x y. (4.7) k

11 Now, for the Lipschitz constant L used in (4.6) we can take the largest eigenvalue λ n and considering the estimations for the Backtracking parameters σ (0, 0.5) and β (σ, 1) we finally get γ k+1 < L(α k+1 + 1) < λ n 2 t k+1 β (1 σ ) β (1 σ ) < 2λ n σ 0.5 = 4λ n σ, and with this the right hand side of inequalities (4.2) is proved. Theorem 4.1 Under the next assumptions: f is the strictly convex quadratic function given by (4.1); λ n < 2λ 1 α k +1,whereλ 1 and λ n are respectably the smallest and the largest eigenvalue of symmetric positive definite matrix A; {v 1, v 2,...,v n } is the orthonormal set of eigenvectors of matrix A; {x k } is the sequence of values constructed by Algorithm 2; the gradients of convex quadratics defined by (4.1), g k = Ax k b, are expressed as: g k = n di k v i, (4.8) for some real constants d1 k,dk 2,...,dk n and for some integer k; the application of the gradient descent method (2.2) on the function (4.1) satisfies the following two statements: { (di k+1 ) 2 δ 2 (di k )2,δ= max 1 σλ 1(α k +1), λ } n(α k +1) 1, i = 1, 2,..., 4λ n λ 1 (4.9) i=1 lim k g k =0. (4.10) Proof By applying the HSM iteration (2.2) on function(4.1) and considering that the gradient of the function (4.1)isg k = Ax k b as well as (4.8), the next holds ( ) ( g k+1 = A x k (α k +1)t k γk 1 g k b = g k (α k + 1)t k γk 1 Ag k = If we take (4.8)in(k + 1)-th iteration, we get g k+1 = n i=1 d k+1 i v i = n i=1 To prove (4.9), it is enough to show that 1 I (α k +1)t k γ 1 k ) A g k. (4.11) ( 1 (α k + 1)t k γ 1 k λ i ) d k i v i. (4.12) λ i (α k +1) 1 tk 1 γ k δ.

12 First, we suppose that λ i (α k + 1) 1 tk 1 γ k,i.e.λ i γ k (α k +1)t k. Considering (4.2) we have: 1 > λ i(α k +1)t k λ 1 γ k (α k +1) 1 tk 1 σλ 1(α k +1) λ i = 1 γ k 4λ n (α k +1) 1 tk 1 γ k σλ 1 1 δ. (4.13) 4λ n (α k + 1) 1 Suppose now that (α k + 1) 1 tk 1 γ k λ i. This leads to the next estimation: λ i λ n 1 < (α k + 1) 1 tk 1 γ k λ 1 (α k + 1) 1 = 1 λ i (α k + 1) 1 tk 1 γ k λ n 1 δ. (4.14) λ 1 (α k + 1) 1 From the expression (4.8) we can conclude that: n ( ) 2 g k 2 = di k, (4.15) i=1 and since the parameter δ, under condition λ n < 2λ 1 α k +1, satisfies 0 < δ < 1the statement (4.10) is valid. 5 Numerical experiments and achieved results Furthermore we support our analytical proofs from the previous two sections by numerical examples. For that purpose we primarily tested the starting SM method as well as presented HSM iteration and did the comparison of these two models. Numerical experiments are based on 25 test functions defined for a large scale unconstrained test problems given in [2]. On each test function 11 experiments are taken with the next number of variables: 1000, 2000, 3000, 5000, 7000, 8000, 10000, 15000, 20000, and We were monitoring and recording three features: the number of iterations, CPU time, and the number of function evaluations. For each algorithm (the SM and the HSM), we took the same stopping criteria used in [26]: g k 10 6 and f(x k+1) f(x k ) f(x k ) Parameters used in Backtracking procedures for both methods are σ = and β = 0.8. The codes used for numerical testing are written in the visual C++ programming language and the testings are conducted on a Workstation Intel Celeron 1.6 GHz. The results displayed in the Table 1 present the total number of iterative steps, CPU time (given in seconds), and the number of function evaluations for total 550 tests (25 test functions, each tested for 11 listed number of variables, after applying both of analyzed schemes). From displayed results in Table 1, we can conclude that concerning the number of iterations the HSM method gives better results then the SM method for 13 test

13 Table 1 Results of numerical experiments for SM and HSM iterations applied on 25 large-scale test functions Test function No. of iterations CPU time No. of funct. evaluation SM HSM SM HSM SM HSM Extended penalty Perturbed quadratic Raydan Diagonal Diagonal Generalized Tridiagonal Extended Himmelblau Quadr. Diag. Perturbed Quadratic QF Extended Quad. Penalty QP Extended Quad. Penalty QP Quadratic QF Extended EP Almost Perturbed Quadratic Engval Quartc Diagonal Tridia t>t e t>t e 7567 t>t e Indef Nonscomp Dixon 3dq Biggsb Hager Raydan Arwhead functions and the SM outperforms the HSM method in testing of 8 test functions. For 4 test functions, both of the methods show the same number of iterations. Considering the total CPU time needed when each of the methods is applied, the HSM is faster for 13 test functions while the SM is faster for 4 test functions. For 8 functions, both models give the same CPU time value. Regarding the total number of function evaluations obtained values are the same as for the total number of iterations: the HSM gives better performances for 13 test functions, the SM for 8 and the both methods perform equally regarding the number of function evaluations for 4 test functions. Specially, for the Tridia test function, time limiter constant t e introduced in [19] exceeds when the SM model is applied, and for that reason in Table 1 we put t>t e instead of numerical values in the relevant fields.

14 Table 2 Average results of the HSM and the SM methods for 24 test functions tested on 11 numerical experiments Average performances HSM SM Number of iterations CPU time (sec) Number of function evaluations To get a general impression about performance evaluation of the HSM and the SM methods, we display the next Table 2. This table shows achieved average results of number of iterations, total CPU time (in seconds) and the number of function evaluations for the both comparative models. From the previous Table 2,we can generally conclude that the HSM for the chosen set of test functions shows better performance regarding to all three analyzed characteristics compared to the SM iteration. More precise, when the number of iterations is considered, the HSM produces approximately 2,73 times smaller number of iterations than the SM. Referring the CPU time, the HSM is about 2.12 times faster than the SM. Finally, considering the number of function evaluations with respect to the HSM and the SM method, the HSM gives nearly 1.8 smaller number. We can generally conclude that according to the implemented numerical experiment the HSM shows averagely twice better efficiency than the SM. Next, we analyze how the value of correction parameter α from Algorithm 2, α (1, 2), affects the efficiency of the HSM model. For that purpose we chose the following values for parameter α: 1.1, 1.5 and 1.8. The aim is to check which of the purposed values induct the best performance of the HSM method. The first value α = 1.1, close to the left limit of this parameter, is used in numerical testing presented in the Table 1. Second choice, α = 1.5, is the middle value, and the third α = 1.8 is close to the right limit of this parameter. The testings are carried trough for the next number of variables: 100, 500, 1000, 1500, 2000, 3000 and For this research we took in consideration first five test functions from the Table 1. Results displayed in the Table 3 indicate that the best choice for parameter α is the closest value to the left limit, α = 1.1. For another two choices the HSM method produces less well results. Nevertheless, for the value α = 1.5, two of five test functions have time limiter parameter greater then t e, while for the value α = 1.8 even three of five test functions give too long execution time in these numerical experiments. Next, we display a Dolan-Moré performance profile for two comparative methods. Herein total 528 test values are included for each of two analyzed characteristics (testings related to the function Tridia are not considered because limiter parameter t e exceeds when the SM method is applied). Left in Fig. 1 we compare the HSM versus the SM subject to the CPU time, while on the right hand side we display a comparison of these two methods subject to the number of iterations. From Fig. 1 it is clear that considering efficiency and the robustness of analyzed models the HSM outperforms the SM iteration.

15 Table 3 Numerical results generated on first 5 test functions for three different values of parameter α Test function No. of iterations CPU time No. of funct. evaluation α = 1.1 α = 1.5 α = 1.8 α = 1.1 α = 1.5 α = 1.8 α = 1.1 α = 1.5 α = 1.8 Extended penalty t>t e 0 3 t>t e t>t e Perturbed quadratic Raydan t>t e t>t e 10 t>t e t>t e t>t e t>t e Diagonal t>t e t>t e 16 t>t e t>t e t>t e t>t e Diagonal Remark 5.1 In order to test the function f(x)= x 2 8x + 40, given as the example function in [10], we modified the HSM method code and made it applicable for the functions of one variable. After applying the HSM model, the solution value is obtained in the fifth iteration, the same as when the (1.5) is applied. Finally, we finish the analysis of numerical performance of the HSM model by comparing it with the Nesterov s accelerated gradient method from [16] equipped with line search procedure. For these numerical tests we had to reduce the number of variables included, since it was too slow to apply Nestorov s line search method, in further noted as NLS method, on a large number of variables. More precisely, we tested all 25 test functions from the Table 1, by applying the NLS method and the HSM method, and on each test function we conducted the next 7 number of variables: 1000, 2000, 3000, 5000, 7000, 8000, The results of these 370 tests are displayed in the following Table 4. Results presented in Table 4 show that considering the all three measured characteristics: number of iterations, CPU time and the number of function evaluations, the HSM method gives better, i.e. smaller, values then NLS method for 19 of total 25 Fig. 1 (Left) Performance profile for the HSM and the SM methods regarding the CPU time metric. (Right) Performance profile for the HSM and SM methods regarding the number of iterations metric

16 Table 4 Results of numerical experiments for NLS and HSM iterations applied on 25 large scale test functions Test function No. of iterations CPU time No. of funct. evaluation NLS HSM NLS HSM NLS HSM Extended penalty Perturbed quadratic Raydan Diagonal Diagonal Generalized Tridiagonal Extended Himmelblau Quadr. Diag. Perturbed t>t e t>t e 254 t>t e Quadratic QF1 t>t e t>t e 107 t>t e Extended Quad. Penalty QP Extended Quad. Penalty QP Quadratic QF Extended EP Almost Perturbed Quadratic t>t e t>t e 127 t>t e Engval Quartc Diagonal Tridia t>t e t>t e 718 t>t e Indef t>t e 7 t>t e 0 t>t e 21 Nonscomp Dixon 3dq Biggsb Hager Raydan Arwhead test functions. The opposite case is for only 2 test functions regarding the number of iterations and the number of function evaluations, and non regarding the CPU time. Both of the methods give the equal numbers of iterations and function evaluations for 4 test functions, while the equal CPU time is obtained for 6 test functions. Even tough we reduced the number of variables in order to provide the testings displayed at Table 4, it can be noticed that for 5 test functions we still could not end the testings when the NLS model was applied since in these cases the time limiter constant t e was grater then specified. The ratio of the efficiency measure of the HSM method and the NLS method can be clearly seen from the next Table 5. In this table an average results of the both models, considering all three analyzed characteristics, are displayed.

17 Table 5 Average results of the HSM and the Nesterov s line search method for 20 test functions tested on 7 numerical experiments conducted on each test function Average performances HSM NLS Number of iterations CPU time (sec) Number of function evaluations 22048, ,9 In Table 5 we can see that due to the testings conducted on total 20 test functions that could be tested by the Nestorov s algorithm, displayed results confirm that the HSM method shows generally multiply performance improvement regarding the tested parameters. Considering the number of iterations, the HSM gives around 14 times lower number then the NLS method, while regarding the number of function evaluations the HSM obtains almost 67 times better results. The greatest difference is spotted regarding the ratio of the CPU time average values of the both methods, which shows about 75 times betterment in favor to the HSM scheme. 6 Conclusion We introduce a gradient descent iterative process with accelerated features, which arise from the hybridization concept introduced in [10]. Defined HSM algorithm modifies accelerated gradient descent SM iteration presented in [26]. With properly defined acceleration parameter we prove linear convergence rate of the HSM for uniformly convex functions and for strictly convex quadratic functions. Derived model is numerically tested and compared with its forerunner, the SM method. Noticeable improvement in favor to the HSM process regarding the number of iterations, CPU time and number of function evaluations is observed. Applied Dolan-Moré performance profiles of both methods subject to the CPU time and the number of iterations also confirm the dominance of defined hybrid scheme. The HSM process itself is numerically examined for three different values of the correction parameter α. As a verification of the significant performance improvement of this hybrid accelerated scheme, the HSM is also compared with one of the first forerunners of the accelerated gradient methods, Nesterov s method presented in [16], which is previously equipped with the line search procedure. The results of numerical experiments show persuasive progress regarding all three tested characteristics. Presented hybridization concept can be applied on different gradient descent schemes in order to upgrade existing performance properties. References 1. Andrei, N.: An acceleration of gradient descent algorithm with backtracking for unconstrained optimization. Numer. Algor. 42, (2006)

18 2. Andrei, N.: An unconstrained optimization test functions collection. vol10/v10a10.pdf 3. Armijo, L.: Minimization of functions having Lipschitz first partial derivatives. Pac. J. Math. 6, 1 3 (1966) 4. Bauschke, H., Combettes, P.L.: Convex analysis and monotone operator theory in Hilbert spaces, p Springer, Berlin (2011) 5. Clarkson, J.A.: Uniformly convex spaces. Trans. Am. Math. Soc. 40, (1936) 6. Dai, Y.H., Liao, L.Z.: R-linear convergence of the Barzilai and Borwein gradient method. IMA J. Numer. Anal. 22, 1 10 (2002) 7. Fletcher, R., Reeves, C.M.: Function minimization by conjugate gradients. Comput. J. 7, (1964) 8. Goldstein, A.A.: On steepest descent. SIAM J. Control 3, (1965) 9. Ishikawa, S.: Fixed points by a new iteration method. Proc. Am. Math. Soc. 44, (1974) 10. Khan, S.H.: A Picard-Mann hybrid iterative process. Fixed Point Theory Appl. 2013, 69. Springer Open Journal (2013) 11. Lemaréchal, C.: A view of line search. In: Auslander, A., Oetti, W., Stoer, J. (eds.), Optimization and Optimal Control, pp Springer, Berlin (1981) 12. Luenberg, D.G., Ye, Y.: Linear and nonlinear programming. Springer Science+Business Media LLC, New York (2008) 13. Mann, W.R.: Mean value methods in iterations. Proc. Am. Math. Soc. 4, (1953) 14. Molina, B., Raydan, M.: Preconditioned Barzilai Borwein method for the numerical solution of partial differential equations. Numer. Algor. 13, (1996) 15. Moré, J.J., Thuente, D.J.: On line search algorithm with guaranteed sufficient decrease, Mathematics and Computer Science Division Preprint MCS-P Argone National Laboratory, Argone (1990) 16. Nesterov, Ye.E.: A method for solving the convex programming problem with convergence rate O( 1 ). Dokl. Acad. Nauk SSSR 269(3), (1983) k 17. Ortega, 2 J.M., Rheinboldt, W.C.: Iterative solution of nonlinear equation in several variables. Academic Press, New York (1970) 18. Petrović M.J.: An accelerated double step size method in unconstrained optimization. Appl. Math. Comput. 250, (2015) 19. Petrović, M.J., Stanimirović, P.S.: Accelerated double direction method for solving unconstrained optimization problems. Math. Probl. Eng. 2014, Article ID , 8 pp (2014) 20. Stanimirović, P.S., Milovanović, G.V., Petrović, M.J.: A transformation of accelerated double step size method for unconstrained optimization. Math. Probl. Eng. 2015, Article ID , 8 pp (2015) 21. Potra, F.A., Shi, Y.: Efficient line search algorithm for unconstrained optimization. J. Optim. Theory Appl. 85, (1995) 22. Powell, M.J.D.: Some global convergence properties of a variable-metric algorithm for minimization without exact line search. AIAM-AMS Proc. Phila. 9, (1976) 23. Picard, E.: Memoire sur la theorie des equations aux derivees partielles et la methode des approximations successives. J. Math. Pures Appl. 6, (1890) 24. Rockafellar, R.T.: Convex analysis. Princeton University Press, Princeton (1970) 25. Shi, Z.-J.: Convergence of line search methods for unconstrained optimization. Appl. Math. Comput. 157, (2004) 26. Stanimirović, P.S., Miladinović, M.B.: Accelerated gradient descent methods with line search. Numer. Algor. 54, (2010) 27. Sun, W., Yuan, Y.-X.: Optimization theory and methods: nonlinear programming. Springer, Berlin (2006) 28. Wolfe, P.: Convergence conditions for ascent methods. SIAM Rev. 11, (1968) 29. Zalinescu, C.: Convex analysis in general vector spaces. World Scientific, Singapore (2002)

19 本文献由 学霸图书馆 - 文献云下载 收集自网络, 仅供学习交流使用 学霸图书馆 ( 是一个 整合众多图书馆数据库资源, 提供一站式文献检索和下载服务 的 24 小时在线不限 IP 图书馆 图书馆致力于便利 促进学习与科研, 提供最强文献下载服务 图书馆导航 : 图书馆首页文献云下载图书馆入口外文数据库大全疑难文献辅助工具