Global Convergence of Perry-Shanno Memoryless Quasi-Newton-type Method. 1 Introduction

Transcription

1 ISSN (print), (online) International Journal of Nonlinear Science Vol.11(2011) No.2,pp Global Convergence of Perry-Shanno Memoryless Quasi-Newton-type Method Yigui Ou, Jun Zhang and Qian Zhou Department of Mathematics, Hainan University, Haikou , P.R.China (Received 11 August 2008, accepted 10 September 2010) Abstract: In this paper, we analyze the global convergence properties of Perry-Shanno memoryless quasi- Newton type method associating with a new line search model. preliminary numerical results are also reported. Keywords: unconstrained optimization; memoryless quasi-newton method; line search; global convergence 1 Introduction In this paper, we consider the following unconstrained optimization problem min f(x) (1) x Rn where f : R n R is continuously differentiable function. Quasi-Newton method is a well-known and useful method for solving unconstrained convex programming and the BFGS method is the most effective quasi-newton type methods for solving unconstrained optimization problems from the computation point of view. For the current iterate x k R n and symmetric positive definite matrix B k R n n, the next iterate is obtained by x k+1 = x k + a k d k (2) where a k > 0 is a step-size obtained by a one-dimensional line search, and d k = B 1 k f(x k) (3) is a descent direction with B 1 k being available and approximating the inverse of the Hessian matrix of f at x k. Throughout this paper, we use to denote the Euclidean vector or matrix norm and denote f(x k ) by g k. Although the quasi-newton type method is known to be remarkably robust in practice, one will not be able to establish truly convergence results for general nonlinear objective functions, that is, one cannot prove that the iterates generated by this method approach a stationary point of the problem from any starting point and any (suitable) initial Hessian approximation. Therefore, there has been an ever-expanding interest in quasi-newton type methods since the first quasi- Newton method was suggested by Davidon (1959) and improved by Fletcher and Powell (1963) (hence the name DFP formula), and there is a vast literature on the problem of convergence properties of the quasi-newton type method for solving problem (1). For details, see Refs[1], [2], [3] and references therein. In 1978, Shanno [4] presented a memoryless quasi-newton-type method, in which the search direction d k is defined by { dk = B 1 k g k (k 1), d 0 = g 0 and B k is updated by the following Perry and Shanno formula B k+1 = y k 2 yk T s I + y kyk T k yk T s y k 2 k s T k y k s k s ks T 2 k (4) Corresponding author. address: ouyigui@tom.com Copyright c World Academic Press, World Academic Union IJNS /455

2 154 International Journal of Nonlinear Science, Vol.11(2011), No.2, pp with s k = x k+1 x k, y k = g k+1 g k. If we use reverse form H k of B k, then we can obtain the formula of H k and the next search direction can be written as H k+1 = yt k s k y k 2 I + 2 s ks T k yk T s 1 k y k 2 (y ks T k + s k yk T ) (5) d k+1 = B 1 k+1 g k+1 = H k+1 g k+1 (6) Moreover, Shanno proved the global convergence properties of this method in association with Wolf step-size selection rules. Subsequently, Werner [5] generalized Shanno s conclusion with a step-size a k satisfying f(x k + a k d k ) f(x k ) min{μ 1 ( g T k d k ), μ 2 ( gt k d k d k )2 } (7) where μ 1 and μ 2 are two positive constants. Recently, Liu and Jing [6] analyzed the convergence of the method with nonmonotone Wolfe line search. As mentioned above, the problem of convergence properties of the quasi-newton type method has been studied extensively and a great deal of achievements have been made. However, the proofs of the convergence properties only have relation to concrete step-size selection rules. Recently, Han, Liu and Guo[7, 8, 9] have proven the global convergence properties of several BFGS-type methods for solving problem (1) associated with general line search model. Motivated by the general line search model and (7), we propose a new line search technique, and consider the global convergence properties of Perry-Shanno memoryless quasi-newton type method associating with this new technique. Compared with [6]-[9], we obtain the strong convergence property: lim k + g k = 0 instead of the weak property: lim k + inf g k = 0 This paper is organized as follows: In section 2, we describe the new line search model and related quasi-newton type method. In section 3, we analyze the convergence property of the method with this new model. In section 4, we present numerical experiments. Some conclusions are summarized in section 5. 2 New line search model In this section, a new method with a new line search model is given. To this end, we first give some definitions as in Ref[7], which is useful subsequently. Definition 1 The function σ : [0, + ) [0, ) is called forcing function if lim i σ(t i ) = 0 for any t i 0, i = 1, 2,, then one has lim i t i = 0. Using the definition of forcing function, the following definition can be given. Definition 2 The general form of step-size selection rules: Let a > 0, β [0, 1), σ 1, σ 2, σ 3 and ψ are forcing functions, choose the step length a k to satisfy one of the following three situations (S1) f(x k + a k d k ) f(x k ) σ 1 (r k ), where r k = gt k d k d k. (S2) f(x k + a k d k ) f(x k ) σ 2 ( a k g T k d k), and if a k d k min{a, ψ(r k )}, then there exists b k satisfying and {b k }, {x k + b k a k d k } are bounded. (S3) f(x k + a k d k ) f(x k ) σ 3 (r k ) d k. f(x k + b k a k d k ) T d k βg T k d k (8) Inspired by the above definition 2 and (7), we introduce a new line search model, in which the step-size a k is defined as f(x k + a k d k ) f(x k ) min{σ 1 (r k d k ), σ 2 (r k ), σ 3 (r k ) d k } (9) where σ 1, σ 2 and σ 3 are forcing functions, and r k = gt k d k d k. Remark 1 Similarly to the proof of Theorem 2 in Ref [6], we can immediately deduce that the most popular step-size rules are special cases of the line search model (9), such as the Goldstein step-size rule, the Wolfe step-size rule and the Arimijo step-size rule. Moreover, if we choose σ 1 (t) = μ 1 t, σ 2 (t) = μ 2 t 2 and σ 3 (t) = σ 1 (t), then (9) is transformed into (7). IJNS for contribution: editor@nonlinearscience.org.uk

3 Y. Ou, J. Zhang, Q. Zhou: Global Convergence of Perry-Shanno Memoryless Quasi-Newton-type Method 155 Remark 2 Obviously, if a k denotes the step-size computed by the general line search model (S1) or (S3), then a k must satisfy the new line search rule (9). That is to say, the step-size a k generated by (9) is easier to find from the theorem point of view. Associated with the new line search model (9), we give a modified Perry-Shanno memoryless quasi-newton type method (MPSMQN) as follows. The MPSMQN mehod Given an initial point x 0 R n and a symmetric positive definite matrix B 0 R n n. k := 0. while g k = 0 { dk = B 1 k g k = H k g k (10) x k+1 = x k + a k d k Remark 3 In the above method, a k is the step-size defined by (9), and B k or H k is updated by the formula (4) and (5), respectively. 3 Convergence analysis In order to analyze the global convergence of our proposed method. We need the following assumptions. Assumptions H1. The level set L 0 = {x R n f(x) f(x 0 )} is contained in a bounded convex set D. H2. The function f(x) is twice continuously differentiable. H3. There exist positive constants c 1 and c 2 such that c 1 z 2 z T 2 f(x)z c 2 z 2, x, z R n (11) Lemma 4 Under Assumption H2 and H3, there exists a constant c 3 > 0 such that y k 2 c 3 (12) y k 2 1 c 1 (13) Proof. For the proof of inequality (12), see Ref[7]. Here we only prove the inequality (13). By using Assumption H2 and H3, we have = 1 From (12) and the Cauchy-Schwartz inequality, it follows that 0 s T k 2 f(x k + ts k )s k dt c 1 s k 2 (14) yk T s k y k 2 s k 2 yk T s 1 (15) k c 1 This proof is completed. Remark 5 From Lemma 1, it follows c 1 y k 2 c 3 (16) Lemma 6 Let the matrix B k be defined by the formula (4). Then B k is symmetric and positive definite, so is H k defined by the formula (5). Proof. We will prove by induction. x T B k x > 0, x = 0 (17) IJNS homepage:

4 156 International Journal of Nonlinear Science, Vol.11(2011), No.2, pp Obviously, B 0 is symmetric and positive definite by the proposed method. We now suppose that (17) holds for some k 1. Then by the update formula (4) we have x T B k+1 x = y k 2 (x T x (st k x)2 s T k s ) + (yt k x)2 k yk T s k (18) From the Cauchy-Schwartz inequality, it immediately follows that x T x (st k x)2 s T k s k 0 (19) In addition, the second term in (18) is also nonnegative because of s T k y k > 0 (see (14)). Therefore we obtain that x T B k+1 x 0 (20) In what follows, we must prove that at least one term in (18) is strictly larger than zero. Since x = 0, the inequality (19) becomes an equality if and only if s k is parallel to x, i.e., there exists a constant γ = 0 such that x = γs k. Thus (y T k x)2 = γ 2 > 0 (21) which indicates that if the first term in (18) equals zero, the second term must be strictly larger than zero. Thus, for any x = 0, we always have x T B k+1 z > 0. In addition, the positive definiteness of H k is obvious, since it is the reverse form of B k. This proof is completed. Theorem 3 Suppose that Assumptions H1, H2 and H3 hold. Let {x k } be an infinitely sequence generated by the memoryless Perry-Shanno quasi-newton type method with the rule (9). Then lim g k = 0 (22) k + Proof. Suppose on the contrary that there exist ε > 0 and an infinite subset of K of {0, 1, 2, } such that By using the update formula (4) and (5), we have g k ε, k K (23) trace(b k ) = n y k 1 2 y T k 1 s k 1 (24) Combining (13) with Lemma 1 yields trace(h k ) = (n 2) yt k 1 s k 1 y k s k 1 2 y T k 1 s k 1 (25) trace(b k ) nc 3 (26) trace(h k ) n c 1 (27) Since which implies where d k 2 gk T H = H kg k 2 kg k gk T H trace(h k ) n (28) kg k c 1 d k 2 gk T d = k d k 2 g k d k cos θ k = cos θ k = gt k d k g k d k d k g k cos θ k n c 1 (29) IJNS for contribution: editor@nonlinearscience.org.uk

5 Y. Ou, J. Zhang, Q. Zhou: Global Convergence of Perry-Shanno Memoryless Quasi-Newton-type Method 157 Note that From (29) and (30), we have d k = H k g k g k B k g k trac(b k ) ε nc 3, k K (30) and r k = gt k d k d k = g k cos θ k c 1 d k n gk T d k = r k d k ε nc 3 c 1ε n 2 c 3, k K (31) c 1 ε n 2 = c 1ε 2 c 3 n 3 c 2, k K (32) 3 Since σ 1, σ 2 and σ 3 are forcing functions, then there exist ε 1 > 0, ε 2 > 0 and ε 3 > 0 such that for any k K. Thus σ 1 ( gk T d k) ε 1 σ 2 (r k ) ε 2 σ 3 (r k ) d k ε 3 k K min{σ 1 ( g T k d k ), σ 2 (r k ), σ 3 (r k ) d k } k K min{ε 1, ε 2, ε 3 } = + (33) On the other hand, it follows from (9) that f(x k+1 f(x k ) for all k, which, together with Assumption H1, implies that lim k f(x k ) exists. Hence from (9) we can deduce that which implies + k=0 min{σ 1 ( g T k d k ), σ 2 (r k ), σ 3 (r k ) d k } f(x 0 ) lim k f(x k) < + min{σ 1 ( gk T d k ), σ 2 (r k ), σ 3 (r k ) d k } < k K This is in contradiction with (33), which implies that the conclusion of Theorem 3 is true. This completes the proof. 4 Numerical experiments To illustrate the feasibility of the proposed mehod, Algorithm MPSMQN was then implemented in Matlab 7.1 and run on a computer for four standard test functions as follows(see Ref [10]). Test 1. Rosenbrock function. f(x) = 100(x 2 x 2 1) 2 + (1 x 1 ) 2. Test 2. Wood function. f(x) = 100(x 2 1 x 2 ) 2 + (x 1 1) 2 + (x 3 1) (x 2 3 x 4 ) ((x 2 1) 2 + (x 4 1) 2 ) (x 2 1)(x 4 1) Test 3. Powell singular function. f(x) = (x x 2 ) 2 + 5(x 3 x 4 ) 2 + (x 2 2x 3 ) (x 1 x 4 ) 4. Test 4. Cube function. f(x) = 100(x 2 x 2 1) 2 + (1 x 1 ) 2. Throughout the computational experiments, we choose the following forcing functions: σ 1 (t) = μ 1 t, σ 2 (t) = μ 2 t 2 and σ 3 (t) = σ 1 (t) where the parameters used are μ1 = 10 4, μ2 = 0.1. The stopping condition is g k To validate the MPSMQN method from a computational point of view, we compare it with the traditional Perry- Shanno memoryless quasi-newton-type method(abbreviated as the PS method) with the following Wolf linesearch, which is one of the most popular line search rules. { f(xk + a k d k ) f(x k ) + ρa k g T k d k g(x k + a k d k ) σg T k d k IJNS homepage:

6 158 International Journal of Nonlinear Science, Vol.11(2011), No.2, pp Table 1: Numerical results for testing problems Test problem n MPSMQN PS Test /131/111 61/205/123 Test /361/ /480/405 Test /223/ /345/305 Test /106/79 45/122/91 where 0 < ρ < 0.5, σ (ρ, 1). The PS method is implemented in the same way. Table 1 lists the numerical results which are given in the form of k/kf/kg, where k, kf and kg denote the number of iterations, function evaluations and gradient evaluations, respectively. Note that we don t give the initial points, since they are the same as that in Ref [10]. Of course, we cannot draw some general conclusions from the rather limited numerical tests. Comparing the results given by our method with the PS method, however, our method is comparable to that in computational effort, which indicates that the proposed algorithm is effective in some sense. Further improvement is expected from more sophisticated implementation. 5 Conclusion In this paper, we propose a new line search model for unconstrained optimization problem. By using this new model, we establish the strong global convergence properties of Perry-Shanno memoryless quasi-newton type method. Numerical results show its feasibility. Acknowledgements This work is partially supported by the NSF of Hainan University(No. HD09XM87). References [1] Y.H.Dai. Convergence properties of the BFGS algorithm. SIAM Journal on Optimization., 13 (2002): [2] Z.J.Shi, Convergence of quasi-newton method with new inexact line search. Journal of Mathematical Analysis and Applications. 315(2006): [3] Z.X.Chen, G.Y.Li and L.Q.Qi. New quasi-newton methods for unconstrained optimization problems. Apllied Mathematics and Computation, 175 (2006): [4] D.F.Shanno. On the convergence of a new conjugate gradient algorithm. SIAM Journal on Numerical Analysis, 15 (1978): [5] J.Werner. Global convergence of quasi-newton methods with practical line searches. Technical Report, NAM-Bericht Nr., 67 (1989). [6] G.H.Liu, L.Jing. Convergence of nonmonotone Perry and Shanno methods for optimization. Computational Optimization and Applications, 16 (2000): [7] J.Y. Han, G.H.Liu. General form of step-size selection rules line search and relevent analysis of global convergence of BFGS algorithm (in chinese). Acta Mathematicae Applicatae Sinica, 1 (1995) [8] Q.Guo, J.G.Liu. Global convergence properties of two modified BFGS-type methods. Journal of Applied Mathematics & Computing, 23(1-2) (2007): [9] Q.Guo, J.G.Liu. Global convergence of a modified BFGS-type method for unconstrained non-convex minimization. Journal of Applied Mathematics & Computing, 24(1-2) (2007): [10] F.J.Zhou, Y.Xiao. A class of nonmonotone stabilization trust region methods. Computing, 53 (1994): IJNS for contribution: editor@nonlinearscience.org.uk