A NOTE ON PAN S SECOND-ORDER QUASI-NEWTON UPDATES

Transcription

1 A NOTE ON PAN S SECOND-ORDER QUASI-NEWTON UPDATES Lei-Hong Zhang, Ping-Qi Pan Department of Mathematics, Southeast University, Nanjing, , P.R.China. Abstract This note, attempts to further Pan s second-order quasi-newton methods([1]). To complement the numerical implementation, the linear convergence of a rank-one second-order update and the least change property are presented. Keywords: Second-order quasi-newton update, linear convergence, least change property. AMS classification: 65H10, 65K05, 90C26 1 Introduction In 1959, Davidon first introduced the quasi-newton method([2]) for solving the unconstrained minimization min f(x). (1.1) x Rn Since then, there has been an ever-expanding interest in quasi-newton methods both in literature(e.g., [3], [4], [5], [6], [7]) and in the implementation(e.g.,[8], [9], [10]). The most general form of this method is x k+1 = x k 1 k f(x k), k = 0, 1, (1.2) where k satisfies the well-known quasi-newton equation, k+1 s k = y k, (1.3) This work was supported by the National Science Foundation of China (No ). Department of Mathematics, Southeast University, longzlh@163.com Department of Mathematics, Southeast University, panpq@seu.edu.cn 1

2 with the notation s k = x k+1 x k, y k = f(x k+1 ) f(x k ). (1.4) Since the advent of (1.3), a vast number of quasi-newton updates which satisfy (1.3) have been proposed. The most efficient and practical one so far, is FGS (independently by royden(1969, 1970), Fletcher (1970), Goldfard (1970), Shanno (1970)). However, on the other hand, the effort to construct some updates not satisfying (1.3) never stops(e.g., [11], [12]). In 1984, Pan([1]) suggested a new quasi-newton equation, k+1 s k = 2y k k s k, (1.5) and he also showed that in an approximate sense, this new equation is of second order, in contrast to the first order of the classical one(1.3). This equation, hence, is called the second-order quasi-newton equation and its relevant updates the second-order quasi- Newton formulae. Although some computational tests implemented in [1] illustrate its attraction for further investigation, some theoretical analysis is still deficient even the convergent analysis. This note, then, is intended to be a step to develop the second-order quasi-newton equation/updates. The local linear convergence of a rank-one updating formula is described first, and then follows the least change property, ie., the property of the closest to the preceding update. 2 Linear Convergence For easy of expression, we shall use F (x) to denote f(x); also, we shall omit suffices and shall denote all quantities corresponding to k + 1 and k by using a tilde superfix, no superfix, so that F (x k+1 ) is written F, and F (x k ) is written F, and the same to others. As a special second-order quasi-newton update, Pan ([1]) gave a rank-one formula, = + 2(y s)st, (2.1) We will show the linear convergence of the iteration (1.2) with the update (2.1). So, first we assume F (x) has the following hypotheses. (A). The mapping F (x) is continuously differentiable in an open convex set D. (). There is an x in D such that F (x ) = 0 and F (x ) is nonsingular. 2

3 (C). F (x) satisfies a Lipschitz condition at x, that is, there is a constant λ such that F (x) F (x ) λ x x x D. (2.2) The proof of the linear convergence follows from Lemma 1 and Lemma 2 below. Lemma 1 (royden, Dennis and Moré, [13], 1973) Let F : R n R n satisfy the hypotheses (A), (), (C) of F (x), and let U be an update function for F (x) such that for all (x, ) domu and U(x, ) F (x ) [1 + α 1 σ(x, x)] F (x ) +α 2 σ(x, x) (2.3) for some constants α 1 and α 2 where x = x 1 F (x) and σ(x, x) = max{ x x, x x } Then there are positive constants ε and δ such that if x 0 D and 0 D M satisfy x 0 x ε and 0 F (x ) δ, then iteration (1.2 ) is well defined and converges linearly to x. This lemma still holds if we take the Frobenius norm. Lemma 2 Let F : R n R n satisfy the hypotheses (A), () of F (x). Then for any x D, F (v) F (u) F (x )(v u) γ max{ v x, u x } v u for all v and u in D. This conclusion follows from a standard result (Ortega & Rheinboldt, 1970:70). Theorem 1 Let F : R n R n satisfy the assumptions (A), (), (C) of F (x), consider the iteration scheme (2.4) x k+1 k+1 = x k 1 k F (x k) = k + 2 (y k k s k )s T k, k = 0, 1, (2.4) with y k = F (x k+1 ) F (x k ), and s k = x k+1 x k. Then this method is well defined and locally convergent at x. 3

4 Proof. According to (2.4), it follows k+1 F (x ) = k F (x ) + 2 (y k k s k )s T k = [ k F (x )][I 2s ks T k ] + 2(y k F (x )s k )s T k. Applying I 2s ks T k 2= 1 and lemma 2 gives k+1 F (x ) F [ k F (x )] F + 2(y k F (x )s k ) s k [ k F (x )] F +2γσ(x k, x k+1 ), which completes the proof with the aid of lemma 1. In fact, this result can be extended to a larger group of updates in which a parameter θ (0, 2] replaces 2 in (2.4), ie., = + θ(y s)st. (2.5) This formula originally suggested by Powell [14] as the modification of royden s rankone update([7]). The proof is the same when noting I θ sst Correspondingly, the second-order equation shall then become = 1 with θ (0, 2]. s = s + θ(y s). (2.6) 3 Least Change Property Now, we are in a position to describe another property related also to the update (2.1). Theorem 2 Given R n n, y R n and some nonzero s R n, define by (2.1), then is the unique solution to the problem, min{ F : s = 2y s}. (3.1) Proof. Let be an arbitrary such that s = 2y s, which implies y s = 1 ( )s. (3.2) 2 4

5 It follows from (3.2) that (y s)st F = 2 F = (s s)st F F. The conclusion that is the unique solution results from the fact that the mapping f : R n n R defined by f() = F is strictly convex in the convex set { : s = 2y s}. The counterpart of theorem 2 concerns a rank-two update, another second-order quasi- Newton formula, = + 2 c(y s)t + (y s)c T c T s 2 (y s)t scc T (c T s) 2. (0 c R n ) (3.3) Analogous to theorem 2, the least change property is also true to this update. Theorem 3 Let R n n be symmetric, and c, s, y R n satisfy c T s > 0; assume also that M R n n is some nonsingular symmetric matrix such that Mc = M 1 s, then the matrix defined by (3.3) is the unique solution to the problem, where M,F = MM F. min{ M,F : s = 2y s, T = }, (3.4) Proof. Assume be a symmetric matrix such that s = 2y s, also implying (3.2). Let Mc = M 1 s = z, E = M( )M, two sides of (3.3), and taking account of (3.2) leads to, Ẽ = M( )M = EzzT + zz T E z T z Ẽ = M( )M. Multiplying by M on the zzt E (z T z) 2 zzt. (3.5) Apparently, Ẽz 2= Ez 2 ; and moreover, if v z, from (3.5), it follows Ẽv 2 Ev 2, which support the conclusion that defined by (3.3) is the solution of (3.4). The uniqueness of this solution results from the fact that the mapping f : R n n R defined by f() = M,F is strictly convex in the convex set { : s = 2y s, T = }. Thus we complete the proof. In particular, setting c = s in (3.3) yields the second-order PS update, and taking c = y, then, gives the second-order DFP(see [1]). More generally, theorem 2 and theorem 3 still hold if 2 in (2.1) and (3.3) is replaced by a real parameter θ 0, and we just summarize them below. 5

6 Theorem 4 Given R n n, y R n and some nonzero s R n, define by (2.5), then is the unique solution to the problem, min{ F : s = s + θ(y s)}. Theorem 5 Let R n n be symmetric, and c, s, y R n satisfy c T s > 0; assume also that M R n n is some nonsingular symmetric matrix such that Mc = M 1 s, then the matrix = + θ c(y s)t + (y s)c T c T s θ (y s)t scc T (c T s) 2, is the unique solution to the problem, where M,F = MM F. min{ M,F : s = s + θ(y s), T = }, 4 Concluding Remarks For a credible algorithm, the convergence analysis is necessary. As the first step to develop the second-order quasi-newton methods, the proposed rank-one update (2.1) possesses the property of linear convergence. The proposition described in section 3 shows the evidence that the proposed updates (2.1) and (3.3) are the best candidates for as the rank-one and rank-two updates, respectively. In practice, however, some other rank-two updates are still attractive(refer to [1]), and the linear and further, the surperlinear convergence remain to be open. References [1] PAN PING-QI. On advanced of second order Quasi-Newton method, Journal of computational mathematics, Vol.2 No.1., Januray [2] W.C.DAVIDON. Variable metric method for minimization. Rep.ANL-5990 Rev,Argonne National Laboratories,Argonne,III, [3] AVRIEL,M. Nolinear programming: Analysis and methods, Prentice-Hall, Inc., Englewood Cliffs, New Jersey, [4] FLETCHER, R. Practical methods of optimization, Unconstrained optimization, John Wiley and Sons, New York and Toronto., Vol 1. 6

7 [5] C.G. ROYDEN. A class of methods for solving nonlinear simultaneous equations. Math,Comp., v.19, 1965,pp MR 33#6825. [6] M.J.D.POWELL. A new algorithm for unconstrained optimization. Nonlinear programming, Academic Press, New York,1970, pp mr42#7043. [7] J.E.DENNIS,JR. AND JORGE J.MORÉ. Quasi-Newton method, motivation and theory. SIAM REVIEW,Vol.19,1977, No.1, [8] FLETCHER, R. and POWELL,M.J.D. On the modification of LDL T factorization, Math,Comp., v.28, 1974,pp [9] RODLIE, K.W. An assessment of two approaches to variable metric methods,math,prog., v.12, 1977,pp [10] P.E. GILL AND W. MURRAY. Quasi-Newton methods for unconstrained optimization. J.Inst.Maths Applics(1972)9, [11] ZENGXIN WEI, GAOHANG YU, GONGLIN YUAN, ZHIGANG LIAN, The superlinear convergence of a modified GFS-type method for uncontrained optimization.computational Optimization and Application., 29, [12] JIANZHONG ZHANG, CHENGXIAN XU, Properties and numerical performance of quasi- Newton methods with modified quasi-newton equations.journal of Computational and Applied Mathematics., 137, [13] G.G. ROYDEN. J. E. DENNIS ADN J. J. MORÉ. On the local and superlinear convergence of quasi-newton methods. Ibid.,12,pp , [14] M.J.D.POWELL. A FORTRAN subroutine for solving systems of nonlinear algebraic equations. Numerical methods for Non-linear Algebraic Equations, Gordon and reach, London,1970, pp MR 49#8330b. [15] J.E.DENNIS, On the convergence of royden s method for nonlinear systems of equation. Math.Comp.,25,1971,pp [16] Y. YUAN and W. SUN, Theroy and mothods of Optimization. Scinece Press of China,