Introduction. A Modified Steepest Descent Method Based on BFGS Method for Locally Lipschitz Functions. R. Yousefpour 1

Transcription

1 A Modified Steepest Descent Method Based on BFGS Method for Locally Lipschitz Functions R. Yousefpour 1 1 Department Mathematical Sciences, University of Mazandaran, Babolsar, Iran; yousefpour@umz.ac.ir Abstract In this paper, the steepest descent method is modified based on the BFGS method. At first, the descent direction is approximated based on the Goldstein subgradient and a positive definite matrix. Then the Wolfe condition is generalized for the locally Lipschitz functions. Next, an algorithm is developed to compute a step length satisfying the generalized Wolfe condition. At each iteration, the positive definite matrix is updated by the BFGS method. By the presented line search algorithm, a pair of subgradients are computed for updating the positive definite matrix such that the secant equation is satisfied. Finally, an algorithm is devolved based on the BFGS method. This algorithm is implemented with MAT- LAB codes. eywords: Lipschitz functions, Armijo and Wolfe conditions, Line search, BFGS method. Introduction The steepest descent direction for the locally Lipschitz functions is computed based on an element of the Goldstein subgradient with minimal norm. By an approximation of this direction, several bundle methods were developed [1-6]. The efficiency of an algorithm, that developed based on an approximation of steepest descent direction, depends on the approximation accuracy. To improve the accuracy of an approximation, we need to compute a larger number of subgradient and, this is time consuming. For example, in [6], the steepest descent direction is approximated by sampling gradients. This approximation is appropriate, but computing this approximation for large scale problems is very expensive. In [4], the 1

2 steepest descent direction is iteratively approximated. This method computes a good approximation for the steepest descent direction by the less number of subgradients. The numerical results showed that this algorithm is more efficient than other methods. If the steepest descent method is suitably approximated, then the constructed algorithm based on this direction will be efficient. On the other hand, the BFGS method has good behavior for some nonsmooth functions. It can be expected that the combination of the steepest descent me and BFGS methods will be efficient. The first step of combining the steepest descent and BFGS methods is that compute an approximation of the steepest descent direction by the Goldstein subgradient and, a positive definite matrix. In each iteration of the main algorithm, the positive definite matrix must be updated by the BFGS method thus, the Wolfe step length condition is generalized based on the Goldstein subgradient. Based on this generalization, we prove that there exist step lengths satisfying this generalization for a descent direction. We generalized the line search algorithms [7] and, prove that these generalize algorithms compute a step length satisfying the generalized Wolfe condition. These algorithms also compute two subdifferentials for updating the approximation of Hessian matrix. By using the similar ideas to [4], the Goldstein subgradient is iteratively approximated until a descent direction is found based on a positive definite matrix. By these ideas, the Goldstein subgradient is efficiently approximated. We prove that this procedure finds a descent direction after finitely many iterations. This descent direction is an approximation of the steepest descent direction based on positive definite matrix. The function is reduced along this direction. In each iteration of a minimization algorithm, the positive matrix is updated by the BFGS method. The presented line search algorithm finds two subgradients such that the positive definite matrix can be updated and, the secant equation is satisfied. Finally, the algorithm is implemented with Matlab and, the results are compared with other methods. Generalized Armijo and Wolfe conditions In this paper, we suppose that H is a positive definite matrix. Suppose that f : R n R is a locally Lipschitz function, v R n and, g = Hv. If 2

3 there exists α > 0 such that f(x+tg) f(x) t v H, t (0,α), then g is a descent direction. Now, the Armijo condition is given as follows. D e f i n i t i o n 1. Let f : R n R be a locally Lipschitz function and, v R n. Suppose that g = Hv is a descent direction at x. A step length α > 0 satisfies in the Generalized Armijo Condition (GAC), if f(x+αg) f(x) c 1 α v H, where c 1 (0,1). Equivalently A(α) 0, where A(α) := f(x+αg) f(x)+c 1 α v H. D e f i n i t i o n 2. Let f : R n R be a locally Lipschitz function, v R n and, g = Hv beadescentdirectionatx. IfW( )isnotdecreasing on a neighborhood of α and, the GAC satisfies in α along direction g at x, then we say that the GWC satisfies at α along direction g at x. Computing descent direction Here, we discuss on finding a descent direction for a locally Lipschitz function f based on ǫ f(x) and, a given positive definite matrix H. Consider the following problem and, suppose that 0 ǫ f(x), ξ 0 = arg min ξ ǫf(x) ξ H. (1) Let g = Hξ 0. We have f (x,g) ξ 0 H < 0 and, by the Leburg Mean Value Theorem, f(x+tg) f(x) t ξ 0 H, t (0,ǫ). (2) Thus, g is a descent direction. But, solving this problem often is impractical. So, ǫ f(x) is approximated by the convex hull of its some elements. More exactly, if W = {v 1,v 2,...,v } ǫ f(x), then we consider convw as an approximation of ǫ f(x). Therefore, we solve the following problem, which is an approximation of (1), v 0 = arg min v H. v conv W 3

4 Minimization Algorithm In the previous Subsection, we found a descent direction based on a positive definite matrix and, an approximation of ǫ f(x). To update the inverse of Hessian matrix approximation, we need a pair of subgradients such that the secant equation is satisfied. Thus, by applying the line search algorithm, a step length satisfying in the GWC and, a pair of subgradients for updating the inverse of Hessian matrix approximation are computed. Now, we present the nonsmooth version of BFGS algorithm. Algorithm 1. Step 0 (Initialization) Let x 1 R n, v 1 1 f(x 1 ), θ ǫ, c 1, θ δ, ǫ 1,δ 1 (0,1), H 1 = I n n, c 2 (c 1,1) and, set = 1, where I n n is the identical matrix. Step 1 (Set new parameters) Set m = 1, H m = H and, x m = x. Step 2 (Compute descent direction) Apply Algorithm 2 in [4] at point x m, with H = Hm, v = vm, δ = δ and, ǫ = ǫ. Let n m be the number of iterations needed for termination of the algorithm and, let w m H m = min{ w H : w conv W m}. If wm H = 0 then Stop else let g m = Hm wm be the descent direction. Step 3 (Line search) If the stopping condition (2) is not satisfied then go to Step 5, else apply the line search algorithm. If the algorithm terminates successfully, then α is the line search parameter satisfying the GWC and, v m+1 ǫ f(x m +αgm ) is a vector such that < (vm+1 ),g m > +c 2 w m H m > 0, else α is the line search parameter satisfying the GAC. Construct the next iterate x m+1 = x m +αgm and, go to Step 4. Step 4 (BFGS update) If the line search algorithm terminates successfully, then set s = αg m and, y = vm+1 v m and H m+1 = H m Hm yyt H m < y,h m y > + sst < y,y >, 4

5 else set H m+1 = I. Set m = m+1 and, go to Step 2. Step 5 (Update parameters) Set ǫ +1 = ǫ θ ǫ, δ +1 = δ θ δ, v+1 1 = vm, x +1 = x m, H +1 = H m and, let = +1. Go to Step 1. Theorem 1. Let f : R n R be a locally Lipschitz function. If the level set M = {x : f(x) f(x 1 )}, is bounded, then either Algorithm 1 terminates finitely at some 0 and, m 0 with {w m0 0 } = 0 or every cluster point of the sequence {x }, generated by Algorithm 1, belongs to the set X = {x R n : 0 f(x)}. REFERENCES 1. A. A. Goldstein. Optimization of Lipschitz continuous functions, Mathematical Programming, 13:14 22, (1977). 2. D. P. Bertseas and S. K. Mitter. A descent numerical method for optimization problems with nondifferentiable cost functionals, SIAM Journal on Control, 11: , (1973). 3. M. Gaudioso and M. F. Monaco. A bundle type approach to the unconstrained minimization of convex nonsmooth functions, Mathematical Programming, 23(2): , (1982). 4. N. Mahdavi-Amiri and R. Yousefpour. An effective nonsmooth optimization algorithm for locally lipschitz functions, Accepted Journal of Optimization Theory Application. 5. P. Wolfe. A method of conjugate subgradients for minimizing nondifferentiable functions, Nondifferentiable Optimization, M. Balinsi and P. Wolfe, eds., Mathematical Programming Study, North- Holland, Amsterdam, 3: , (1975). 6. J. V. Bure, A. S. Lewis, and M. L. Overton. A robust gradient sampling algorithm for nonsmooth, nonconvex optimization, SIAM Journal of Optimization, 15: , (2005). 7. J. Nocedal and S. J. Wright Numerical optimization, Springer, (1999). 5