5 Quasi-Newton Methods

Transcription

1 Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min f(x) x where f(x) can be computed, but not H(x). Two options: Estimate H(x). using finite differences: discrete Newton (not discussed in the class). Approximate H(x) using Quasi-Newton methods. The basic idea of Quasi-Newton methods The Newton step (H and f(x) evaluated at each iteration): p = H 1 f(x) The Quasi-Newton step: accumulate an approximation B (k) H (k) using free information p = B 1 f(x) How do we approximate H by B? Secant equation: How do we want B (k+1) to behave in the direction of f(x (k+1) ) f(x (k) )? At step k, we have f(x (k) ) and we compute f(x (k+1) ) where x (k+1) x (k) = s (k). H(x) (k) satisfies: H (k) s (k) =lim h 0 f(x (k) + hs (k) ) f(x (k) ) h We ask the same property of our approximation B (k+1) : B (k+1) s (k) = f(x (k+1) ) f(x (k) )

2 Unconstrained Convex Optimization 27 Least-change conditions: min B (k+1) B (k) B (k+1) subject to the secant condition. The solution depends on the choice of norm. We can impose other constraints as well, such as: We might want to keep B (k+1) positive definite. We might know that H is sparse and we want B to have the same structure. Popular Quasi-Newton methods: DFP (Davidon 1959, Flecher-Powell 1963) The choice of norm in the minimization problem: W 1/2 (B (k+1) B (k) )W 1/2 F where W 1 is a symmetric p.d. matrix satisfying the secant condition. (In fact, the solution is independent of the choice of such W. The results upgrade C rather than B: C (k+1) = C (k) C(k) y (k) y (k)t C (k) y (k)t C (k) y (k) + s(k) s (k)t y (k)t s (k) where y (k) = f(x (k+1) ) f(x (k) ). BFGS (Broyden, Fletcher, Goldfarb, Shanno 1970) The choice of norm in the minimization problem: W 1/2 (C (k+1) C (k) )W 1/2 F where W is a symmetric p.d. matrix satisfying the secant condition. (In fact, the solution is independent of the choice of such W. The results upgrade B: B (k+1) = B (k) B(k) s (k) s (k)t B (k) s (k)t B (k) s (k) + y(k) y (k)t y (k)t s (k) Note that this method is in some sense dual to DFP. Stability of the algorithm

3 Unconstrained Convex Optimization 28 Updating C can be hazardous when H is close to singular. Updating B leaves the problem of solving a linear system at each iteration. Example: There are two economical alternatives for BFGS to solving the linear systems: 1. Updating the inverse B (k) (F (k) = B (k) 1 ). By the Sherman-Morrison-Woodbury formula for computing inverse of matrices updated by a rank-2 correction, we can obtain F (k+1) = F (k) + y(k)t (F (k) y (k) + s (k) ) s (k) s (k)t s(k) y (k)t F (k) + F (k) y (k) s (k)t (y (k)t s (k) ) 2 y (k)t s (k) 2. Updating a factorization. If we have a Cholesky factorization of B (k) as B (k) = L (k) D (k) L (k)t, then we want B (k+1) = L (k+1) D (k+1) L (k+1)t. This can be formed by formulas analogous to the Sherman-Morison-Woodbury formula. Updating a factorization can help to enforce symmetry and positive definiteness. Convergence rate With an exact line search, all of these methods have an n, 2n, or (n +2) step quadratic convergence rate. As an example, an n step quadratic convergence rate means that: x (k+n) x lim < k x (k) x 2 Weakening the line search to a Wolfe or Goldstein-Armijo search generally gives a superlinear convergence. How do we check optimality / convergence? H (k) B (k) should decrease as k increases. The update should not fail, meaning there should be no division by zero, and the change should be non-negligible. The algorithm should behave well on quadratic functions (Newton method on quadratics: terminate in 1 step; most Quasi-Newton methods have n or n +1 step termination). How the algorithm looks like putting the picture together? Until x (k) is a good enough solution,

4 Unconstrained Convex Optimization 29 Compute a search direction p (k) from p (k) = C (k) f(x (k) ) (or solve B (k) p (k) = f(x (k) ). set x (k+1) = x (k) + α (k) p (k) where α (k) satisfies the Goldstein-Armijo or Wolfe linesearch conditions Update C (k+1) (or B (k+1) ). k k +1. Example: Implement BFGS method with backtracking linesearch, where c =0.1,ρ = 1 2. Test it on the function f(x) =(x x 2 2), starting x =(50, 50). f(x) =e x 1+3x e x 1 3x e x 1 0.1, starting x =(2.0, 1.0). How do we set initial length of α during linesearch? For Quasi-Newton methods the initial step should be always set to be α 0 =1, to ensure the unit step lengths are taken whenever they satisfy the termination condition and allows the rapid rate-of-convergence of these methods. Further reading - Limited Memory Quasi-Newton Method: Stores only previous l gradient vectors and use that to compute the descent direction.

5 Unconstrained Convex Optimization 30 BFGS on quadratic function BFGS on exponential function Figure 5.1: Example of BFGS performances.