4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion:

Transcription

1 Unconstrained Convex Optimization 21 4 Newton Method H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: f(x + p) f(x)+p T f(x)+ 1 2 pt H(x)p ˆf(p) In general, ˆf(p) won t approximate f well at all, except in a neighborhood of the point x of the taylor series expansion. But if our step p is not too big, it is okay. If we try to minimize ˆf(p) with respect to p by setting the derivative equal to 0: f(x)+h(x)p = 0 we need to define p to be: H(x)p = f(x) The Newton direction is obtained by solving the linear system involving the Hessian matrix and the negative gradient. If f is actually a quadratic function, the second-order Taylor approximation is exact and x + p will be the global minimum. How well does the Newton Method work? Theorem: Suppose f C 2 (S) and there is a positive scalar λ such that H(x) H(y) λ x y for all points x, y in a neighborhood of x. Then if x (k) is sufficiently close to x and if H(x ) is positive definite, then there exists constant c such that: e (k+1) c e (k) 2 Let e (k) = x (k) x be the error at iteration k. Use Taylor series: 0 = f(x (k) e (k) )= f(x (k) ) H (k) e (k) +O( e (k) 2 ) Multiplying by (H (k) ) 1, we get: Because 0 = p (k) e (k) + O( e (k) 2 ) p (k) e (k) =(x (k) x (k+1) ) (x (k) x )= e (k+1)

2 Unconstrained Convex Optimization 22 So e (k+1) = O( e (k) 2 ). This rate of convergence is quadratic convergence, which means if we have an error of 10 1 t some iteration, two iterations later the error will be about 10 4 (if c 1). After four iterations, it will be about (machine precision!). When Newton s method works, it often converges very fast to very high accuracy! Example: Implement Newton method with backtracking linesearch, where c =0.1,ρ= 1 2. Test it on the function f(x) =(x x 2 2), starting x =(50, 50). f(x) =e x 1+3x e x 1 3x e x 1 0.1, starting x =(2.0, 1.0). How do we set initial length of α during linesearch? For Newton methodsthe initial step should be always set to be α 0 =1, to ensure the unit step lengths are taken whenever they satisfy the termination condition and allows the rapid rate-of-convergence of these methods. When does Newton s method work? Recall 1: H(x)p = f(x) involves solving the linear system of the Hessian matrix and the negative gradient. For convex functions, H(x) is p.s.d. Recall 2 (Matrix Basics): Singularity of a matrix: A matrix A is said to be nonsingular if it has an inverse, that is if there exists a matrix A 1 such that AA 1 = A 1 A = I.

3 Unconstrained Convex Optimization 23 The existence of A 1 is directly related to the determinat of A, denoted as det(a) = n j=1 ( 1)i+j a ij M ij for i =1, 2,,nwhere M ij is the determinant of the matrix obtained by deleting the row and column containing a ij. Matrix A is nonsingular if det(a) = 0. A p.d. matrix is non-singular. A p.s.d matrix can be singular or non-singular. A singular p.s.d. matrix has at least one eigenvalue =0. So when does Newton s Method work? If H(x) is positive definite (H(x) is non-singular) and not closes to singular. this linear system has a unique solution, and p T f(x) =p T H(x)p > 0 so p is a descent direction. If H(x) is close to singular ( H(x) can still be positive definite) We will have computational difficult to solve the linear system. If H(x) fails to be positive definite ( H(x) maybe singular) We may fail to have a solution to the linear system. We may take ascent directions. To run the Newton method successfully, we need the Hessian H(x) to be positive definite everywhere we need to evaluate it. Conclusion: Advantages of Newton method: When it can be run successfully, it will usually reliably and quickly converges to the solution with very good accuracy. Disadvantages of Newton method: It requires the computation and storage of H(x) at each iteration. It requires solving a linear system involving H(x). For a dense N N matrix, the most reliable methods of finding this Newton step takes O(N 3 ) per iteration.

4 Unconstrained Convex Optimization 24 It can fail if H(x) fails to be positive definite. Further reading - Newton Method with Modified Hessian. We can modify Newton whenever we are not sure that the direction it generates is descent. Use the Hessian whenever it is p.d. and not close to singular. Replace H(x) by Ĥ(x) =H(x)+E whenever H(x) is close to singularity or fails to be p.d. So that Ĥ(x) is symmetric p.d. Ĥ(x) is not too close to singular, i.e., its smallest eigenvalue is bounded below by a constant bigger than zero. Popular methods: Greenstadt s method: Modify eigenvalues. Levenberg-Marquardt method: Add a scaled identity matrix Modified Cholesky Stratigies: Perform Choleskey factorization of the Hessian and modify the diagonal elements

5 Unconstrained Convex Optimization 25 Newton method on quadratic function Newton method on exponential function Figure 4.1: Example of Newton method performances.