Maria Cameron. f(x) = 1 n

Transcription

1 Maria Cameron 1. Local algorithms for solving nonlinear equations Here we discuss local methods for nonlinear equations r(x) =. These methods are Newton, inexact Newton and quasi-newton. We will show that the Newton method converges Q-superlinearly if the Jacobian J is continuous and J(x ) is nonsingular at the solution point x. Furthermore, the convergence is Q-quadratic if the Jacobian is Lipschitz-continuous. However, the Newton method is not globally convergent. In inexact Newton methods, the step is made not all the way to the solution of the linear model (recall the 1d case where at each step we jump to the point where the tangent line intersects the x-axis), but just some distance along this direction, making sure that the residual is sufficiently reduced. This strategy allows us to avoid solving a linear system of equations that is expensive to do in high dimensions. Contrary to inexact Newton methods, the quasi-newton methods (e.g., Broyden s method) do not use the exact Jacobian but an approximation to it. This approximation is built in the process of iterations. This is very handy in the case where the exact Jacobian is hard to calculate. All these methods exhibit a fast super linear convergence under certain conditions but suffer the same shortcoming: their behavior can be erratic unless the initial approximation is close enough to the solution. However, one can improve their behavior dramatically by keeping track of some merit function, often chosen to be f(x) = 1 n rj 2 (x), 2 and using the step sizes wisely. Then the problem of finding a zero of r(x) is reduced to finding a minimum of f(x), hence to a minimization problem. We will discuss it in the chapter Optimization. j=1 2. Newton method for solving nonlinear equations In the Newton method, as we have done it in 1d, at every iteration, we define linear model M k := r(x k ) + J(x k )p. Then we make a step p k right into the zero of the liner model, i.e., the Newton step p k+1 is the solution of J(x k )p k = r(x k ). Thus the algorithm is the following. Algorithm 1: Newton Method Choose x ; While r(x k ) > tol Solve J(x k )p = r(x k ) for p k ; 1

2 2 Set x k+1 = x k + p k ; End The local convergence properties of the Newton method are summarized in the following theorem. Theorem 1. Suppose that r is continuously differentiable in a convex open set D R n. Let x D be a non degenerate solution of r(x) =, and let {x k } be the sequence of iterates generated by Algorithm 1. Then when x k D is sufficiently close to x, we have x k+1 x = o( x k x ), indicating local Q-superlinear convergence. When r is Lipschitz-continuously differentiable near x, we have for all x k sufficiently close to x that indicating local Q-quadratic convergence. x k+1 x = O( x k x 2 ), Proof. The proof is conducted in three steps. First we show that r(x k ) = J(x k )(x k x ) + o( x k x ). Then we show that x k+1 x = x k + p k x = x k + J 1 (x k )r(x k ) x = o( x k x ). Finally we will show that the Lipschitz continuity of J leads to Q-quadratic convergence. (1) (1) Since r(x ) = we have r(x k ) = r(x k ) r(x ) = J(x k )(x k x ) + = J(x k )(x k x ) + o( x k x ). Let us explain why [J(x k + t(x x k )) J(x k )](x k x )dt [J(x k + t(x x k )) J(x k )](x k x )dt = o( x k x ). We can bound this integral from above in absolute value as [J(x k + t(x x k )) J(x k )](x k x 1 )dt J(x k +t(x x k )) J(x k ) (x k x ) dt. Set p k := x k x. Since J(x) is continuous we have lim J(x k + tp k ) J(x k ) p k dt =. p p k This implies Eq. (1). (2) Since J(x ) is nonsingular, there exists a ball B δ (x ) of radius δ around x and a positive constant β such that J 1 (x) β, x B δ (x ) D.

3 If x k B δ (x ) we have p k = J 1 (x k )r(x k ) = J 1 (x k )(J(x k )(x k x )+o( x k x )) = x k +x +o( x k x )). Hence x k+1 x = x k + p k x = o( x k x ). (3) If J(x) is Lipschitz-continuous we have [J(x k + t(x x k )) J(x k )](x k x )dt Therefore Hence Then J(x k + t(x x k )) J(x k ) (x k x ) dt Mt x k x 2 dt =O( x k x 2 ). r(x k ) = J(x k )(x k x ) + O( x k x 2 ). p k = J 1 (x k )r(x k ) = J 1 (x k )(J(x k )(x k x ) + O( x k x 2 )) = x k + x + O( x k x 2 ). x k+1 x = x k + p k x = O( x k x 2 ), i.e., the Newton method converges Q-quadratically Inexact Newton methods. Solving equation J(x k )p k = r(x k ) for p k exactly can be expensive in high dimensions. Inexact Newton methods use the search directions p k that satisfy the condition (2) r k + J k p k η k r k, for some η k [, η], where η is a constant. The most important methods in this class make use of iterative techniques for solving linear systems of the form Jp = r. The convergence theory for these methods depends only on the condition (2) and not on particular technique used to calculate p k. Its results are summarized in the following theorem. Theorem 2. Suppose that r is continuously differentiable in a convex open set D R n. Let x D be a non-degenerate solution of r(x) =, and let x k be a sequence of iterates generated by an inexact Newton method. Then when x k D is sufficiently close to x, the following are true: If η in Eq. (2) is sufficiently small, then the convergence of {x k } to x is Q-linear. If η k, the convergence is Q-superlinear. 3

4 4 If, in addition, J is Lipschitz-continuous on D and η k = O( r k ), then the convergence is Q-quadratic. Exercise Prove it Broyden s method. Secant methods, also known as quasi-newton methods, do not require calculation of the Jacobian J(x). Instead, they construct their own approximation to J(x), updating it at each iteration so that it mimics the behavior of the true Jacobian over the step just taken. Example Secant method in 1D is defined by the recurrence relation x n x n 1 (3) x n+1 = x n f(x n ) f(x n ) f(x n 1 ). Exercise Prove that it converges with the order (4) α = Let B k be the Jacobian approximation at iteration k. Assuming that B is nonsingular we define the step p k to the next iterate as the solution of The standard notations are B k p k + r(x k ) =, x k+1 = x k + p k. s k = x k+1 x k, y k = r(x k+1 ) r(x k ). At this point, s k = p k, but in the practical methods where a merit function is being monitored, usually a step is done in the direction of p k but of different length. The length is chosen to satisfy Wolfe s conditions that we will discuss later. We have that s k and y k are related via y k = J(x k + ts k )s k dt J(x k+1 )s k + o( s k ). We require the updated Jacobian to satisfy the following equation known as the secant equation (5) y k = B k+1 s k, which insures that B k+1 and J(x k+1 ) have similar behavior along the direction s k. Eq. (5) does not determine B k+1 uniquely for n > 1: It is a system of n equations with n 2 unknowns. Various quasi-newton methods differ in the way how they determine B k+1 satisfying Eq. (5). The most successful practical algorithm is Broyden s method for which the update formula is (6) B k+1 = B k + (y B ks k )s T k The Broyden update makes the smallest possible change to the Jacobian measured by the Euclidean norm B k B k+1.

5 Theorem 3. Among all maurices B satisfying Bs k = y k, the matrix B k+1 defined Eq. (6) minimizes the difference B B k. Proof. B k+1 B k = = (y k B k s k )s T k (B B k )s k s T k B B k s k s T k s T k s = B B k. k Here we have used the property of the norm that AB A B and the fact that s k s T k s T k s = 1. k This is easy to check. Since s k s T k is symmetric, we have s k s T k s T k s = max x T s k s T k x k x =1 s T k s = s k 2 k s k 2 = 1. Broyden s algorithm work as follows. Algorithm: Broyden s method Choose x and B ; for k =, 1, 2,... Solve B k p k = r(x k ) for p k ; Choose α k by performing a line search along direction p k ; x k+1 = x k + α k p k ; s k = x k+1 x k ; y k = r(x k+1 ) r(x k ); B k+1 = B k + (y B ks k )s T k end ; Under certain assumptions, Broyden s method converges super linearly, that is, x k+1 x = o( x k x ). Theorem 4. Suppose that r is continuously differentiable in a convex open set D R n. Let x D be a non degenerate solution of r(x) =. Then there are positive constants ɛ and δ such that if the starting point x and the initial approximate Jacobian B satisfy the sequence {x k } generated by x x δ, B J(x ) ɛ, x k+1 = x k B 1 k r(x k), s k = x k+1 x k, B k+1 = B k + (y B ks k )s T k 5

6 6 is well-defined and converges Q-superlinearly to x. References [1] C. T. Kelley, Iterative Methods for Linear and Nonlinear Equations, SIAM, [2] J. Nocedal, S. Wright, Numerical Optimization, Springer, 1999