MATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N.

Transcription

1 MATH 3795 Lecture 13. Numerical Solution of Nonlinear Equations in R N. Dmitriy Leykekhman Fall 2008 Goals Learn about different methods for the solution of F (x) = 0, their advantages and disadvantages. Convergence rates. MATLAB s fsolve D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 1

2 Nonlinear Equations. Goal: Given a function f : R N R N we want to find x R N such that F (x ) = 0. Definition A point x with F (x ) = 0 is called a root of F or zero of F. We want to extend the methods from the last lecture, like Newton s method and Secant Method to find roots to vector valued problems. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 2

3 Convergence of Sequences. Let {x k } be a sequence of vectors in R N and let denote a vector norm. 1. The sequence is called q-linearly convergent if there exists c (0, 1) and ˆk N such that x k+1 x c x k x for all k ˆk. 2. The sequence is called q-superlinearly convergent if there exists a sequence {c k } with c k > 0 and lim k c k = 0 such that or, equivalently, if x k+1 x c k x k x x k+1 x lim k x k x = The sequence is called q-quadratically convergent to x if lim k x k = x and if there exists c > 0 and ˆk N such that x k+1 x c x k x 2 for all k ˆk. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 3

4 Taylor Expansion. Given a differentiable function F, we define its Jacobian F 1 F x 1 (x)... 1 x n (x) F (x) = JF (x) =. F n x 1 (x)... If F is continuously differentiable around x 0, then F (x) F (x 0 ) + F (x 0 )(x x 0 ). F n x n (x) D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 4

5 Taylor Expansion. Example Let F : R 2 R 2 be given by Then its Jacobian is If x 0 = ( 0 0 ( F (x) F (x) = ( x x e x1 1 + x ( ) F 2x1 2x (x) = 2 e x1 1 3x 2 2 ) close to ) ( x1, then for x = x 2 ) ( e e 1 0 ) ( 0 0 ) ( ) ( x1 = x 2 ) 2 e 1 (1 + x 1 ) 2 ) D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 5

6 Newton s Method. Suppose we are given an approximation x 0 of a root x of F. Taylor approximation of F around x 0 gives F (x ) = F (x 0 + (x x 0 )) F (x 0 ) + F (x 0 )(x x 0 ). We use M(x) = F (x 0 ) + F (x 0 )(x x 0 ) as a model for F. The root of M(x) i.e. the solution of linear system F (x 0 )(x x 0 ) = F (x 0 ) is used as an approximation of the root of F. Write the previous identity as F (x 0 )s 0 = F (x 0 ) x 1 = x 0 + s 0. step (correction) D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 6

7 Newton s Method. Input: Initial values x(0), tolerance tol, maximum number of iterations maxit Output: approximation of the root 1 For k = 0:maxit do 2 Compute F (x(k))s(k) = -F(x(k)). (LU-decomposition) 3 Compute x(k+1) = x(k) + s(k). 4 Check for truncation 5 End D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 7

8 Newton s Method. Example Consider F (x) = ( x x e x1 1 + x with starting point x 0 = (1.5, 2) T and stopping criteria F (x k ) 2 1e 10, the Newton s method gives the computed solution x = (1.0000, ) T and the history k x k 2 F (x k ) 2 s k e e e e e e e e e e e e e e e e e e e e e ) D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 8

9 Convergence of Newton s Method. Theorem Let D R N be an open set and let F : D R N be differentiable on D with Lipschitz continuous derivative, i.e. let L > 0 be such thtat F (y) F (x) L y x x, y D. If x D is a root and if F (x ) is nonsingular, then there exists an ɛ > 0 such that Newton s method with starting point x 0 with x 0 x < 0 generates iterates x k which converge to x, and which obey lim x k = x, k x k+1 x c x k x 2 for all k and some positive constant c. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 9

10 Convergence of Newton s Method. Theorem Let D R N be an open set and let F : D R N be differentiable on D with Lipschitz continuous derivative, i.e. let L > 0 be such thtat F (y) F (x) L y x x, y D. If x D is a root and if F (x ) is nonsingular, then there exists an ɛ > 0 such that Newton s method with starting point x 0 with x 0 x < 0 generates iterates x k which converge to x, and which obey lim x k = x, k x k+1 x c x k x 2 for all k and some positive constant c. Newton s method is locally q-quadratically convergent (under the assumptions stated in the theorem). D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 9

11 Stopping Criteria. We have discussed iterative methods which generate sequences x k with (under suitable conditions) lim k x k = x. When do we stop the iteration? For a given tolerance tol a > 0 we want to find x k such that x k x < tol a, stop if absolute error is small; or x k x < tol r x, stop if relative error is small. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 10

12 Stopping Criteria. We have discussed iterative methods which generate sequences x k with (under suitable conditions) lim k x k = x. When do we stop the iteration? For a given tolerance tol a > 0 we want to find x k such that x k x < tol a, stop if absolute error is small; or x k x < tol r x, stop if relative error is small. For some t k [0, 1] F (x k ) = F (x k ) F (x ) 1 2 (F (x )) 1 1 x k x, if x k is suff. close to x Hence, if F (x k ) < tol f, and if x k is sufficiently close to x, then x k x < 2tol f (F (x )) 1. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 10

13 Stopping Criteria. We have discussed iterative methods which generate sequences x k with (under suitable conditions) lim k x k = x. When do we stop the iteration? For a given tolerance tol a > 0 we want to find x k such that x k x < tol a, stop if absolute error is small; or x k x < tol r x, stop if relative error is small. For some t k [0, 1] F (x k ) = F (x k ) F (x ) 1 2 (F (x )) 1 1 x k x, if x k is suff. close to x Hence, if F (x k ) < tol f, and if x k is sufficiently close to x, then x k x < 2tol f (F (x )) 1. Limit maximum number of iterations. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 10

14 Variations of Newton s Method. Recall Newton s Method: Input: Initial values x 0, tolerance tol, maximum number of iterations maxit Output: approximation of the root 1. For k = 0,..., maxit do 2. Compute s k = f(x k )/f (x k ). 3. x k+1 = x k + s k. 4. Check for truncation 5. End Requires the evaluation of derivatives and solution of a linear system. Linear system solves are done using LU decomposition (cost 2/3n 3 flops for each matrix factorization). If Jacobian/derivative evaluations are expensive or difficult to compute, or if the LU factorization of the derivative is expensive, the following variations are useful Finite difference Newton method and Secant method. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 11

15 Finite Difference Newton Method. Recall F i (x) x j F i (x 1,..., x j 1, x j + h, x j+1,..., x n ) F i (x) = lim. h 0 h D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 12

16 Finite Difference Newton Method. Recall F i (x) x j F i (x 1,..., x j 1, x j + h, x j+1,..., x n ) F i (x) = lim. h 0 h Thus, for all i = 1,..., n F 1(x) x j. F n(x) x j = lim h 0 F (x 1,..., x j 1, x j + h, x j+1,..., x n ) F (x). h D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 12

17 Finite Difference Newton Method. Recall F i (x) x j F i (x 1,..., x j 1, x j + h, x j+1,..., x n ) F i (x) = lim. h 0 h Thus, for all i = 1,..., n F 1(x) x j. F n(x) x j = lim h 0 The idea is to replace F 1(x) x j. F n(x) x j F (x 1,..., x j 1, x j + h, x j+1,..., x n ) F (x). h 1 h j ( F (x1,..., x j 1, x j + h, x j+1,..., x n ) F (x) ). Good choice for the step size h j = ɛ x. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 12

18 Finite Difference Newton Method. Input: Initial values x 0, tolerance tol, maximum number of iterations maxit Output: approximation of the root 1. For k = 0,..., maxit do 2. Compute finite difference approximation of F (x k ) i.e., compute matrix B with columns B of F (x k ) Be j = 1 h j ( F (x1,..., x j 1, x j + h, x j+1,..., x n ) F (x) ) 3. Factor B 4. Solve Bs k = F (x k ) 5. Compute x k+1 = x k + s k 6. Check for truncation 7. End D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 13

19 Secant Method. Recall that in the secant method we replace the derivative f (x k+1 ) by f(x k) f(x k+1 ) x k x k+1. If we set b k+1 = f(x k) f(x k+1 ) x k x k+1 then b k+1 satisfies b k+1 (x k+1 x k ) = f(x k+1 ) f(x k ) which is called the secant equation The next iterate is given by x k+2 = x k+1 + s k+1, where s k+1 is the solution of b k+1 s k+1 = f(x k+1 ). D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 14

20 Secant Method. Try to extend this to the problem of finding a root of F : R n R n. Given two iterates x k, x k+1 R n we try to find a nonsingular matrix B k+1 R n n which satisfies the so-called secant equation B k+1 (x k+1 x k ) = F (x k+1 ) F (x k ). Then we compute the new iterate as follows Solve B k+1 s k+1 = F (x k+1 ), x k+2 = x k+1 + s k+1. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 15

21 Secant Method. There is a problem with this approach. If n = 1 the the secant equation has a unique solution. b k+1 (x k+1 x k ) = f(x k+1 ) f(x k ) If n > 1, the we need to determine n 2 entries of B k+1 from n equations B k+1 (x k+1 x k ) = F (x k+1 ) F (x k ). There is no unique solution. For example, for n = 2, x k+1 x k = (1, 1) T and F (x k+1 ) F (x k ) = (1, 2), then the matrices ( ) ( ) , satisfy secant equation. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 16

22 Secant Method. Therefore we chose B k+1 R n n as the solution of min B B k F. B(x k+1 x k )=F (x k+1 ) F (x k ) Motivation: B k+1 should satisfy the secant equation B(x k+1 x k ) = F (x k+1 ) F (x k ) and B k+1 should be as close to the old matrix B k as possible to preserve as much information contained in B k as possible. Common notation s k = x k+1 x k and y k = F (x k+1 ) F (x k ). With these definition the previous minimization problem becomes Unique solution min B B k F. s:t: Bs k =yk B k+1 = B k + (yk B ks k )s T k s T k s k Broyden update. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 17

23 Broyden s Method. Input: Initial values x 0, tolerance tol, maximum number of iterations maxit Output: approximation of the root 1 For k = 0,..., maxit do 2 Solve B k s k = F (x k ) for s k 3 Set x k+1 = x k + s k. 4 Evaluate F (x k+1 ) 5 Check for truncation 6 Set y k = F (x k+1 ) F (x k ) 7 Set B k+1 = B k + (yk B ks k )s T k s T k s k 8 End. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 18

24 Broyden s Method. Note that due to the definition of y k and s k we have that B k+1 = B k + (yk B ks k )s T k s T k s k = B k + F (x k+1)s T k s T k s k To start Broyden s method we need an initial guess x 0 for the root x and an initial matrix B 0 R n n. In practice one often chooses B 0 = F (x 0 ) or B 0 = γi; where γ is a suitable scalar. Other choices, for example finite difference approximations to F (x 0 ) or choices based on the specific structure of F are also used. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 19

25 Broyden s Method. Example Consider with starting point x 0 = F (x) = ( ( x x e x1 1 + x and stopping criteria F (x k ) 2 1e 10. ) ) and B 0 = F (x 0 ), D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 20

26 Broyden s Method. the Broyden s method produces k x k 2 F (x k ) 2 s k e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 21

27 MATLAB s fsolve. Similar to a build-in function fzero, Optimization Toolbox has a function fsolve that tries to solve a system of nonlinear equations F (x) = 0. Warning: You need this Toolbox in order to use this function. Syntax: x = fsolve(fun,x0) x = fsolve(fun,x0,options) x = fsolve(problem) [x,fval] = fsolve(fun,x0) [x,fval,exitflag] = fsolve(...) [x,fval,exitflag,output] = fsolve(...) [x,fval,exitflag,output,jacobian] = fsolve(...) D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 22

28 MATLAB s fsolve. x = fsolve(fun,x0) starts at x0 and tries to solve the equations described in fun. [x,fval] = fsolve(fun,x0) returns the value of the objective function fun at the solution x. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 23

29 MATLAB s fsolve. Thus in our example with ( ) x 2 F (x) = 1 + x e x1 1 + x [x,fval] = fsolve( ex1,[1.5 2]) produces and x 0 = ( ) x = fval = 1.0e-012 * D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 24

30 MATLAB s fsolve. Example Let s say you want to solve 2x 1 x 2 = e x1 x 1 + 2x 2 = e x2. In other words we want to solve nonlinear system 2x 1 x 2 e x1 = 0 x 1 + 2x 2 e x2 = 0. D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 25

31 MATLAB s fsolve. First we create a function in a m-file. function varargout = ex3(x) % % % [F] = ex3(x) returns the function value in F % [F,Jac] = ex3(x) returns the function value in F and % the Jacobian in Jac % % return the function value varargout{1} = [ 2*x(1)-x(2)-exp(-x(1)); -x(1) + 2*x(2) - exp(-x(2))]; if( nargout > 1 ) % return the Jacobian as the second argument varargout{2} = [ 2+exp(-x(1)) -1; -1 2+exp(-x(2))]; end D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 26

32 MATLAB s fsolve. Now calling [x,fval] = fsolve( ex3,[-5-5]) produces x = fval = 1.0e-006 * D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 27

33 MATLAB s fsolve. We got 6 digits of accuracy. If we need more we can use options = optimset( TolFun,1e-10) then calling [x,fval] = fsolve( ex3,[-5-5],options) produces x = fval = 1.0e-013 * D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 28

34 MATLAB s fsolve. You could learn more about fsolve by typing help fsolve or looking at the function on mathworks website fsolve Similarly you learn there more about the options by typing help optimset or just optimset D. Leykekhman - MATH 3795 Introduction to Computational Mathematics Linear Least Squares 29