Multivariate Newton s Method

Transcription

1 Multivariate Newton s Method 1 Nonlinear Systems derivation of the method examples with Julia 2 Nonlinear Optimization computing the critical points with Newton s method MCS 471 Lecture 6(b) Numerical Analysis Jan Verschelde, 29 June 2018 Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

2 Multivariate Newton s Method 1 Nonlinear Systems derivation of the method examples with Julia 2 Nonlinear Optimization computing the critical points with Newton s method Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

3 Newton s method for nonlinear systems Consider a system of two equations in two variables: { f (x, y) = 0 g(x, y) = 0. Suppose we have an approximation for a solution (x 0, y 0 ) and we would like to compute x and y so x 1 = x 0 + x and y 1 = y 0 + y satisfy the system: { f (x1, y 1 ) = f (x 0 + x, y 0 + y) = 0 g(x 1, y 1 ) = g(x 0 + x, y 0 + y) = 0. How to compute x and y? Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

4 Taylor series in two variables f (x 0 + x, y 0 + y) = f (x 0, y 0 ) + f x (x 0, y 0 ) x + f y (x 0, y 0 ) y + g(x 0 + x, y 0 + y) = g(x 0, y 0 ) + g x (x 0, y 0 ) x + g y (x 0, y 0 ) y + where f x (x 0, y 0 ) and f y (x 0, y 0 ) are the partial derivatives of f with respect to x and y evaluated at (x 0, y 0 ); g x (x 0, y 0 ) and g y (x 0, y 0 ) are the partial derivatives of g with respect to x and y evaluated at (x 0, y 0 ); and the represent the higher order terms in the series, in ( x) 2, ( x)( y), and ( y) 2. Because x and y are already small numbers, the higher order terms are even smaller. Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

5 in matrix format Because f (x 0 + x, y 0 + y) = 0 and g(x 0 + x, y 0 + y) = 0: 0 = f (x 0, y 0 ) + f x (x 0, y 0 ) x + f y (x 0, y 0 ) y + 0 = g(x 0, y 0 ) + g x (x 0, y 0 ) x + g y (x 0, y 0 ) y + we solve for x and y: f x (x 0, y 0 ) g x (x 0, y 0 ) f y (x 0, y 0 ) [ x g y (x y 0, y 0 ) ] [ f (x0, y = 0 ) g(x 0, y 0 ) ]. Th solution ( x, y) of the linear system updates x 0 and y 0 : x 1 := x 0 + x and y 1 := y 0 + y. Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

6 the Jacobian matrix Given a system of n equations in m unknowns f(x) = 0, with f = (f 1, f 2,..., f n ) and x = (x 1, x 2,..., x m ), f(x) = f 1 (x 1, x 2,..., x m ) = 0 f 2 (x 1, x 2,..., x m ) = 0. f n (x 1, x 2,..., x m ) = 0, the Jacobian matrix of f is the matrix of all first order partial derivatives: f 1 f 1 f 1 x 1 x 2 x m f 2 f 2 f 2 J f = x 1 x 2 x m f n f n f n x 1 x 2 x m Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

7 a numerical example Consider the system f(x, y) = { e x y = 0 xy e x = 0 f(1, e) = 0. Let us do one step with Newton s method, starting at (0.9, 2.5). The Jacobian matrix is [ e J f = x ] [ ] E E +0 y e x A = J x f (0.9, 2.5) = 4.0 E E 1 f(0.9, 2.5) = [ 4.0 E E 1 ] x = 1.1 E 1, x = E +0 y = 2.3 E 1, y = E +0 Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

8 Multivariate Newton s Method 1 Nonlinear Systems derivation of the method examples with Julia 2 Nonlinear Optimization computing the critical points with Newton s method Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

9 computing the Jacobian matrix with SymPy using SymPy x, y = Sym("x, y") function SymPyDerivatives(f::String, g::string) # # Returns the string representations of all # partial derivatives of the expression in x and y, # given in the strings f and g. # formf = parse(f) evalformf = eval(formf) fx = diff(evalformf, x) fy = diff(evalformf, y) formg = parse(g) evalformg = eval(formg) gx = diff(evalformg, x) gy = diff(evalformg, y) return [string(fx) string(fy); string(gx) string(gy)] end Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

10 code for one Newton step function NewtonStep(fun::Array{SymPy.Sym,1}, jac::array{sympy.sym,2}, x0::float64, y0::float64) # # Runs one step with Newton s method # valfun = -SymPyFun(fun, x0, y0) nfx = norm(valfun) valmat = SymPyMatrixEvaluate(jac, x0, y0) update = valmat\valfun ndx = norm(update) x1 = x0 + update[1] y1 = y0 + update[2] sfx nfx) sdx ndx) sx1 x1) sy1 y1) println(" $sfx $sdx $sx1 $sy1 ") return [x1, y1] end Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

11 we observe quadratic convergence In the output below, there are four columns: 1 the norm of the residual, 2 the norm of the update, 3 the value for x, 4 the value for y. Observe the quadratic convergence: 2.13e e e e e e e e e e e e e e e e e e e e+00 Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

12 Multivariate Newton s Method 1 Nonlinear Systems derivation of the method examples with Julia 2 Nonlinear Optimization computing the critical points with Newton s method Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

13 nonlinear optimization Consider the minimization of maximization of a function in n variables, x = (x 1, x 2,..., x n ). f (x 1, x 2,..., x n ) The minima and maxima occur wher the gradient vanishes. ( f f =, f,..., f ),. x 1 x 2 x n All critical points satisfy f x 1 (x) = 0 f x 2 (x) = 0. f x n (x) = 0 Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14

14 the Hessian If we take the Jacobian matrix of f x 1 (x) = 0 f x 2 (x) = 0. f x n (x) = 0 then we arrive at the second partial derivatives of f : 2 f x1 2 (x) 2 f x 2 x 1 (x). 2 f x nx 1 (x) 2 f x 1 x 2 (x) 2 f x 1 x n (x) 2 f (x) 2 f x2 2 x 2 x n (x) f x nx 2 (x) 2 f (x) xn 2 If f is continuous, then the matrix is symmetric. If close to a minimum, then the matrix is positive definite. Numerical Analysis (MCS 471) Multivariate Newton s Method L-6(b) 29 June / 14