Nonlinear Equations and Continuous Optimization

Transcription

1 Nonlinear Equations and Continuous Optimization Sanzheng Qiao Department of Computing and Software McMaster University March, 2014

2 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

3 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

4 Problem setting Find roots f(x) = 0 Often, methods are iterative (roots cannot be found in finite number of steps). Example Compute square roots x 2 A = 0

5 Problem setting Find roots f(x) = 0 Often, methods are iterative (roots cannot be found in finite number of steps). Example Compute square roots x 2 A = 0 Find the side of the square whose area is A

6 Compute square roots Start with a rectangle whose one side is x c, then the other side is A/x c so that its area is A. Make the rectangle more square by setting the new side: x + = 1 ) (x c + Axc 2 Then x c = x + and iterate.

7 Compute square roots Start with a rectangle whose one side is x c, then the other side is A/x c so that its area is A. Make the rectangle more square by setting the new side: x + = 1 ) (x c + Axc 2 Then x c = x + and iterate. A better form (x c Axc ) x + = x c 1 2

8 Compute square roots Three issues to be addressed Initialization (x 0 ) Convergence (x k x?) and rate (how fast?) Termination

9 Initialization Write A in base 4: A = m 4 e, 0.25 m < 1 then A = m 2 e. Now we can assume 4 1 A < 1. Linear interpolation of f(a) = A at A = 0.25, 1.0: p(a) = (1+2A)/

10 Initialization (cont.) Initial error bound: Differentiating 1+2A A 3 with respect to A and then setting the derivative to zero to find the maximum, it can be shown that A (1+2A)/3 0.05

11 Initialization (cont.) Initial error bound: Differentiating 1+2A A 3 with respect to A and then setting the derivative to zero to find the maximum, it can be shown that A (1+2A)/ Initial value: x 0 = (1+2A)/3 Initial error: e

12 Convergence A relation between x k+1 and x k : x k+1 = 1 ) (x 2 k + Axk Denote the error e k = x k A, then the relation between e k+1 and e k : e k+1 = x k+1 A = 1 2 ( x k ) 2 A xk = 1 2 x k e2 k

13 Convergence A relation between x k+1 and x k : x k+1 = 1 ) (x 2 k + Axk Denote the error e k = x k A, then the relation between e k+1 and e k : e k+1 = x k+1 A = 1 2 ( x k ) 2 A xk = 1 2 x k e2 k It can be shown that 0.5 x k 1.0.

14 Convergence (cont.) Since the initial error e , e k e 2 k 1 e2k 0 (0.05)2k We have shown the convergence (e k 0 as k ).

15 Convergence (cont.) Since the initial error e , e k e 2 k 1 e2k 0 (0.05)2k We have shown the convergence (e k 0 as k ). How fast? Rate: quadratic e k+1 cek 2, each iteration doubles the accuracy.

16 Termination Recall: e k (0.05) 2k < 10 2k.

17 Termination Recall: e k (0.05) 2k < 10 2k. When k = 3, e k < When k = 4, e k <

18 Termination Recall: e k (0.05) 2k < 10 2k. When k = 3, e k < When k = 4, e k < Three iterations are enough for IEEE single precision (2 24 ). Four iterations are enough for IEEE double precision (2 53 ).

19 Example Compute 3

20 Example Compute 3 Scale: 3 =

21 Example Compute 3 Scale: 3 = Initial: x 0 = ( )/3 = 2.5/3

22 Example Compute 3 Scale: 3 = Initial: x 0 = ( )/3 = 2.5/3 Iterate: x n+1 = x n (x n 0.75/x n )/2 x 5 = x 4. n x n error < 10 16

23 Example Compute 3 Scale: 3 = Initial: x 0 = ( )/3 = 2.5/3 Iterate: x n+1 = x n (x n 0.75/x n )/2 x 5 = x 4. Scale back: x n x n error < 10 16

24 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

25 Generic algorithm If f(a) f(b) 0 and f(x) is continuous on [a, b], then f(x) has a root on [a, b]. while (b-a)>tol m = (a+b)/2; if f(a)*f(m)<=0 b = m; else a = m; end; end; r = (a + b)/2;

26 Generic algorithm Two problems in the generic algorithm: The while-loop may not terminate.

27 Generic algorithm Two problems in the generic algorithm: The while-loop may not terminate. When a and b are two neighboring floating-point numbers and (b-a)>tol,(a+b)/2 is rounded to either a or b.

28 Generic algorithm Two problems in the generic algorithm: The while-loop may not terminate. When a and b are two neighboring floating-point numbers and (b-a)>tol,(a+b)/2 is rounded to either a or b. Redundant function evaluations.

29 An improved algorithm fa = f(a); while (b-a)>tol + eps*max( a, b ) m = (a + b)/2; fm = f(m); if fa*fm<=0 b = m; else a = m; fa = fm; end; end; r = (a + b)/2;

30 An improved algorithm fa = f(a); while (b-a)>tol + eps*max( a, b ) m = (a + b)/2; fm = f(m); if fa*fm<=0 b = m; else a = m; fa = fm; end; end; r = (a + b)/2; Note: eps*max( a, b ) is about the distance between two consecutive floating-point numbers near max( a, b ). (ulp)

31 Convergence Since b k a k (b a)/2 k, x [a k, b k ], and x k = (a k + b k )/2, we have e k = x k x b k a k 2 = b a 2 k+1 0 In this case, e k+1 0.5e k. Improve accuracy by 1 bit per iteration or 1 decimal digit for every three or so iterations.

32 Convergence In general, linear convergence rate: for some constant c < 1. e k+1 ce k

33 Convergence In general, linear convergence rate: for some constant c < 1. e k+1 ce k Difficulty: Locate the interval [a, b].

34 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

35 Idea The tangent line of f(x) at x c : Set y = 0 y = f(x c )+(x x c )f (x c )

36 Newton s method Newton s method x + = x c f(x c) f (x c )

37 Newton s method Newton s method x + = x c f(x c) f (x c ) Example. Square root problem revisited, find a zero of f(x) = x 2 A x + = x c x2 c A = x c 1 ) (x c Axc 2x c 2

38 Complex case Example f(x) = x 2 + x + 1 (zeros ( 1±i 3)/2) i x i error 0 i i i i i i converge x 6 = x 5

39 Convergence No guarantee of convergence (unlike bisection). For example, f(x) = atan(x), x + = x c (1+x 2 c)atan(x c ) x 0 = 1.5 (> )

40 Convergence No guarantee of convergence (unlike bisection). For example, f(x) = atan(x), x + = x c (1+x 2 c )atan(x c) x 1 =

41 Convergence No guarantee of convergence (unlike bisection). For example, f(x) = atan(x), x + = x c (1+x 2 c )atan(x c) x 2 =

42 Convergence No guarantee of convergence (unlike bisection). For example, f(x) = atan(x), x + = x c (1+x 2 c)atan(x c ) x 3 =

43 Convergence f(x) = atan(x), x + = x c (1+x 2 c)atan(x c ) x 0 = 1.3 ( 1.3 < )

44 Convergence f(x) = atan(x), x + = x c (1+x 2 c)atan(x c ) x 1 =

45 Convergence f(x) = atan(x), x + = x c (1+x 2 c)atan(x c ) x 1 =

46 Convergence f(x) = atan(x), x + = x c (1+x 2 c)atan(x c ) x 1 =

47 Convergence Conditions for convergence (qualitative): x 0 close enough to x f (x) does not change sign near x f(x) is not too nonlinear near x Newton s method is a local method.

48 Convergence Conditions for convergence (qualitative): x 0 close enough to x f (x) does not change sign near x f(x) is not too nonlinear near x Newton s method is a local method. Difficulty: Finding x 0.

49 Hybrid methods Combining bisection and Newton s methods Bracketing interval [a, b], and x c = a or b if x + = x c f(x c )/f (x c ) [a, b] bracketing interval [a, x + ] or [x +, b] else m = (a+b)/2; bracketing interval [a, m] or [m, b] Termination criteria: Any one of (b k a k ) < δ f(x c ) < δ Too many function evaluations

50 Avoiding derivatives Approximation f (x c ) f(x c +δ c ) f(x c ) δ c

51 Avoiding derivatives Approximation f (x c ) f(x c +δ c ) f(x c ) δ c Choice of δ c Example. Secant method (δ c = x x c ) f (x c ) f(x c) f(x ) x c x

52 Secant method x + = x c x c x f(x c ) f(x ) f(x c)

53 Secant method x + = x c x c x f(x c ) f(x ) f(x c) Usually, the convergence rate (if it converges) is (1+ 5)/2 1.6 e k+1 ce 1.6, superlinear, between quadratic and linear. k

54 Zeros of a polynomial Finding the zeros of a polynomial p = x n + c n 1 x n c 1 x 1 + c 0 Many methods were proposed.

55 Zeros of a polynomial Finding the zeros of a polynomial p = x n + c n 1 x n c 1 x 1 + c 0 Many methods were proposed. The eigenvalues of its companion matrix c c 1 C(p) = c 2, det(xi C(p)) = p c n 1

56 Example The zeros of the polynomial x 3 1 are the eigenvalues of

57 Example The zeros of the polynomial x 3 1 are the eigenvalues of One real and two complex conjugate eigenvalues.

58 Note How to compute the eigenvalues of a matrix? Finding the zeros of a polynomial used to be the way of finding the eigenvalues of a matrix A.

59 Note How to compute the eigenvalues of a matrix? Finding the zeros of a polynomial used to be the way of finding the eigenvalues of a matrix A. Text book method: The eigenvalues of a matrix A are the zeros of its characteristic polynomial det(λi A).

60 Note Now, we have efficient and reliable methods for computing eigenvalues of a matrix. QR method, John G.F. Francis and Vera N. Kublanovskaya, late 1950s. We find the zeros of a polynomial by computing the eigenvalues of its companion matrix.

61 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

62 Problem setting f 1 (x 1,..., x n ) = 0 f 2 (x 1,..., x n ) = 0. f n (x 1,..., x n ) = 0 Denote f(x) = 0 f: vector-valued function x: vector

63 Newton s method where s c is the solution of x + = x c + s c f(x c )+J(x c )s c = 0, i.e., s c = J 1 (x c )f(x c ), where J(x c ) is the Jacobian of f at x c : [ ] fi J(x) = = x j f 1 x 1. f n x 1 f 1 x n.. f n x n

64 Example A system of nonlinear equations { x 2 1 x 2 2 = 0 2x 1 x 2 = 1, with starting point x 0 = [ 0 1 ] Solution: x 1 = x 2 = 1/ 2

65 Example f(x) = [ f1 f 2 ] = [ x 2 1 x 2 2 2x 1 x 2 1 ] The Jacobian is and [ ] 2x1 2x J(x) = 2 2x 2 2x 1 J(x 0 ) = [ ]

66 Example Step 1: x 1 = x 0 J 1 (x 0 ) f(x 0 )

67 Example Step 1: x 1 = x 0 J 1 (x 0 ) f(x 0 ) Solving for d 0 in J(x 0 )d 0 = f(x 0 ), we have [ ] 0.5 d 0 = 0.5 Thus x 1 = [ 0 1 ] [ ] = [ ]

68 Example Step 2: J(x 1 ) = x 2 = x 1 J 1 (x 1 ) f(x 1 ) [ ] [, f(x 1 ) = ]

69 Example Solving for d 1 in J(x 1 )d 1 = f(x 1 ), we have [ ] 0.25 d 1 = 0.25 Thus x 2 = [ ] [ ] = [ ]

70 Avoiding derivatives The jth column of J(x) f 1 x j. f n x j = f x j can be approximated by the difference f(x 1,..., x j +δ, x j+1,..., x n ) f(x 1,..., x n ) δ

71 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

72 Problem setting min f(x) or max f(x) x S x S x: vector f(x): objective function and real-valued S: support

73 Problem setting min f(x) or max f(x) x S x S x: vector f(x): objective function and real-valued S: support Find a zero of the gradient f(x) = f(x) x 1. f(x) x n

74 Newton s method View the gradient f(x) as a vector-valued function and apply the Newton s method for solving nonlinear systems. At current x c, find the correction s c : x + = x c + s c where s c is the solution of f(x c )+H(x c )S c = 0. The matrix H(x c ) (the Jacobian of the gradient at x c ) is called the Hessian of f at x c ( 2 f(x c )): H i,j = 2 f x i x j.

75 Example Minimizing f : R 2 R: f(x) = x3 1 3 x 1x x 2. Perform one iteration of Newton s method for minimizing f using the starting point [ ] 0 x 0 =. 1

76 Example Apply the Newton s method for finding a zero of the gradient [ x f(x) = 1 2 ] x2 2 2x 1 x The Hessian [ H(x) = 2 f(x) = 2x 1 2x 2 2x 2 2x 1 ]

77 Example Step 1: x 1 = x 0 2 f(x 0 ) 1 f(x 0 ) [ ] [ 0 0 1/2 = 1 1/2 0 [ ] 1/2 = 1/2 ] [ 1 1 ]

78 Outline 1 Introduction 2 Bisection Method 3 Newton s Method 4 Systems of Nonlinear Equations 5 Continuous Optimization 6 Software Packages

79 Software Packages IMSL zporc, zplrc, zpocc MATLAB roots, fzero NAG c02agf, c02aff NAPACK czero Octave fsolve

80 Summary Issues in an iterative method: Initialization, convergence and rate of convergence, termination. The example of computing square root Bisection method: Numerical termination problem Newton s method: Initial value, convergence problems Newton s method for systems of nonlinear equations, Jacobian matrix Newton s method for minimization, gradient and Hessian.

81 References [1 ] George E. Forsyth and Michael A. Malcolm and Cleve B. Moler. Computer Methods for Mathematical Computations. Prentice-Hall, Inc., Ch 7.