Chapter 6. Nonlinear Equations. 6.1 The Problem of Nonlinear Root-finding. 6.2 Rate of Convergence

Transcription

1 Chapter 6 Nonlinear Equations 6. The Problem of Nonlinear Root-finding In this module we consider the problem of using numerical techniques to find the roots of nonlinear equations, f () =. Initially we eamine the case where the nonlinear equations are a scalar function of a single independent variable,. Later, we shall consider the more difficult problem where we have a system of n nonlinear equations in n independent variables, f() =. Since the roots of a nonlinear equation, f () =, cannot in general be epressed in closed form, we must use approimate methods to solve the problem. Usually, iterative techniques are used to start from an initial approimation to a root, and produce a sequence of approimations () ; () ; () ;::: ; which converge toward a root. Convergence to a root is usually possible provided that the function f () is sufficiently smooth and the initial approimation is close enough to the root. 6. Rate of Convergence Let () ; () ; () ;:::

2 7 NONLINEAR EQUATIONS be a sequence which converges to a root, andlet (k) = (k),. If there eists a number p and a non-zero constant c such that (k+) lim k! (k) p = c; (6.) then p is called the order of convergence of the sequence. For p = ; ; 3 the order is said to be linear, quadratic and cubic respectively. 6.3 Termination Criteria Since numerical computations are performed using floating point arithmetic, there will always be a finite precision to which a function can be evaluated. Let denote the limiting accuracy of the nonlinear function near a root. This limiting accuracy of the function imposes a limit on the accuracy,, of the root. The accuracy to which the root may be found is given by: = f () : (6.) This is the best error bound for any root finding method. Note that is large when the first derivative of the function at the root, f (), is small. In this case the problem of finding the root is illconditioned. This is shown graphically in Figure 6.. f () f () f ( ) large ill-conditioned FIGURE 6.: small well-conditioned When the sequence () ; () ; () ;::: ; converges to the root,, then the differences (k), (k,) will decrease until (k),.

3 6.4 BISECTION METHOD 7 With further iterations, rounding errors will dominate and the differences will vary irregularly. The iterations should be terminated and (k) be accepted as the estimate for the root when the following two conditions are satisfied simultaneously:. (k+), (k) (k), (k,) and (6.3) (k), (k,). < : (6.4) + (k) is some coarse tolerance to prevent the iterations from being terminated before (k) is close to, i.e. before the step size (k), (k,) becomes small. The condition (6.4) is able to test the relative offset when (k) is much larger than andmuchsmallerthan. In practice, these conditions are used in conunction with the condition that the number of iterations not eceed some user defined limit. 6.4 Bisection Method Suppose that f () is continuous on an interval, () ; (). A root is bracketed on the interval, () ; () if f, () and f, () have opposite sign. Let () = () + (), () (6.5) be the mid-point of the interval, () ; (). Three mutually eclusive possibilities eist: if, f () =then the root has been found; if, f () has the same sign as, f () then the root is in the interval, () ; () ; if, f () has the same sign as, f () then the root is in the interval, () ; (). In the last two cases, the size of the interval bracketing the root has decreased by a factor of two. The net iteration is performed by evaluating the function at the mid-point of the new interval. After k iterations the size of the interval bracketing the root has decreased to: (), () k

4 7 NONLINEAR EQUATIONS The process is shown graphically in Figure 6.. The bisection method is guaranteed to converge to a root. If the initial interval brackets more than one root, then the bisection method will find one of them. f () () () () (3) (5) (4) FIGURE 6.: Since the interval size is reduced by a factor of two at each iteration, it is simple to calculate in advance the number of iterations, k, required to achieve a given tolerance,, in the solution: (), () k = log : The bisection method has a relatively slow linear convergence. 6.5 Secant Method The bisection method uses no information about the function values, f (), apart from whether they are positive or negative at certain values of. Suppose that f, (k) < f, (k,) ; Then we would epect the root to lie closer to (k) than (k,). Instead of choosing the new estimate to lie at the midpoint of the current interval, as is the case with the bisection method, the secant method chooses the -intercept of the secant line to the curve, the line through,, (k,) ;f (k,) and,, (k) ;f (k). This places the new estimate closer to the endpoint for which f () has the smallest absolute value. The new estimate is: (k+) = (k), f, (k), (k), (k,) f ( (k) ), f ( (k,) ) : (6.6)

5 6.6 REGULA FALSI 73 Note that the secant method requires two initial function evaluations but only one new function evaluation is made at each iteration. The secant method is shown graphically in Figure 6.3. f () () () (3) () FIGURE 6.3: The secant method does not have the root bracketing property of the bisection method since the new estimate, (k+), of the root need not lie within the bounds defined by (k,) and (k).as a consequence, the secant method does not always converge, but when it does so it usually does so faster than the bisection method. It can be shown that the order of convergence of the secant method is: + p 5 :68: 6.6 Regula Falsi Regula falsi is a variant of the secant method. The difference between the secant method and regula falsi lies in the choice of points used to form the secant. While the secant method uses the two most recent function evaluations to form the secant line through,, (k,) ;f (k,) and,, (k) ;f (k), regula falsi forms the secant line through the most recent points that bracket the root. The regula falsi steps are shown graphically in Figure 6.4. In this eample, the point () remains active for many steps. The advantage of regula falsi is that like the bisection method, it is always convergent. However, like the bisection method, it has only linear convergence. Eamples where the regula falsi method is slow to converge are not hard to find. One eample is shown in Figure 6.5.

6 74 NONLINEAR EQUATIONS f () () () (3) () FIGURE 6.4: f () (3) () () () FIGURE 6.5:

7 6.7 NEWTON S METHOD Newton s Method The methods discussed so far have required only function values to compute the new estimate of the root. Newton s method requires that both the function value and the first derivative be evaluated. Geometrically, Newton s method proceeds by etending the tangent line to the function at the current point until it crosses the -ais. The new estimate of the root is taken as the abscissa of the zero crossing. Newton s method is defined by the following iteration:, (k+) = (k), f (k) f ( (k) ) : (6.7) The update is shown graphically in Figure 6.6. f () () () () FIGURE 6.6: Newton s method may be derived from the Taylor series epansion of the function: f ( +) =f () +f () + f () () + ::: (6.8) For a smooth function and small values of, the function is approimated well by the first two terms. Thus f ( +) =implies that =, f() f. Far from a root the higher order terms are () significant and Newton s method can give highly inaccurate corrections. In such cases the Newton iterations may never converge to a root. In order to achieve convergence the starting values must be reasonably close to a root. An eample of divergence using Newton s method is given in Figure 6.7. Newton s method ehibits quadratic convergence. Thus, near a root the number of significant digits doubles with each iteration. The strong convergence makes Newton s method attractive in cases where the derivatives can be evaluated efficiently and the derivative is continuous and nonzero in the neighbourhood of the root (as is the case with multiple roots). Whether the secant method should be used in preference to Newton s method depends upon the relative work required to compute the first derivative of the function. If the work required to

8 76 NONLINEAR EQUATIONS f () () () FIGURE 6.7: evaluate the first derivative is greater than :44 times the work required to evaluate the function, then use the secant method, otherwise use Newton s method. It is easy to circumvent the poor global convergence properties of Newton s method by combining it with the bisection method. This hybrid method uses a bisection step whenever the Newton method takes the solution outside the bisection bracket. Global convergence is thus assured while retaining quadratic convergence near the root. Line searches of the Newton step from (k) to (k+) are another method for achieving better global convergence properties of Newton s method. 6.8 Eamples In the following eamples, the bisection, secant, regula falsi and Newton s methods are applied to find the root of the non-linear function f () =, between [; 3]. L and R are the left and right bracketing values and new is the new value determined at each iteration. is the distance from the true solution, n f is the number of function evaluations required at each iteration. The bisection method is terminated when conditions (6.3) and (6.4) are simultaneously satisfied for =,4. The remaining methods determine the solution to the same level of accuracy as the bisection method. The eamples show the linear convergence rate of the bisection and regula falsi methods, the better than linear convergence rate of the secant method and the quadratic convergence rate of Newton s method. It is also seen that although Newton s method converges in 5 iterations, 5 function and 5 derivative evaluations, to give a total of evaluations, are required. This contrasts to the secant method where for the 8 iterations, 9 function evaluations are made.

9 6.8 EXAMPLES 77 Bisection Method - f() = - k L R new f L () f R () f( new ) ε n f k+ - k k - k- k - k- /+ k Secant Method - f() = - k f( ) f( ) ε n f k - k- /+ k f() = - f() E Regula Falsi Method - f() = - k f( ) f( ) new f( new ) ε n f k - k- /+ k Newton s Method - f() = -; f () = k f() f () ε n f n f k - k- /+ k

10 78 NONLINEAR EQUATIONS 6.9 Laguerre s Method In the previous sections we have investigated methods for finding the roots of general functions. These techniques may sometimes work for finding roots of polynomials, but often problems arise because of multiple roots or comple roots. Better methods are available for nonlinear functions with these types of roots. One such technique is Laguerre s method. Consider the n th order polynomial: P n () =(, )(, ) :::(, n ): (6.9) Taking the logarithm of both sides gives: Let: and ln P n () = ln, + ln, + :::+ ln, n : (6.) G = d ln P n() d (6.) =, +, + :::+, n (6.) = P n () P n () (6.3) H =, d ln P n () (6.4) d = (, ) + (, ) + :::+ P n = (), P () n P n () (, n ) (6.5) P n () : (6.6) If we make the assumption that the root is located a distance from our current estimate, (k), and all other roots are located at a distance, then: (k), = (6.7) (k), i = for i =; 3;::: ;n: (6.8)

11 6.9 LAGUERRE S METHOD 79 Equations (6.3) and (6.6) can now be epressed as: Solving for gives: G = + n, (6.9) H = + n, : (6.) = n G p (n, ) (nh, G ) (6.) where the sign should be taken to yield the largest magnitude for the denominator. The new estimate of the root is obtained from the old using the update: (k+) = (k), : (6.) In general, will be comple., Laguerre s method requires that, P n (k),, P n (k) and P n (k) be computed at each step. The method is cubic in its convergence for real or comple simple roots. Laguerre s method will converge to all types of roots of polynomials, real, comple, single or multiple. It requires comple arithmetic even when converging to a real root. However, for polynomials with all real roots, it is guaranteed to converge to a root from any starting point. Afteraroot,,ofthen th order polynomial, P n (), has been found, the polynomial should be divided by the quotient, (, ), to yield an (n, ) order polynomial. This process is called deflation. It saves computation when estimating further roots and ensures that subsequent iterations do not converge to roots already found Eample Following is a simple eample that illustrates the use of Laguerre s method. Consider the third order polynomial: P () =, 4 +3 P () =, 4 P () =

12 8 NONLINEAR EQUATIONS. Let the initial estimate of the first root be: () =:5 then G = () () H = G, =, 4, 4 () +3 () =,:4, 4 () +3 =4:6 G p =,:5 (choosing,4 as the largest denominator) (, )(H, G ) The first root is then: () = (), =:5, (,:5) = :. The second root is found by dividing P by (, ). The trivial result is that = Horner s method for evaluating a polynomial and its derivatives The polynomial P n () = a + a + a + :::+ a n n should not be calculated by evaluating the powers of individually. It is much more efficient to calculate the polynomial using Horner s method by noting that P n () = a + (a + (a + :::+ a n )). Using this approach the polynomial value and the first and second derivatives may be evaluated efficiently using the following algorithm: p a n ; p :; p :; for i in n, ::: loop end loop; p p + p ; p p + p; p p + a i ;

13 6.9 LAGUERRE S METHOD 8 p p ; If the polynomial and its derivatives are evaluated using the individual powers of, (n +3n)= operations are required for the value, (n + n, )= for the first derivative and (n, n, )= for the second derivative, i.e. a total of (3n +3n, 4)= operations. Horner s method requires 6n + operations to evaluate the polynomial and both derivatives and thus requires less operations when n>3. If ust the function evaluation is required, then Horner s method will require less operations for all n Eample Following is an eample of how Horner s method can be used to evaluate P 3 () =+, and its derivatives.. P =, P =and P =.. P =+, P =+and P =, P =+, P = +, 3 and P = (, 3)+. 4. P = +, 3, P = (, 3) + (, 3)+and P = ((, 3)+)+. 5. P = (3, 3) Deflation Dividing a polynomial of order n by a factor (, ) may be performed using the following algorithm: r a n ; a n :; for i in n, ::: loop q a i ; a i r; r r + q;

14 8 NONLINEAR EQUATIONS end loop; The coefficients of the new polynomial are stored in the array a i, and the remainder in r. 6. Systems of Nonlinear Equations Consider a general system of n nonlinear equations in n unknowns: f i ( ; ;::: ; n )=;i=; ;::: ;n: Newton s method can be generalised to n dimensions by eamining Taylor s series in n dimensions: where f i, f (k+) i = f i, (k) + f i, (k) (k+), (k) + :::, (k) is an n n matri called the Jacobian, having elements: f i () : (6.3) If these gradients are approimately calculated, the method is called a quasi-newton method. One way to approimately form the gradients is to calculate f i (, ) and f i ( + ) and then form the finite difference approimation: f i () f i( + ), f i (, ) (6.4) The vector is a vector of zeros, ecept for a small perturbation in the th position. Setting f i, (k+) =gives Newton s formula in n dimensions: f i, (k) (k+),, (k) =,f (k) i (6.5) or, where (k+) = (k+), (k) f i ((k) ) (k+) =,f i, (k) (6.6) is the vector of updates. Equation (6.6) is a set of n linear equations in n unknowns, (k+). If the Jacobian is non-singular, then the system of linear equations

15 6. SYSTEMS OF NONLINEAR EQUATIONS 83 can be solved to provide the Newton update: (k+) = (k) + (k+) : (6.7) Each step of Newton s method requires the solution of a set of linear equations. For small n the set of linear equations may be solved using LU decomposition. For large n, alternative iterative methods may be required. As with the one-dimensional version, Newton s method converges quadratically if the initial estimate, (), is sufficiently close to a root. Newton s method in multiple dimensions suffers from the same global convergence problems as its one-dimensional counterpart. 6.. Eample Consider finding and y such that the following are true: f (; y) =, + y, y, y += (6.8) f (; y) = + + y +y +y += (6.9) This is a contrived problem, but it serves to illustrate the application of Newton s in multiple dimensions. Solving for the simultaneous roots of these equations is equivalent to finding the roots of the single factored equation: f (; y) =( + y +), (, y, ) (6.3) Inspection shows that this equation has the root (;,). The first step in applying Newton s method is to determine the form of the Jacobian and the right hand side function vector for calculating the update vector for each iteration. These are: ( (k) ;y (k) ( (k) ;y (k) ) " # " 7 (k+),f ( (k) ;y (k) ) 4@f ( (k) ;y (k) ( (k) ;y (k) ) 5 (k+) = (6.3) y,f ( (k) ;y " #" # " # (, y, ) (y,, ) (k+), +, y +y +y, = (6.3) ( + y +) ( + y +),,, y, y, y, (k+) y

16 84 NONLINEAR EQUATIONS The algorithm for solving this problem is: k ; >; specify starting point: ( (k) ;y (k) ); while: >and k<n (k+),f, f ;(LU factorisation and solution) (k+) (k) + (k+) ; end loop; (k+), (k) k k +; ; Where N is some specified maimum number of iterations and is some specified convergence tolerance. Sequences of Newton steps converging toward the root (;,) for the starting points (,;,3), (;,3) and (; ) are shown in figures (6.8) and (6.9). f(,y) y FIGURE 6.8: Sequence of Newton steps plotted on the surface defined by Equation (6.3).

17 6. SYSTEMS OF NONLINEAR EQUATIONS 85 5 start at (, 3) f(,y) 5 start at (,) 5 start at (, 3) Number of iterations FIGURE 6.9: Convergent behaviour of Equation (6.3) with Newton iteration. 6.. Eample Newton s method for finding roots in multiple dimensions becomes very useful when finding approimate solutions to non-linear partial differential equations. Consider the diffusion equation in one-dimension, where the diffusivity, D, is an eponential function of the @ Aec = This is a non-linear partial differential equation. A finite difference discretisation of this equation for the i th node is: f i = cn+ i, c n i t n, Ae(,)c i +cn+ i,, (, ) c n i,, c n + i cn i+ + c n+ i,, c n+ i + c n+ i+ (6.34) Notice that this is a non-linear equation whose roots are the unknown concentrations c n+, i, cn+ i and c n+ i+. If there are m discrete nodes in our problem, then there are m equations and m unknown concentrations, so Newton s method for multiple dimensions can be applied.

18 86 NONLINEAR EQUATIONS A row in the Jacobian for the internal node i will n+ i or n+ i, n, Ae(,)c i +cn+ n+ i+, Ae (,)c n i +cn+ i t (, ), c n, c n+, i, cn+ i + c n+ i+, i,, c n i + cn i+ + n, Ae(,)c i +cn+ i i (6.35)! (6.36) The corresponding right hand side vector entry for the Newton update is,f i. Note that in contrast to the case of the linear diffusion equation, the system of discrete finite difference equations are now time varying and must be constructed and factorised for each time step. If Dirichlet boundary conditions of and are applied at nodes and n, respectively, then the functions whose roots are to be found are: f = c n+, (6.37) f n = c n+ n (6.38) The corresponding rows in the Jacobian will be: h i h i (6.39) (6.4) The right hand side vector entries for the Newton update are, c n+ and,c n+ n, respectively. Note that the unknown dependent variables for this non-linear problem are c n+. Let these unknowns be referred to simply as c so that the k th Newton update becomes c (k). The algorithm for solving this problem is similar to that of Eample :

19 6. SYSTEMS OF NONLINEAR EQUATIONS 87 for t in T loop k ; >; specify starting point: c (k) ; (This will be the solution from the previous time step) while: >and k<n (k+),f, f ; (LU factorisation and solution) end loop; c (k+) c (k) + (k+) ; (k+), (k) k k +; end loop; ; Results for a problem of length m with = m, t = s and = are shown in Figure 6.. The non-linear solutions were calculated using: D(c) =:368e c (6.4) The coefficient was chosen such that D = m s, at c = kgm,. The linear solutions were calculated using D =:63 m s,. This value was chosen as the integrated mean value of D(c) from c = kg m, to c = kg m,. The variation of the diffusion coefficient with concentration is shown in Figure 6.. The non-linear diffusion coefficient is smaller than the constant diffusion coefficient at lower concentrations. This is observed in the solutions, as the concentration profile for the non-linear diffusion does not advance as far as for constant diffusion over the same interval of time.

20 88 NONLINEAR EQUATIONS L 5s NL 5s L s NL s L 5s NL 5s Concentration Concentration (m).5 Linear Non linear Time (s) (a) Profiles (b) Time history at =5m FIGURE 6.: Comparative solutions for constant diffusion and non-linear diffusion.. D=.368ep(c) D=.63.9 Diffusion Coefficient Concentration FIGURE 6.: Non-linear and constant diffusion functions. Solving this eample problem produced the following non-linear iteration information over the course of the first time step: Time: second Iteration: LNorm(F) =.E+ Delta=.6E+ Iteration: LNorm(F) =.5549E- Delta=.9876E+ Iteration: 3 LNorm(F) =.8958E-3 Delta=.3755E- Iteration: 4 LNorm(F) =.78E-6 Delta=.5565E-3 Iteration: 5 LNorm(F) =.48E-3 Delta=.438E-6

21 6. SYSTEMS OF NONLINEAR EQUATIONS 89 Note that F is the right hand side vector, so its magnitude (or L norm) may be thought of as an indication of how good the solution at that iteration is. The solution is accurate when F is very small. Delta is a measure of how much the solution is changing from one iteration to the net. The Newton iterations are clearly converging.