CLASS NOTES Computational Methods for Engineering Applications I Spring 2015

Transcription

1 CLASS NOTES Computational Methods for Engineering Applications I Spring 2015 Petros Koumoutsakos Gerardo Tauriello (Last update: July 2, 2015) IMPORTANT DISCLAIMERS 1. REFERENCES: Much of the material (ideas, definitions, concepts, eamples, etc) in these notes is taken (in some cases verbatim) for teaching purposes, from several references, and in particular the references listed below. A large part of the lecture notes is adapted from The Nature of Mathematical Modeling by Neil Gershenfeld (Cambridge University Press, 1st ed., 1998). Therefore, these notes are only informally distributed and intended ONLY as study aid for the final eam of ETHZ students that were registered for the course Computational Methods for Engineering Applications I in SS CONTENT: The present Notes are a first LaTe draft for the Lectures of Computational Methods for Engineering Applications I. The notes have been checked but there is no guarantee that they are free of mistakes. Please use with care. Again, these notes are only intended for use by the ETHZ students that were registered for the course Computational Methods for Engineering Applications I in SS 2015 and ONLY a study aid for their final eam.

2 Chapter 4 Solving Non-Linear Equations 4.1 Key Points After this class, you should understand the following points regarding the solution of non-linear equations: What are the key algorithms for non-linear equation solving When to use derivatives or their approimation in Newton s methods What is the Jacobian for Newton s method and what is its role Why do we approimate the Jacobian in Newton s methods

3 2 Chapter 4. Solving Non-Linear Equations 4.2 Introduction So far we have looked at linear systems of equations which we could write as A = b, where A is an invertible N N matri and and b are vectors of length N. For function fitting, we determined A and b from the data and computed the N unknown parameters in by solving the system A = b or equivalently solving the linear set of equations F ( ) = A b = 0. The task becomes harder if we try to find parameters = ( 1,..., N ) for a general system of N non-linear equations f i ( ) = 0 (i = 1,..., N), which we commonly write as F ( ) = 0. A single non-linear equation has the form = 0 (4.2.1) The task is to find solutions of such equations (i.e. finding the root of ). Looking at the graph of below the goal is to find points, where the curve crosses the horizontal ais. * * Eample: (adapted from book Numerical Analysis by R.L. Burden and J.D. Faires) The amount of pressure required to sink a large heavy object in a soft homogeneous soil that lies above a hard base soil can be predicted by the amount of pressure required to sink smaller objects. Let p denote the amount of pressure required to sink a circular plate of radius r at a distance d in the soft soil, where the hard soil lies at a distance D > d below the surface. p can be approimated by an equation of the form: p = k 1 e k 2r + k 3 r, (4.2.2) where k 1, k 2, k 3 depend on d and the consistency of the soil but not on r. Now the task is to find k 1, k 2, k 3. To do that we need to have 3 equations which we can obtain by taking plates of different radii r 1, r 2, r 3 and sinking them. After doing so we will get the following

4 4.3. Preliminaries 3 system of equations: p 1 = k 1 e k 2r 1 + k 3 r 1, p 2 = k 1 e k 2r 2 + k 3 r 2, (4.2.3) p 3 = k 1 e k 2r 3 + k 3 r 3. Eample with one unknown: Population growth Let N(t) be the number of people at time t. Let λ be the birth rate. Then the equation describing the populaition growth has the form: dn(t) dt = λn(t). (4.2.4) Assume that we also have an immigration at a constant rate u. Then the population growth equation will include an additional term: dn(t) dt = λn(t) + u. (4.2.5) The solution of the Eq. (4.2.5) is given by the following formula: where N 0 is a constant indicating the initial population size. N(t) = N 0 e λt + u λ (eλt 1), (4.2.6) Numerical eample. Suppose we have initially 1,000,000 people. Assume that 15,000 people immigrate every year and that 1,080,000 people are present at the end of the year. What is the birth rate of this population? To find the birth rate we need to solve the following equation for λ (remember that t = 1 [year]): 1, 080, 000 = 1, 000, 000e λ + 15, 000 (e λ 1). (4.2.7) λ As in the previous eample we have a non-linear equation and the question now is how one can solve it. 4.3 Preliminaries Eistence and Uniqueness Recall that for linear problems finding the solution of A = b required that A 1 0 or that A is non-singular. For a function f : R R define the bracket as the interval, where the sign of f differs at the endpoints.

5 4 Chapter 4. Solving Non-Linear Equations If f is continuous and signf(a) signf(b), then the Intermediate Value Theorem implies that there eists [a, b] such that f( ) = 0. Note I : There is no simple analog of this theorem for the multi-dimensional case. Note II : Non-linear equations may have roots for which = 0 and f () = 0 (i.e. they have roots with multiplicity m > 1) as shown in the figures below: =1 =1 =1 =1 Graph of the function = Graph of the function = Sensitivity and Conditioning Conditioning of root finding problem is opposite to that of evaluating function. The condition number for the root of f : R R is 1 f ( (4.3.1) ) The root is ill-conditioned if the tangent line is nearly horizontal. Furthermore, roots with multiplicity m > 1 are ill-conditioned. For systems of N non-linear equations, the condition number of the root finding problem for root of F : R N R N is J 1 ( ), where J is the Jacobian of f. J is a N N matri with elements J( ) i,j = f i ( )/ j. Consider an approimate solution to = 0: f( ) 0 or 0, (4.3.2) where is the true solution. We notice that a small residual f( ) implies an accurate solution (i.e. 0) only if the problem is well-conditioned.

6 4.4. Bisection Method 5 Convergence Rate Many schemes proceed by iteratively improving an estimate (k) with a better estimate (k+1). The error for (k) is given by E (k) = (k) (4.3.3) The sequence of (k) converges with rate r if E (k+1) lim k E (k) r = C (4.3.4) where C is finite and non zero. There are three common cases according to the value of r: r = 1: linear r > 1: superlinear r = 2: quadratic 4.4 Bisection Method The bisection method is based on the intermediate value theorem. The method works by beginning with an initial bracket and then halving the length until a solution has been isolated as accurately as desired. We start with an initial bracket [a, b] and a desired tolerance tol. We then refine [a, b] according to Algorithm 1 (see also Fig. 4.2). Notes: The bisection method makes no use of actual function values, only of their signs.

7 6 Chapter 4. Solving Non-Linear Equations The bisection method is certain to converge, but slowly. At each iteration the length of interval containing the solution is reduced by 1/2. So the method is linearly convergent with r = 1 and C = 0.5. In terms of binary numbers, one bit of accuracy is gained in the approimate solution for each iteration of the bisection method. Given the starting interval [a, b] after k iterations, the interval is (b a)/2 k, so achieving an error tolerance of tol requires m = log 2 ((b a)/tol) steps, regardless of f. Algorithm 1 Bisection Method Steps: while (b a) > tol do m (a + b)/2 if sign(f(a)) = sign(f(m)) then a m else b m end if end while f(b) a P1 P2 P4 P3 b f(p1) f(a) Figure 4.2: Iterative approach used in the bisection method, where P k denotes the m value in Algorithm 1 at the k-th iteration.

8 4.5. Newton s Method 7 f( (0) ) * θ (2) (1) (0) tan( ) = f((0) ) (0) (1) = f 0 ( (0) ) ) (0) (1) = f((0) ) f 0 ( (0) ) ) (1) = (0) f( (0) ) f 0 ( (0) ) Figure 4.3: Graphical eample of Newton iterations as we follow the tangents of. 4.5 Newton s Method In a linear problem the equivalent of the linear problem A b = 0 is = 0 as in Eq. (4.2.1) (or F ( ) = 0 for a system of equations). We epect to solve = 0 by starting from a guess (k) and improving it by computing the net guess (k+1). Everything in a sense depends on what is known of (k). Normally, we can calculate f( (k) ) and see how close it is to 0. Can we estimate f( (k+1) ) from f( (k) ) too? Let s use the first terms of a Taylor series: f( (k+1) ) = f( (k) ) + f ( (k) )( (k+1) (k) ). (4.5.1) Note that we dropped all higher terms from the Taylor series. We can now solve f( (k+1) ) = 0 in Eq. (4.5.1) and get that: f ( (k) )( (k+1) (k) ) = f( (k) ) (4.5.2) (k+1) = (k) f((k) ) f ( (k) ) (4.5.3) Graphically the procedure is visualized in Fig Newton s method approimates the non-linear function f near (k) by the tangent line at f( (k) ). Convergence for simple roots is quadratic. For roots of multiplicity m > 1, the convergence is linear unless we modify the iteration (Eq. (4.5.3)) to (k+1) = (k) mf( (k) )/f ( (k) ) Problems with Newton s Method Newton s method requires evaluation of both function and derivative at each iteration. Evaluating the derivative may be epensive or inconvenient so we replace this with a finite difference approimation.

9 8 Chapter 4. Solving Non-Linear Equations KEY IDEA: If something is epensive or inconvenient to compute, replace it with a numerical approimation. Let f ( (k) ) = f((k) ) f( (k 1) ) (k) (k 1) (4.5.4) then the method reads as (k+1) = (k) f( (k) (k) (k 1) ) f( (k) ) f( (k 1) ) (4.5.5) This is called the secant method. The convergence rate of the method is r Secant Method The secant method approimates the non-linear function f by a secant line through previous two iterations (see Fig. 4.4). As we saw above the secant method uses a linear approimation of the function to construct its derivative. This idea can be etended so that higher order interpolations can be used. Quadratic interpolation (Müller s method) has convergence r Other eamples include inverse interpolation, or linear fractional intepolation. k k-1 k+1 Figure 4.4: Iterative approach used in the secant method.

10 4.5. Newton s Method Solving systems of non-linear equations We are given a system of N non-linear equations f i ( ) = 0 (i = 1,..., N) where = ( 1,..., N ) is a vector of N unknowns: f 1 ( ) 0 f F ( ) = 2 ( ). = 0. = 0 (4.5.6) f N ( ) 0 The picture of Newton s method (see Fig. 4.3) is similar when trying to solve Eq. (4.5.6). We now have N surfaces f i ( ), each with a tangent plane at (k). The surfaces intersect in a curve that crosses the plane at 0 at. The tangent planes, on the other hand, intersect in a tangent line that crosses (k+1). The method succeeds when (k+1) is close to. The algebraic derivations remain however the same or quite similar. The main change is that the derivative f is replaced by the Jacobian matri J, which is a N N matri with elements J( ) i,j = f i ( )/ j : f 1 ( ) 1 f 2 ( ) 1 J( ) =. f N ( ) 1 The modified Taylor series (see Eq. (4.5.1)) now becomes and the update rule (see Eq. (4.5.3)) becomes f 1 f 1 ( ) ( ) 2 N f 2 f 2 ( ) ( ) 2 N..... f N f N ( ) ( ) 2 N (4.5.7) F ( (k+1) ) = F ( (k) ) + J( (k) )( (k+1) (k) ) (4.5.8) In practice, we never invert J( (k) ) and instead solve for y k and update as (k+1) = (k) + y k. (k+1) = (k) J 1 ( (k) ) F ( (k) ) (4.5.9) J( (k) ) y k = F ( (k) ) (4.5.10) The convergence of Newton s method for systems of equations is usually quadratic, provided that the Jacobian matri is non-singular. We note that for a linear system, we have F ( ) = A b and J( ) = A. A single iteration of Newton s method then computes (1) = (0) J 1 ( (0) ) F ( (0) ) = (0) A 1 A (0) + A 1 b = A 1 b, (4.5.11)

11 10 Chapter 4. Solving Non-Linear Equations which gives the same solution as when simply solving the system A = b. To sum up: we start with an initial approimation (0) and iteratively improve the approimation by computing new approimations (k) as in Algorithm 2. Algorithm 2 Newton s method Input: (0), {vector of length N with initial approimation} tol, {tolerance: stop if (k) (k 1) < tol} k ma, {maimal number of iterations: stop if k > k ma } Output: (k), {solution of F ( (k) ) = 0 within tolerance tol} (or a message if k > k ma reached) Steps: k 1 while k k ma do Calculate F ( (k 1) ) and N N matri J( (k 1) ) Solve the N N linear system J( (k 1) ) y = F ( (k 1) ) (k) (k 1) + y if y < tol then break end if k k + 1 end while Computational Cost For large N and dense Jacobians J, the cost of Newton s method can be substantial. It costs O(N 2 ) to build the matri J and solving the linear system in Eq. (4.5.10) costs O(N 3 ) operations (if J is a dense matri). Secant Updating Method Reduce the cost of Newton s Method by: using function values at successive iterations to build approimate Jacobians and avoid eplicit evaluations of the derivatives (note that this is also necessary if for some reason you cannot evaluate the derivatives analytically) update factorization (to solve the linear system) of approimate Jacobians rather than refactoring it each iteration

12 4.5. Newton s Method 11 Most secant methods are superlinear. Modified Newton Method: The simplest approach is to keep J fied during the iterations of Newton s method. So instead of updating or recomputing J, we compute J (0) = J( (0) ) once and (instead of Eq. (4.5.10)) solve J (0) y k = F ( (k) ). (4.5.12) This can be done efficiently by performing a LU decomposition of J (0) once and use it over and over again for different F ( (k) ). This removes the O(N 3 ) cost of solving the linear system. We only have to evaluate F ( (k) ) and can then compute y k in O(N 2 ) operations. This method can only succeed if J is not changing rapidly. Quasi Newton Method: A method in between Newton and modified Newton is the quasi Newton method which changes J at every step without recomputing the derivatives f i ( )/ j (see Eq. (4.5.7)). It instead updates J (0) = J( (0) ) by using information from the step itself. After the first step we know: = (1) (0) and F = F ( (1) ) F ( (0) ). So derivatives of the f i are in the direction of. Then the net J (1) is adjusted to satisfy: for eample by the rank-1 update J (1) = J (0) + J (1) = F (4.5.13) ( F ) J (0) ( ) T ( ) T ( ). (4.5.14) The advantage of the rank-1 update is that it allows the LU decomposition of J (1) to be computed in O(N 2 ) given the LU decomposition of J (0). As in the modified Newton s method, this essentially reduces the cost of solving the linear system from O(N 3 ) to O(N 2 ).