Spring 2012 Introduction to numerical analysis Class notes. Laurent Demanet

Transcription

1 Spring 2012 Introduction to numerical analysis Class notes Laurent Demanet Draft February 28, 2012

2 2

3 Preface Good additional references are Burden and Faires, Introduction to numerical analysis Suli and Mayers, Introduction to numerical analysis Trefethen, Spectral methods in Matlab as well as the pdf notes on numerical analysis by John Neu. If the text points you to some external reference for further study, the latter is obviously not part of the material for the class. We will very likely skip several sections in chapters 7 and 8 for Work in progress! 3

4 4

5 Chapter 1 Series and sequences Throughout these notes we ll keep running into Taylor series and Fourier series. It s important to understand what is meant by convergence of series before getting to numerical analysis proper. These notes are sef-contained, but two good extra references for this chapter are Tao, Analysis I; and Dahlquist and Bjorck, Numerical methods. A sequence is a possibly infinite collection of numbers lined up in some order: a 1, a 2, a 3,... A series is a possibly infinite sum: a 1 + a 2 + a We ll consider real numbers for the time being, but the generalization to complex numbers is a nice exercise which mostly consists in replacing each occurrence of an absolute value by a modulus. The questions we address in this chapter are: What is the meaning of an infinite sum? Is this meaning ever ambiguous? How can we show convergence vs. divergence? When can we use the usual rules for finite sums in the infinite case? 5

6 6 CHAPTER 1. SERIES AND SEQUENCES 1.1 Convergence vs. divergence We view infinite sums as limits of partial sums. Since partial sums are sequences, let us first review convergence of sequences. Definition 1. A sequence (a j ) j=0 is said to be ɛ-close to a number b if there exists a number N 0 (it can be very large), such that for all n N, a j b ɛ. A sequence (a j ) j=0 is said to converge to b if it is ɛ-close to b for all ɛ > 0 (however small). We then write a j b, or lim j a j = b. If a sequence does not converge we say that it diverges. Unbounded sequences, i.e., sequences that contain arbitrarily large numbers, always diverge. (So we never say converges to infinity, although it s fine to say diverges to infinity.) Examples: e n 0 as n, and convergence is very fast. n/(n + 2) 1 as n, and convergence is rather slow. ( 1) n is bounded, but does not converge. log(n) as n, so the sequence diverges. For a proof that log(n) takes on arbitrarily large values, fix any large integer m. Does there exist an n such that log(n) m? Yes, it suffices to take n e m. Definition 2. Consider a sequence (a j ) j=0. We define the N-th partial sum S N as N S N = a 0 + a a N = a j. We say that the series j a j converges if the sequence of partial sums S N converges to some number b as N. We then write j=0 a j = b. Again, if a series does not converge we say that it diverges. j=0

7 1.1. CONVERGENCE VS. DIVERGENCE 7 Example 1. Consider j=0 2 j, i.e., This series converges to the limit 2. To prove this, consider the partial sum N S N = 2 j. j=0 Let us show by induction that S N = 2 2 N. The base case N = 0 is true since 2 0 = For the induction case, assume S N = 2 2 N. We then write S N+1 = S N + 2 (N+1) = (2 2 N ) + 2 (N+1) = 2 2 (N+1), the desired conclusion. Example 2. The previous example was the x = 1/2 special case of the socalled geometric series 1 + x + x 2 + x WIth a similar argument, we obtain the limit as x j = 1 1 x, j=0 provided the condition x < 1 holds. This expression can also be seen as the Taylor expansion of 1/(1 x), centered at zero, and with radius of convergence 1. Example 3. Consider the so-called harmonic series This series diverges. To see this, let us show that the N partial sum is comparable to log(n). We use the integral test S N = N j=1 1 j N x dx. (Insert picture here) The latter integral is log(n +1), which diverges as a sequence. The partial sums, which are larger, must therefore also diverge.

8 8 CHAPTER 1. SERIES AND SEQUENCES Example 4. Consider j=1 1 n q, for some q > 0. As a function of q, this is the Riemann zeta function ζ(q). (A fascinating object for number theorists.) We ve seen above that q = 1 leads to divergence. A similar integral test would show that the series converges when q > 1, while it diverges when q 1. We now switch to a finer understanding of convergence: certain series are absolutely convergent, while others are conditionally convergent. This will affect what type of algebraic manipulations can be done on them. Definition 3. A series j=0 a j is said to be absolutely convergent if j=0 a j converges. If a series is not absolutely convergent, but nevertheless converges, we say that it is conditionally convergent. The subtlety with conditional convergence is that alternating plus and minus signs may lead to convergence because of cancelations when summing consecutive terms. Example 5. Consider This series is not absolutely convergent, because it reduces to the harmonic series when absolute values of the terms are taken. It is nevertheless convergent, hence conditionally convergent, as the following argument shows. Assume N is even, and let S N = N ( 1) j. j j=1 Terms can be grouped 2-by-2 to obtain, for j 1, 1 j 1 j + 1 = 1 j(j + 1).

9 1.1. CONVERGENCE VS. DIVERGENCE 9 A fortiori, 1 j(j+1) 1 j 2, so S N N 1 j=1,3,5,... which we know converges. If on the other hand N is odd, then S N = S N N+1. Both terms S N and 1/(N + 1) are converging sequences, so their sum converges as well. This proves convergence. Note that the series converges to 1 j 2, = log(2). This is the special case x = 1 in the Taylor expansion of log(1 + x) about x = 0. In passing, without proof, here is a useful test to check convergence of alternating series. Theorem 1. (Alternating series test) Consider the series ( 1) j a j, j=0 where a j > 0. If (a j ) converges to zero (as a sequence), then the series is convergent. The main problem with conditionally convergent series is that if the terms are rearranged, then the series may converge to a different limit. The safe zone for handling infinite sums as if they were finite is when convergence is absolute. Theorem 2. Let f : Z + Z + be a bijection, i.e., f is a rearrangement of the nonnegative integers. Consider a series j=0 a j. If this series is absolutely convergent, then a j = a f(j). j=0 Here is what usually happens when the assumption of absolute convergence is not satisfied. j=0

10 10 CHAPTER 1. SERIES AND SEQUENCES Example 6. Consider again which as we have seen equals log(2) (1 1/2) = log(2) 1/2 = We can rearrange the terms of the series by assigning two negative terms for each positive term: This series is also convergent, but happens to converge to (log(2) 1)/2 = Other operations that can be safely performed on absolutely convergent series are passing absolute values inside the sum, and exchanging sums. Again, complications arise if the series is only conditionally convergent. (See Tao, Analysis I, for counter-examples.) Theorem 3. The following operations are legitimate for absolutely convergent series. Passing absolute values inside sums: a j j=0 a j. j=0 Swapping sums: a j,k = j=0 k=0 k=0 j=0 a j,k Note in passing that the same is true for integrals of unbounded integrands or integrals over unbounded domains: they need to be absolutely convergent (integrability of the absolute value of the function) for the integral swap to be legitimate. This is the content of Fubini s theorem. Again, there are striking counter-examples when the integrals are not absolutely convergent and the swap doesn t work (See Tao, Analysis I).

11 1.2. THE BIG-O NOTATION The big-o notation Here are a few useful pieces of notation for comparing growth or decay of sequences, used extensively by numerical analysts. They are called the big- O, little-o, and big-theta notations. Big-O is used much more often than the other two. They occur when comparing decay rates of truncation errors, and runtimes of algorithms. Definition 4. Consider two nonzero sequences f n and g n for n = 0, 1, 2,.... We write f n = O(g n ) when there exists C > 0 such that f n C g n. We write f n = o(g n ) when f n /g n 0 as n. We write f n = Θ(g n ) when f n = O(g n ) and g n = O(f n ). Examples: f n = n 2 and g n = n 3 : we have n 2 = O(n 3 ) and n 2 = o(n 3 ) but n 2 Θ(n 3 ). f n = n and g n+2 n = n : we have f n 3 n = O(g n ) and f n = Θ(g n ), but f n o(g n ). Exponentials always dominate polynomials: n a = o(e bn ) whatever a > 0 and b > 0. Conversely, e bn = o(n a ). Out loud, we can read the expression f n = O(g n ) as f n in on the order of g n. The same notations can be used to compare sequences indexed by a parameter that goes to zero, such as (typically) the grid spacing h. The definition above is simply adapted by letting h 0 rather than n. Examples: f(h) = h 2 and g(h) = h 3 : this time we have g(h) = O(f(h)) and g(h) = o(f(h)) when the limit of interest is h 0. Powers of h don t converge to zero nearly as fast as this exponential: e a/h = o(h b ) whatever a > 0 and b > 0. SImilarly, we may wish to compare functions f and g of a continuous variable x as either x or x 0; the definition is again modified in the obvious way. Whenever a O( ) or o( ) is written, some underlying limit is understood.

12 12 CHAPTER 1. SERIES AND SEQUENCES

13 Chapter 2 Integrals as sums and derivatives as differences We now switch to the simplest methods for integrating or differentiating a function from its function samples. A careful study of Taylor expansions reveals how accurate the constructions are. 2.1 Numerical integration Consider a function f we ll specify which assumptions we need to make about it in a minute. Let us reformulate the integral 1 0 f(x) dx by breaking up the interval [a, b] into subintervals [x j 1, x j ], with x j = jh, h = 1/N, and 0 = x 0 < x 1 <... < x N = 1. Together, the (x j ) are called a Cartesian grid. Still without approximation, we can write 1 N xj f(x) dx = f(x) dx. 0 x j 1 j=1 A quadrature is obtained when integrals are approximated by quantities that depend only on the samples f(x j ) of f on the grid x j. For instance, we can 13

14 14CHAPTER 2. INTEGRALS AS SUMS AND DERIVATIVES AS DIFFERENCES approximate the integral over [x j 1, x j ] by the signed area of the rectangle of height f(x j 1 ) and width h: xj x j 1 f(x) dx hf(x j 1 ). Putting the terms back together, we obtain the rectangle method: 1 0 f(x) dx h N f(x j 1 ). (Insert picture here) To understand the accuracy of this approximation, we need a detour through Taylor expansions. Given a very smooth function f(x), it sometimes makes sense to write an expansion around the point x = a, as f(x) = f(a) + f (a)(x a) f (a)(x a) ! f (a)(x a) j=1 As a compact series, this would be written as f(x) = n=0 1 n! f (n) (a)(x a) n, where n! = n (n 1) is the factorial of n. The word sometimes is important: the formula above requires the function to be infinitely differentiable, and even still, we would need to make sure that the series converge (by means of the analysis of the previous chapter). The radius of convergence of the Taylor expansion is the largest R such that, for all x such that x a < R, the series converges. It is also possible to truncate a Taylor series after N terms, as f(x) = N 1 n=0 1 n! f (n) (a)(x a) n + 1 N! f (N) (y)(x a) N, (2.1) where y is some number between x and a. The formula does not specify what the number y is; only that there exists one such that the remainder takes the form of the last term. In spite of the fact that y is unspecified, this is a much more useful form of Taylor expansion than the infinite series:

15 2.1. NUMERICAL INTEGRATION 15 We often only care about the size of the remainder, and write inequalities such as 1 N! f (N) (y)(x a) N 1 ( ) max N! f (N) (z) x a N, z [x,a] or, for short, 1 N! f (N) (y)(x a) N = O( x a N ), where all the constants are hidden in the O. So, the remainder in on the order of x a N. The limit implicit in the O notation is here x a 0. The finite-n formula is valid whenever the function is N times differentiable, not infinitely differentiable! In fact, the Taylor series itself may diverge, but equation (2.1) is still valid. In order to study the problem of approximating x j x j 1 f(x) dx, let us expand f in a (short, truncated) Taylor series near x = x j 1 : f(x) = f(x j 1 ) + f (y(x))h, where y(x) [x j 1, x]. We ve written y(x) to highlight that it depends on x. This works as long as f has one derivative. Integrating on both sides, and recognizing that f(x j 1 ) is constant with respect to x, we get (recall x j x j 1 = h) xj x j 1 f(x) dx = hf(x j 1 ) + h xj x j 1 f (y(x)) dx. We don t know much about y(x), but we can certainly write the inequality xj h f (y(x)) dx h x j 1 max f (y) y [x j 1,x j ] So, as long as f is differentiable, we have xj xj x j 1 1 dx = h 2 x j 1 f(x) dx = hf(x j 1 ) + O(h 2 ), max f (y). y [x j 1,x j ]

16 16CHAPTER 2. INTEGRALS AS SUMS AND DERIVATIVES AS DIFFERENCES where the derivative of f hides in the O sign. The integral over [0, 1] has N such terms, so when they are added up the error compounds to N O(h 2 ) = O(h) (because h = 1/N). Thus we have just proved that 1 0 f(x) dx h N f(x j 1 ) + O(h). j=1 The error is on the order of the grid spacing h itself, so we say the method is first-order accurate (because the exponent of h is one.) Choosing the (signed) height of each rectangle from the left endpoint x j 1 does not sound like the most accurate thing to do. Evaluating the function instead at x j 1/2 = (j 1/2)h is a better idea, called the midpoint method. It is possible to show that 1 0 f(x) dx h N f(x j 1/2 ) + O(h 2 ), j=1 provided the function f is twice differentiable (because f hides in the O sign). The accuracy is improved, since h 2 gets much smaller than h as h 0. We say the midpoint method is second-order accurate. Another solution to getting order-2 accuracy is to consider trapezoids instead of rectangles for the interval [x j 1, x j ]. The area of the trapezoid spanned by the 4 points (x j 1, 0); (x j 1, f(x j 1 ); (x j, f(x j ); (x j, 0) is h(f(x j 1 )+ f(x j ))/2. This gives rise to the so-called trapezoidal method, or trapezoidal rule. (Insert picture here) Let us compare this quantity to the integral of f over the same interval. Consider the truncated Taylor expansions f(x) = f(x j 1 ) + f (x j 1 )(x x j 1 ) + O(h 2 ), f(x j ) = f(x j 1 ) + f (x j 1 )h + O(h 2 ), where the second derivative of f appears in the O sign. Integrating the first relation gives xj x j 1 f(x) dx = hf(x j 1 ) + h2 2 f (x j 1 ) + O(h 3 ).

17 2.1. NUMERICAL INTEGRATION 17 The area of the trapezoid, on the other hand, is (using the second Taylor relation) h 2 (f(x j 1 + f(x j )) = hf(x j 1 ) + h2 2 f (x j 1 )h + O(h 3 ). Those two equations agree, except for the terms O(h 3 ) which usually differ in the two expressions. Hence xj x j 1 f(x) dx = h 2 (f(x j 1) + f(x j )) + O(h 3 ). We can now sum over j (notice that all the terms appear twice, except for the endpoints) to obtain 1 0 f(x) dx = h N 1 2 f(0) + h j=1 f(x j ) + h 2 f(1) + O(h2 ). The trapezoidal rule is second-order accurate. All it took is a modification of the end terms to obtain O(h 2 ) accuracy in place of O(h). Example: f(x) = x 2 in [0, 1]. We have 1 0 x2 dx = 1/3. For the rectangle rule with N = 4 and h = 1/4, consider the gridpoints x = 0, 1/4, 1/2, 3/4, and 1. Each rectangle has height f(x j 1 ) where x j 1 is the left endpoint. We have 1 0 x 2 dx 1 4 [0 + ] This is quite far (O(h), asweknow) from 1/3. = = For the trapezoidal rule, consider the same grid. We now also consider the grid point at x = 1, but the contribution of both x = 0 and x = 1 is halved. 1 x 2 dx = 1 [ ] = = This is much closer (O(h 2 ) as a matter of fact) from 1/3. We ll return to the topic of numerical integration later, after we cover polynomial interpolation.

18 18CHAPTER 2. INTEGRALS AS SUMS AND DERIVATIVES AS DIFFERENCES 2.2 Numerical differentiation The simplest idea for computing an approximation to the derivative u (x j ) of a function u from the samples u j = u(x j ) is to form finite difference ratios. On an equispaced grid x j = jh, the natural candidates are: the one-sided, forward and backward differences + u j = u j+1 u j h and the two-sided centered difference, = u j u j 1, h 0 u j = u j+1 u j 1. 2h Let us justify the accuracy of these difference quotients at approximating derivatives, as h 0. The analysis will depend on the smoothness of the underlying function u. Assume without loss of generality that x j 1 = h, x j = 0, and x j+1 = h. For the forward difference, we can use a Taylor expansion about zero to get u(h) = u(0) + hu (0) + h2 2 u (ξ), ξ [0, h]. (The greek letter ξ is xi.) When substituted in the formula for the forward difference, we get u(h) u(0) = u (0) + h h 2 u (ξ). The error is a O(h) as soon as the function has two bounded derivatives. We say that the forward difference is a method of order one. The analysis for the backward difference is very similar. For the centered difference, we now use two Taylor expansions about zero, for u(h) and u( h), that we write up to order 3: u(±h) = u(0) ± hu (0) + h2 2 u (0) ± h3 6 u (ξ), with ξ either in [0, h] or [ h, 0] depending on the choice of sign. Subtract u( h) from u(h) to get (the h 2 terms cancel out) u(h) u( h) 2h = u (0) + h2 3 u (ξ).

19 2.2. NUMERICAL DIFFERENTIATION 19 The error is now O(h 2 ) provided u is three times differentiable, hence the method is of order 2. The simplest choice for the second derivative is the centered second difference 2 u j = u j+1 2u j + u j 1 h 2. It turns out that 2 = + = +, i.e., the three-point formula for the second difference can be obtained by calculating the forward difference of the backward difference, or vice-versa. 2 is second-order accurate: the error is O(h 2 ). To see this, write the Taylor expansions around x = 0 to fourth order: u(±h) = u(0) ± hu (0) + h2 2 u (0) ± h3 6 u (0) + h4 24 u (ξ). with ξ either in [0, h] or [ h, 0] depending on the choice of sign. The odd terms all cancel out when calculating the second difference, and what is left is u j+1 2u j + u j 1 h 2 = u (0) + h2 12 u (ξ). So the method really is O(h 2 ) only when u is four times differentiable. Example: f(x) = x 2 again. We get f (x) = 2x and f (x) = 2. + f(x) = (x + h)2 x 2 h = 2xh + h2 h = 2x + h. f(x) = x2 (x h) 2 = h 0 f(x) = (x + h)2 (x h) 2 2h 2xh h2 h = 2x h. = 4xh 2h = 2x. 2 f(x) = (x + h)2 2x 2 + (x h) 2 = 2h2 h 2 h = 2. 2 The error is manifestly h for the forward difference, and h for the backward difference (both are O(h).) Coincidentally, the error is zero for 0 and 2 : centered differences differentiate parabolas exactly. This is an exception: finite differences are never exact in general. Of course, 0 = O(h 2 ) so there s no contradiction. When u has less differentiability than is required by the remainder in the Taylor expansions, it may happen that the difference schemes may have lower

20 20CHAPTER 2. INTEGRALS AS SUMS AND DERIVATIVES AS DIFFERENCES order than advertised. This happens in numerical simulations of some differential equations where the solution is discontinuous, e.g., in fluid dynamics (interfaces between phases). Sometimes, points on the left or on the right of x j are not available because we are at one of the edges of the grid. In that case one should use a one-sided difference formula (such as the backward or forward difference.) You can find tables of high-order one-sided formulas for the first, second, and higher derivatives online. The more systematic way of deriving finite difference formulas is by differentiating a polynomial interpolant. We return to the topics of integration and differentiation in the next chapter.

21 Chapter 3 Interpolation Interpolation is the problem of fitting a smooth curve through a given set of points, generally as the graph of a function. It is useful at least in data analysis (interpolation is a form of regression), industrial design, signal processing (digital-to-analog conversion) and in numerical analysis. It is one of those important recurring concepts in applied mathematics. In this chapter, we will immediately put interpolation to use to formulate high-order quadrature and differentiation rules. 3.1 Polynomial interpolation Given points N + 1 points x j R, 0 j N, and sample values y j = f(x j ) of a function at these points, the polynomial interpolation problem consists in finding a polynomial p N (x) of degree N which reproduces those values: y j = p N (x j ), j = 0,..., N. In other words the graph of the polynomial should pass through the points (x j, y j ). A degree-n polynomial can be written as p N (x) = N n=0 a nx n for some coefficients a 0,..., a N. For interpolation, the number of degrees of freedom (N + 1 coefficients) in the polynomial matches the number of points where the function should be fit. If the degree of the polynomial is strictly less than N, we cannot in general pass it through the points (x j, y j ). We can still try to pass a polynomial (.e.g a line) in the best approximate manner, but this is a problem in approximation rather than interpolation; we will return to it later in the chapter on least-squares. 21

22 22 CHAPTER 3. INTERPOLATION Let us first see how the interpolation problem can be solved numerically in a direct way. Use the expression of p N into the interpolating equations y j = p N (x j ): N a n x n j = y j, j = 0,..., N. n=0 In these N + 1 equations indexed by j, the unknowns are the coefficients a 0,..., a N. We are in presence of a linear system V a = y, N V jn a n = y j, n=0 with V the so-called Vandermonde matrix, V jn = x n j, i.e., 1 x 0 x N 0 1 x 1 x N 1 V = x N x N N We can then use a numerical software like Matlab to construct the vector of abscissas x j, the right-hand-side of values y j, the V matrix, and numerically solve the system with an instruction like a = V \ y (in Matlab). This gives us the coefficients of the desired polynomial. The polynomial can now be plotted in between the grid points x j (on a finer grid), in order to display the interpolant. Historically, mathematicians such as Lagrange and Newton did not have access to computers to display interpolants, so they found explicit (and elegant) formulas for the coefficients of the interpolation polynomial. It not only simplified computations for them, but also allowed them to understand the error of polynomial interpolation, i.e., the difference f(x) p N (x). Let us spend a bit of time retracing their steps. (They were concerned with applications such as fitting curves to celestial trajectories.) We ll define the interpolation error from the uniform (L ) norm of the difference f p N : f p N := max f(x) p N (x), x where the maximum is taken over the interval [x 0, x N ].

23 3.1. POLYNOMIAL INTERPOLATION 23 Call P N the space of real-valued degree-n polynomials: P N = { N a n x n : a n R}. n=0 Lagrange s solution to the problem of polynomial interpolation is based on the following construction. Lemma 1. (Lagrange elementary polynomials) Let {x j, j = 0,..., N} be a collection of disjoint numbers. For each k = 0,..., N, there exists a unique degree-n polynomial L k (x) such that { 1 if j = k; L k (x j ) = δ jk = 0 if j k. Proof. Fix k. L k has roots at x j for j k, so L k must be of the form 1 L k (x) = C j k(x x j ). Evaluating this expression at x = x k, we get 1 = C j k(x k x j ) C = 1 j k (x k x j ). Hence the only possible expression for L k is j k L k (x) = (x x j) j k (x k x j ). These elementary polynomials form a basis (in the sense of linear algebra) for expanding any polynomial interpolant p N. 1 That s because, if we fix j, we can divide L k (x) by (x x j ), j k. We obtain L k (x) = (x x j )q(x) + r(x), where r(x) is a remainder of lower order than x x j, i.e., a constant. Since L k (x j ) = 0 we must have r(x) = 0. Hence (x x j ) must be a factor of L k (x). The same is true of any (x x j ) for j k. There are N such factors, which exhausts the degree N. The only remaining degree of freedom in L k is the multiplicative constant.

24 24 CHAPTER 3. INTERPOLATION Theorem 4. (Lagrange interpolation theorem) Let {x j, j = 0,..., N} be a collection of disjoint real numbers. Let {y j, j = 0,..., N} be a collection of real numbers. Then there exists a unique p N P N such that p N (x j ) = y j, j = 0,..., N. Its expression is p N (x) = N y k L k (x), (3.1) k=0 where L k (x) are the Lagrange elementary polynomials. Proof. The justification that (3.1) interpolates is obvious: p N (x j ) = N y k L k (x j ) = k=0 N y k L k (x j ) = y j. It remains to see that p N is the unique interpolating polynomial. For this purpose, assume that both p N and q N take on the value y j at x j. Then r N = p N q N is a polynomial of degree N that has a root at each of the N + 1 points x 0,..., x N. The fundamental theorem of algebra, however, says that a nonzero polynomial of degree N can only have N (complex) roots. Therefore, the only way for r N to have N + 1 roots is that it is the zero polynomial. So p N = q N. k=0 By definition, p N (x) = N f(x k )L k (x) k=0 is called the Lagrange interpolation polynomial of f at x j. Example 7. Linear interpolation through (x 1, y 1 ) and (x 2, y 2 ): L 1 (x) = x x 2 x 1 x 2, L 2 (x) = x x 1 x 2 x 1, p 1 (x) = y 1 L 1 (x) + y 2 L 2 (x) = y 2 y 1 x 2 x 1 x + y 1x 2 y 2 x 1 x 2 x 1 = y 1 + y 2 y 1 x 2 x 1 (x x 1 ).

25 3.1. POLYNOMIAL INTERPOLATION 25 Example 8. (Example (6.1) in Suli-Mayers) Consider f(x) = e x, and interpolate it by a parabola (N = 2) from three samples at x 0 = 1, x 1 = 0, x 2 = 1. We build L 0 (x) = (x x 1)(x x 2 ) (x 0 x 1 )(x 0 x 2 ) = 1 x(x 1) 2 Similarly, So the quadratic interpolant is L 1 (x) = 1 x 2, L 2 (x) = 1 x(x + 1). 2 p 2 (x) = e 1 L 0 (x) + e 0 L 1 (x) + e 1 L 2 (x), = 1 + sinh(1) x + (cosh(1) 1) x 2, x x 2. Another polynomial that approximates e x reasonably well on [ 1, 1] is the Taylor expansion about x = 0: t 2 (x) = 1 + x + x2 2. Manifestly, p 2 is not very different from t 2. (Insert picture here) Let us now move on to the main result concerning the interpolation error of smooth functions. Theorem 5. Let f C N+1 [a, b] for some N > 0, and let {x j : j = 0,..., N} be a collection of disjoint reals in [a, b]. Consider p N the Lagrange interpolation polynomial of f at x j. Then for every x [a, b] there exists ξ(x) [a, b] such that f(x) p N (x) = f (N+1) (ξ(x)) π N+1 (x), (N + 1)! where N+1 π N+1 (x) = (x x j ). j=1 An estimate on the interpolation error follows directly from this theorem. Set M N+1 = max x [a,b] f (N+1) (x)

26 26 CHAPTER 3. INTERPOLATION (which is well defined since f (N+1) is continuous by assumption, hence reaches its lower and upper bounds.) Then f(x) p N (x) M N+1 (N + 1)! π N+1(x) In particular, we see that the interpolation error is zero when x = x j for some j, as it should be. Let us now prove the theorem. Proof. (Can be found in Suli-Mayers, Chapter 6) In conclusion, the interpolation error: depends on the smoothness of f via the high-order derivative f (N+1) ; has a factor 1/(N + 1)! that decays fast as the order N ; and is directly proportional to the value of π N+1 (x), indicating that the interpolant may be better in some places than others. The natural follow-up question is that of convergence: can we always expect convergence of the polynomial interpolant as N? In other words, does the factor 1/(N + 1)! always win over the other two factors? Unfortunately, the answer is no in general. There are examples of very smooth (analytic) functions for which polynomial interpolation diverges, particularly so near the boundaries of the interplation interval. This behavior is called the Runge phenomenon, and is usually illustrated by means of the following example. Example 9. (Runge phenomenon) Let f(x) for x [ 5, 5]. Interpolate it at equispaced points x j = 10j/N, where j = N/2,..., N/2 and N is even. It is easy to check numerically that the interpolant diverges near the edges of [ 5, 5], as N. See the Trefethen textbook on page 44 for an illustration of the Runge phenomenon. (Figure here) If we had done the same numerical experiment for x [ 1, 1], the interpolant would have converged. This shows that the size of the interval matters. Intuitively, there is divergence when the size of the interval is larger than the features, or characteristic length scale, of the function (here the width of the bump near the origin.)

27 3.2. POLYNOMIAL RULES FOR INTEGRATION 27 The analytical reason for the divergence in the example above is due in no small part to the very large values taken on by π N+1 (x) far away from the origin in contrast to the relatively small values it takes on near the origin. This is a problem intrinsic to equispaced grids. We will be more quantitative about this issue in the section on Chebyshev interpolants, where a remedy involving non-equispaced grid points will be explained. As a conclusion, polynomial interpolants can be good for small N, and on small intervals, but may fail to converge (quite dramatically) when the interpolation interval is large. 3.2 Polynomial rules for integration In this section, we return to the problem of approximating b u(x)dx by a a weighted sum of samples u(x j ), also called a quadrature. The plan is to form interpolants of the data, integrate those interpolants, and deduce corresponding quadrature formulas. We can formulate rules of arbitrarily high order this way, although in practice we almost never go beyond order 4 with polynomial rules Polynomial rules Without loss of generality, consider the local interpolants of u(x) formed near the origin, with x 0 = 0, x 1 = h and x 1 = h. The rectangle rule does not belong in this section: it is not formed from an interpolant. The trapezoidal rule, where we approximate u(x) by a line joining (0, u(x 0 )) and (h, u(x 1 )) in [0, h]. We need 2 derivatives to control the error: u(x) = p 1 (x) + u (ξ(x)) x(x h), 2 p 1 (x) = u(0)l 0 (x) + u(h)l 1 (x), h 0 L 0 (x)dx = h 0 L 0 (x) = h x h, L 1(x) = x h, h 0 u (ξ(x)) 2 L 1 (x)dx = h/2, (areas of triangles) x(x h) dx C max u (ξ) h 3. ξ

28 28 CHAPTER 3. INTERPOLATION The result is h 0 ( ) u(0) + u(h) u(x) dx = h + O(h 3 ). 2 As we have seen, the terms combine as 1 0 u(x) dx = h N 1 2 u(x 0) + h u(x j 1 ) + h 2 u(x N) + O(h 2 ). j=1 Simpson s rule, where we approximate u(x) by a parabola through ( h, u(x 1 )), (0, u(x 0 )), and (h, u(x 1 )) in [ h, h]. We need three derivatives to control the error: u(x) = p 2 (x) + u (ξ(x)) (x + h)x(x h), 6 p 2 (x) = u( h)l 1 (x) + u(0)l 0 (x) + u(h)l 1 (x), x(x h) (x + h)(x h) L 1 (x) =, L 2h 2 0 (x) =, L h 2 1 (x) = h L 1 (x)dx = h h 0 The result is h 0 h h u (ξ(x)) 6 h L 1 (x)dx = h/3, h h L 0 (x)dx = 4h/3, (x + h)x(x h) dx C max u (ξ) h 4. ξ ( ) u( h) + 4u(0) + u(h) u(x) dx = h + O(h 4 ). 3 (x + h)x 2h 2, The composite Simpson s rule is obtained by using this approximation on [0, 2h] [2h, 4h]... [1 2h, 1], adding the terms, and recognizing that the samples at 2nh (except 0 and 1) are represented twice. 1 ( ) u(0) + 4u(h) + 2u(2h) + 4u(3h) u(1 2h) + 4u(1 h) + u(1) u(x)dx = h 3 It turns out that the error is in fact O(h 5 ) on [ h, h], and O(h 4 ) on [0, 1], a result that can be derived by using symmetry considerations (or canceling the terms from Taylor expansions in a tedious way.) For this, we need u to be four times differentiable (the constant in front of h 5 involves u.)

29 3.3. POLYNOMIAL RULES FOR DIFFERENTIATION 29 The higher-order variants of polynomial rules, called Newton-Cotes rules, are not very interesting because the Runge phenomenon kicks in again. Also, the weights (like 1,4,2,4,2, etc.) become negative, which leads to unacceptable error magnification if the samples of u are not known exactly. Piecewise spline interpolation is not a good choice for numerical integration either, because of the two leftover degrees of freedom, whose arbitrary choice affects the accuracy of quadrature in an unacceptable manner. (We ll study splines in a later section.) We ll return to (useful!) integration rules of arbitrarily high order in the scope of spectral methods. 3.3 Polynomial rules for differentiation A systematic way of deriving finite difference formulas of higher order is to view them as derivatives of a polynomial interpolant passing through a small number of points neighboring x j. For instance (again we use h, 0, h as reference points without loss of generality): The forward difference at 0 is obtained from the line joining (0, u(0)) and (h, u(h)): p 1 (x) = u(0)l 0 (x) + u(h)l 1 (x), L 0 (x) = h x h, L 1(x) = x h, p u(h) u(0) 1(0) =. h We already know that u (0) p 1(0) = O(h). The centered difference at 0 is obtained from the line joining ( h, u( h)) and (h, u(h)) (a simple exercise), but it is also obtained from differentiating the parabola passing through the points ( h, u( h)), (0, u(0)), and (h, u(h)). Indeed, p 2 (x) = u( h)l 1 (x) + u(0)l 0 (x) + u(h)l 1 (x), L 1 (x) = x(x h) 2h 2, L 0 (x) = p 2(x) = u( h) 2x h 2h 2 (x + h)(x h) h 2, L 1 (x) = + u(0) 2x h 2 + u(h)2x + h 2h 2, (x + h)x 2h 2,

30 30 CHAPTER 3. INTERPOLATION p u(h) u( h) 2(0) =. 2h We already know that u (0) p 2(0) = O(h 2 ). Other examples can be considered, such as the centered second difference (-1 2-1), the one-sided first difference (-3 4-1), etc. Differentiating one-sided interpolation polynomials is a good way to obtain one-sided difference formulas, which comes in handy at the boundaries of the computational domain. The following result establishes that the order of the finite difference formula matches the order of the polynomial being differentiated. Theorem 6. Let f C N+1 [a, b], {x j, j = 0,..., N} some disjoint points, and p N the corresponding interpolation polynomial. Then f (x) p N(x) = f (N+1) (ξ) N! π N(x), for some ξ [a, b], and where π N (x) = (x η 1)... (x η N ) for some η j [x j 1, x j ]. The proof of this result is an application of Rolle s theorem that we leave out. (It is not simply a matter of differentiating the error formula for polynomial interpolation, because we have no guarantee on dξ/dx.) A consequence of this theorem is that the error in computing the derivative is a O(h N ) (which comes from a bound on the product π N (x).) It is interesting to notice, at least empirically, that the Runge s phenomenon is absent when the derivative is evaluated at the center of the interval over which the interpolant is built. 3.4 Piecewise polynomial interpolation The idea of piecewise polynomial interpolation, also called spline interpolation, is to subdivide the interval [a, b] into a large number of subintervals [x j 1, x j ], and to use low-degree polynomials over each subintervals. This helps avoiding the Runge phenomenon. The price to pay is that the interpolant is no longer a C function instead, we lose differentiability at the junctions between subintervals, where the polynomials are made to match.

31 3.4. PIECEWISE POLYNOMIAL INTERPOLATION 31 If we use polynomials of order n, then the interpolant is piecewise C, and overall C k, with k n 1. We could not expect to have k = n, because it would imply that the polynomials are identical over each subinterval. We ll see two examples: linear splines when n = 1, and cubic splines when n = 3. (We have seen in the homework why it is a bad idea to choose n = 2.) Linear splines We wish to interpolate a continuous function f(x) of x [a, b], from the knowledge of f(x j ) at some points x j, j = 0,..., N, not necessarily equispaced. Assume that x 0 = a and x N = b. The piecewise linear interpolant is build by tracing a straight line between the points (x j 1, f(x j 1 )) and (x j, f(x j ))); for j = 1,..., N the formula is simply s L (x) = x j 1 x x j x j 1 f(x j 1 ) + x x j 1 x j x j 1 f(x j ), x [x j 1, x j ]. In this case we see that s L (x) is a continuous function of x, but that its derivative is not continuous at the junction points, or nodes x j. If the function is at least twice differentiable, then piecewise linear interpolation has second-order accuracy, i.e., an error O(h 2 ). Theorem 7. Let f C 2 [a, b], and let h = max j=1,...,n (x j x j 1 ) be the grid diameter. Then f s L L [a,b] h2 8 f L [a,b]. Proof. Let x [x j 1, x j ] for some j = 1,..., N. We can apply the basic result of accuracy of polynomial interpolation with n = 1: there exists ξ [x j 1, x j ] such that f(x) s L (x) = 1 2 f (ξ) (x x j 1 )(x x j ), x [x j 1, x j ]. Let h j = x j x j 1. It is easy to check that the product (x x j 1 )(x x j ) takes its maximum value at the midpoint x j 1+x j, and that the value is h 2 2 j/4. We then have f(x) s L (x) h2 j 8 max f (ξ), x [x j 1, x j ]. ξ [x j 1,x j ] The conclusion follows by taking a maximum over j.

32 32 CHAPTER 3. INTERPOLATION We can express any piecewise linear interpolant as a superposition of tent basis functions φ k (x): s L (x) = k c k φ k (x), where φ k (x) is the piecewise linear function equal to 1 at x = x k, and equal to zero at all the other grid points x j, j k. Or in short, φ k (x j ) = δ jk. On an equispaced grid x j = jh, an explicit formula is where φ k (x) = 1 h S (1)(x x k ), S (1) (x) = x + 2(x h) + + (x 2h) +, and where x + denotes the positive part of x (i.e., x if x 0, and zero if x < 0.) Observe that we can simply take c k = f(x k ) above. The situation will be more complicated when we pass to higher-order polynomials Cubic splines Let us now consider the case n = 3 of an interpolant which is a third-order polynomial, i.e., a cubic, on each subinterval. The most we can ask is that the value of the interpolant, its derivative, and its second derivative be continuous at the junction points x j. Definition 5. (Interpolating cubic spline) Let f C[a, b], and {x j, j = 0,..., N} [a, b]. An interpolating cubic spline is a function s(x) such that 1. s(x j ) = f(x j ); 2. s(x) is a polynomial of degree 3 over each segment [x j 1, x j ]; 3. s(x) is globally C 2, i.e., at each junction point x j, we have the relations s(x j ) = s(x+ j ), s (x j ) = s (x + j ), s (x j ) = s (x + j ), where the notations s(x j ) and s(x+ j ) refer to the adequate limits on the left and on the right.

33 3.4. PIECEWISE POLYNOMIAL INTERPOLATION 33 Let us count the degrees of freedom. A cubic polynomial has 4 coefficients. There are N + 1 points, hence N subintervals, for a total of 4N numbers to be specified. The interpolating conditions s(x j ) = f(x j ) specify two degrees of freedom per polynomial: one value at the left endpoint x j 1, and one value at the right endpoint x j. That s 2N conditions. Continuity of s(x) follows automatically. The continuity conditions on s, s are imposed only at the interior grid points x j for j = 1,..., N 1, so this gives rise to 2(N 1) additional conditions, for a total of 4N 2 equations. There is a mismatch between the number of unknowns (4N) and the number of conditions (4N 2). Two more degrees of freedom are required to completely specify a cubic spline interpolant, hence the precaution to write an interpolant in the definition above, and not the interpolant. The most widespread choices for fixing these two degrees of freedom are: Natural splines: s (x 0 ) = s (x N ) = 0. If s(x) measures the displacement of a beam, a condition of vanishing second derivative corresponds to a free end. Clamped spline: s (x 0 ) = p 0, s (x N ) = p N, where p 0 and p N are specified values that depend on the particular application. If s(x) measures the displacement of a beam, a condition of vanishing derivative corresponds to a horizontal clamped end. Periodic spline: assuming that s(x 0 ) = s(x N ), then we also impose s (x 0 ) = s (x N ), s (x 0 ) = s (x N ). Let us now explain the algorithm most often used for determining a cubic spline interpolant, i.e., the 4N coefficients of the N cubic polynomials, from the knowledge of f(x j ). Let us consider the natural spline. It is advantageous to write a system of equations for the second derivatives at the grid points, that we denote σ j = s (x j ). Because s(x) is piecewise cubic, we know that s (x) is piecewise linear. Let h j = x j x j 1. We can write s (x) = x j x σ j 1 + x x j 1 σ j, x [x j 1, x j ]. h j h j

34 34 CHAPTER 3. INTERPOLATION Hence s(x) = (x j x) 3 σ j 1 + (x x j 1) 3 σ j + α j (x x j 1 ) + β j (x j x). 6h j 6h j We could have written ax + b for the effect of the integration constants in the equation above, but writing it in terms of α j and β j makes the algebra that follows simpler. The two interpolation conditions for [x j 1, x j ] can be written s(x j 1 ) = f(x j 1 ) f(x j 1 ) = σ j 1h 2 j 6 + h j β j, s(x j ) = f(x j ) f(x j ) = σ jh 2 j + h j α j. 6 One then isolates α j, β j in this equation; substitutes those values in the equation for s(x); and evaluates the latter at x j to get s(x j ) = f(x j ). Given that σ 0 = σ N = 0, we end up with a system of N 1 equations in the N 1 unknowns σ 1,..., σ N. Skipping the algebra, the end result is ( f(xj+1 ) f(x j ) h j σ j 1 + 2(h j+1 +h j )σ j + h j+1 σ j+1 = 6 f(x ) j) f(x j 1 ). h j+1 h j We are in presence of a tridiagonal system for σ j. It can be solved efficiently with Gaussian elimination, yielding a LU decomposition with bidiagonal factors. Unlike in the case of linear splines, there is no way around the fact that a linear system needs to be solved. Notice that the tridiagonal matrix of the system above is diagonally dominant (each diagonal element is strictly greater than the sum of the other elements on the same row, in absolute value), hence it is always invertible. One can check that the interpolation for cubic splines is O(h 4 ) well away from the endpoints. This requires an analysis that is too involved for the present set of notes. Finally, like in the linear case, let us consider the question of expanding a cubic spline interpolant as a superposition of basis functions s(x) = k c k φ k (x). There are many ways of choosing the functions φ k (x), so let us specify that they should have as small a support as possible, that they should have the

35 3.4. PIECEWISE POLYNOMIAL INTERPOLATION 35 same smoothness C 2 as s(x) itself, and that they should be translates of each other when the grid x j is equispaced with spacing h. The only solution to this problem is the cubic B-spline. Around any interior grid point x k, is supported in the interval [x k 2, x k+2 ], and is given by the formula φ k (x) = 1 4h S (3)(x x 3 k 2 ), where S (3) (x) = x 3 + 4(x h) (x 2h) 3 + 4(x 3h) (x 4h) 3 +, and where x + is the positive part of x. One can check that φ k (x) takes the value 1 at x k, 1/4 at x k±1, zero outside of [x k 2, x k+2 ], and is C 2 at each junction. It is a bell-shaped curve. it is an interesting exercise to check that it can be obtained as the convolution of two tent basis functions of linear interpolation: S (3) (x) = cs (1) (x) S (1) (x), where c is some constant. Now with cubic B-splines, we cannot put c k = f(x k ) anymore, since φ k (x j ) δ jk. The particular values of c k are the result of solving a linear system, as mentioned above. Again, there is no way around solving a linear system. In particular, if f(x k ) changes at one point, or if we add a datum point (x k, f k ), then the update requires re-computing the whole interpolant. Changing one point has ripple effects throughout the whole interval [a, b]. This behavior is ultimately due to the more significant overlap between neighboring φ k in the cubic case than in the linear case.

36 36 CHAPTER 3. INTERPOLATION

37 Chapter 4 Methods for ODE 37

38 38 CHAPTER 4. METHODS FOR ODE

39 Chapter 5 Ordinary differential equations 5.1 Initial-value problems Initial-value problems (IVP) are those for which the solution is entirely known at some time, say t = 0, and the question is to solve the ODE y (t) = f(t, y(t)), y(0) = y 0, for other times, say t > 0. We will consider a scalar y, but considering systems of ODE is a straightforward extension for what we do in this chapter. We ll treat both theoretical questions of existence and uniqueness, as well as practical questions concerning numerical solvers. We speak of t as being time, because that s usually the physical context in which IVP arise, but it could very well be a space variable Theory Does a solution exist, is it unique, and does it tend to infinity (blow up) in finite time? These questions are not merely pedantic. As we now show with two examples, things can go wrong very quickly if we posit the wrong ODE. Example 10. Consider y = y, y(0) = 0. By separation of variables, we find dy = dt y(t) = y 39 (t + C)2. 4

40 40 CHAPTER 5. ORDINARY DIFFERENTIAL EQUATIONS Imposing the initial condition yields C = 0, hence y(t) = t 2 /4. However, y(t) = 0 is clearly another solution, so we have non-uniqueness. In fact, there is an infinite number of solutions, corresponding to y(t) = 0 for 0 t t for some t, which then takes off along the parabola the parabola y(t) = (t t ) 2 /4 for times t t. Example 11. Consider y = y 2, y(0) = 1. By separation of variables, we find dy y = dt y(t) = 1 2 t + C. The initial condition gives C = 1, hence y(t) = 1/(1 t). It blows up at time t = 1, because y(t) has a vertical asymptote. We say that the solution exists locally in any interval to the left of t = 1, but we don t have global existence. Blowup and non-uniqueness are generally, although not always 1, unrealistic in applications. A theory of existence and uniqueness, including global existence (non blowup), is desired to guide us in formulating valid models for physical phenomena. The basic result is the following. Theorem 8. (Picard) For given T, C, and y 0, consider the box B in (t, y) space, given by B = [0, T ] [y 0 C, y 0 + C]. Assume that f(t, y) is continuous over B; f(t, y) K when (t, y) B; (boundedness) f(t, u) f(t, v) L u v) when (t, u), (t, v) B. continuity). (Lipschitz 1 In nonlinear optics for instance, laser pulses may collapse. This situation is somewhat realistically modeled by an ODE that blows up, although not for times arbitrarily close to the singularity.

41 5.1. INITIAL-VALUE PROBLEMS 41 Assume furthermore that C K L (elt 1). Then there exists a unique y C 1 [0, T ], such that y (t) = f(t, y(t)), y(0) = y 0, and such that y(t) y 0 C. In short, the solution exists, is unique, and stays in the box for times 0 t T. Proof. The technique is called Picard s iteration. See p.311 in Suli-Mayers Numerics Here is an overview of some of the most popular numerical methods for solving ODEs. Let t n = nh, and denote by y n the approximation of y(t n ). 1. Explicit Euler. y n+1 = y n + hf(t n, y n ). This formula comes from approximating the derivative y at t = t n by a forward difference. It allows to march in time from the knowledge of y n, to get y n Implicit Euler. y n+1 = y n + hf(t n+1, y n+1 ). This time we use a backward difference for approximating the derivative at t = t n+1. The unknown y n+1 appears implicitly in this equation, it still needs to be solved for as a function of y n, using (for instance) Newton s method. The strategy is still to march in time, but at every step there is a nonlinear equation to solve. 3. Midpoint rule (implicit). y n+1 = y n + h 2 [f(t n, y n ) + f(t n+1, y n+1 )]. The derivative is approximated by a centered difference, at t = t+n+t n+1 2 (the midpoint). This gives a more balanced estimate of the slope. It is an implicit method: y n+1 needs to be solved for. y n+1 y n h

42 42 CHAPTER 5. ORDINARY DIFFERENTIAL EQUATIONS 4. Improved Euler, Runge-Kutta 2 (explicit). ỹ n+1 = y n + hf(t n, y n ), y n+1 = y n + h 2 [f(t n, y n ) + f(t n+1, ỹ n+1 )]. This is the simplest of predictor-corrector methods. It is like the midpoint rule, except that we use a guess ỹ n+1 for the unknown value of y n+1 in the right-hand side, and this guess comes from the explicit Euler method. Now y n+1 only appears in the left-hand side, so this is an explicit method. 5. Runge-Kutta 4 (explicit). y n+1 = y n + h[k 1 + 2k 2 + 2k 3 + k 4 ], where the slopes k 1,..., k 4 are given in succession by k 1 = f(t n, y n ), k 2 = f(t n + h 2, y n + h 2 k 1), k 3 = f(t n + h 2, y n + h 2 k 2), k 4 = f(t n + h, y n + hk 3 ). 6. There are also methods that involve not just the past value y n, but a larger chunk of history y n 1, y n 2,etc. These methods are called multistep. They are in general less flexible than the one-step methods described so far, in that they require a constant step h as we march in time. Two features of a numerical method are important when choosing a numerical method: Is it convergent, i.e., does the computed solution tend to the true solution as h 0, and at which rate? Is it stable, i.e., if we solve with different initial conditions y 0 and ỹ 0, are the computed solutions close in the sense that y n ỹ n C y 0 ỹ 0, with n = O(1/h), and C independent of h?

43 5.1. INITIAL-VALUE PROBLEMS 43 To understand convergence better in the case of one-step methods, let us write the numerical scheme as and introduce the local error as well as the global error y n+1 = Ψ(t n, y n, h), e n+1 (h) = Ψ(t n, y(t n ), h) y(t n+1 ), E n (h) = y n y(t n ). Convergence is the study of the global error. The local error, however, is easier to understand. A numerical method is called consistent if the local error decays sufficiently fast as h 0 that there is hope that the global error would be small as well. The particular rate at which the local error decays is related to the notion of order of an ODE solver. Definition 6. (Consistency) Ψ is consistent if, for any n 0, e n (h) lim h 0 h = 0 Definition 7. (Order) Ψ is of order p if e n (h) = O(h p+1 ). The basic convergence result for one-step solvers, that we will not prove, is that if the local error is O(h p+1 ), then the global error is O(h p ). This convergence result is only true as stated for one-step methods; for multi-step methods we would also need an assumption of stability (discussed below). Intuitively, the local errors compound over the O(1/h) time steps necessary to reach a given fixed time t, hence the loss of one power of h. Of course the local errors don t exactly add up; they do up to a multiplicative constant. It is the behavior of the global error that dictates the notion of order of the numerical scheme. It is a good exercise to show, using elementary Taylor expansions, that the explicit and implicit Euler methods are of order 1, and that the midpoint rule and improved Euler methods are of order 2. It turns out that Runge-Kutta 4 is of order 4, but it is not much fun to prove that. Quantifying stability as stated above is a complex question. The important ideas already appear if we study the representative setting of linear stability.