Ordinary Differential Equations

Ordinary Differential Equations Professor Dr. E F Toro Laboratory of Applied Mathematics University of Trento, Italy eleuterio.toro@unitn.it http://www.ing.unitn.it/toro September 19, 2014 1 / 55

Motivation Many problems in engineering, physics, chemistry, biology, ecology, social sciences and many others, are modelled by Ordinary Differential Equations (ODEs). Example 1: The simplest example of an ODE is q (t) = C 1, C 1 : Constant. (1) q(t) is a function of one variable: t. q : dependent variable t : independent variable Equation (1) involves all functions q(t) whose first derivative with respect to t is a constant C 1. Clearly, all solutions of (1) have the form (verify) q(t) = C 0 + C 1 t, C 0 : Constant. (2) There are infinitely many solutions, they are all straight lines in the t q plane, with a constant slope C 1 and constant intercept C 0 of the q axis. 2 / 55

Note: a solution of (1) is a function q(t) such that when substituted into (1) gives an identity. Example 2: q (t) = β q(t), β : Constant. (3) Clearly, any function q(t) = Ke βt, K : Constant (4) is a solution of (3). Differentiating (4): q (t) = β(ke βt ) = β q(t) Example 3: q (t) + q(t) = 0, One solution is: q(t) = sin t. (5) Exercise: Find all solutions of ODE in (5). Example 4: q (t) = 3γ γ 2 t 2 + q 2 (t), γ : Constant. (6) Exercise: Verify by substitution that q(t) = γt 1/t (7) is a solution of the differential equation, for t away from 0. 3 / 55

Definition: The order of an ODE is the order of the highest derivative present in the equation. Examples 1, 2 and 4 above are first order ODEs, while example 3 is a second-order ODE. Definition: An ODE is called non-linear if there are terms involving products of q(t) and its derivatives. Otherwise the ODE is linear. ODEs 1 to 3 above are linear, while example 4 is non-linear. Definition: A linear, variable-coefficient ODE has the form a 0 (t)q(t) + a 1 (t)q (t) + + a k (t)q (k) (t) = b(t), (8) where a k (t) are prescribed functions of t, the coefficients, and q (k) (t) = dk q(t). (9) dtk In all examples above the solution is non unique. To secure uniqueness one must impose additional conditions on the solution to the problem. We shall study two kinds of such problems: Initial Value Problems (IVP) and Boundary Value Problems (BVP) 4 / 55

Initial Value Problem (IVP) for ODEs Definition: An Initial Value Problem (IVP) has the form ODE: q (t) = s(t, q(t)), } t (a, b) IC: q(a) = q 0. s(t, q) is a prescribed function of two variables, q 0 is a prescribed constant and [a, b] denotes a real interval. Theorem: existence and uniqueness. Let s(t, q) be a function defined and continuous in the set (10) D = {(t, q) : < a t b <, q R}. (11) Assume in addition that there exists a constant L > 0 such that s(t, q 2 ) s(t, q 1 ) L q 2 q 1 (Lipschitz condition) (12) for all t [a, b] and for any pair q 1, q 2 in R. Then there exists only one function q(t) that satisfies the ODE in (10) t [a, b] as well as the prescribed initial condition q(a) = q 0. Proof: omitted 5 / 55

Systems of ODEs We consider first-order systems of ODEs, involving a vector Q(t) of m unknowns q i (t), i = 1,..., m. The IVP is ODEs: Q } (t) = S(t, Q(t)), t [a, b] ICs: Q(a) = Q 0 (13) q 1 (t) q (0) 1 s 1 (t, q 1,..., q m ) Q(t) = q 2 (t)..., q (0) Q0 = 2, S(t, Q(t)) = s 2 (t, q 1,..., q m )....... q m (t) q m (0) s m (t, q 1,..., q m ) (14) Q 0 (constant) and S(t, Q(t)) are prescribed vectors. Example: q 1 (t) = α 1q 1 (t) + β 1 q 1 (t)q 2 (t) s 1 (t, q 1 (t), q 2 (t)), q 2 (t) = α 2q 1 (t) + β 2 q 1 (t)q 2 (t) s 2 (t, q 1 (t), q 2 (t)). q 1 (t) and q 2 (t) are the unknowns and α 1, α 2, β 1, β 1 are parameters. } (15) 6 / 55

Example: A linear system Q (t) = AQ(t), t [a, b], A = a 11 a 12... a 1m a 21 a 22... a 2m... a m1 a m2... a mm A is the coefficient matrix, constant here. Note that (16) is the system analogue of the model equation (3). High-order ODEs. Example.. (16) q (t) + αq (t) + βq(t) + γ = 0. (17) α, β and γ are known constant parameters. This single second-order ODE may be written as a system of two first-order ODEs, by first defining the following two functions q 1 (t) = q(t), q 2 (t) = q (t). (18) 7 / 55

Then we have q 1 (t) = } q (t) = q 2 (t), q 2 (t) = q (t) = γ αq (t) βq(t) = γ αq 2 (t) βq 1 (t), (19) obtaining the first-order linear system q 1 (t) = q 2(t), q 2 (t) = βq 1(t) αq 2 (t) γ, which written in matrix form reads [ ] [ q1 (t) 0 1 = q 2 (t) β α ] [ q1 (t) q 2 (t) } ] [ 0 + γ ] (20). (21) Remark: autonomous system. The explicit presence of t may be eliminated by representing t as a new component q 0 for the vector of unknowns, that is Q = [q 0, q 1, q 2..., q m ]. Then one adds the new ODE q 0 = 1. Then the IVP (13) can be written as ODEs: Q (t) = S(Q), t [a, b] ICs: Q(a) = Q 0. } (22) 8 / 55

Example: write in autonomous form the following system q 1 (t) = q 1(t) q 2 (t) + 2t t 2 t 3 }, q 2 (t) = q 1(t) + q 2 (t) 4t 2 + t 3. (23) Solution: Setting q 0 = t we obtain q 0 1 q 1 = q 1 q 2 + 2q 0 q 2 q 2 0 q3 0 q 1 + q 2 4q0 2 + q3 0. (24) Exercise: verify. 9 / 55

Numerical Approximation: Basic Notions Realistic problems involving ODEs cannot be solved analytically and we thus use numerical methods to compute approximate solutions. Let us consider the IVP q } (t) = s(t, q(t)), t [a, b], q(a) = q 0 (25). The true solution of the continuous problem is defined at every point t [a, b]. Instead, a numerical approximation first considers a finite number of points t n [a, b] resulting from partition of [a, b] into a finite number of subintervals. Then one seeks approximate values q n to the true solution q(t n ). The simplest case is a regular partition of [a, b] into N subintervals of constant length h, with associated N + 1 equally spaced points t n as follows a = t 0 < t 1 <... t n 1 < t n < t n+1... t N = b. (26) h is a discretization parameter, called the time step or steplength, with h = b a N, tn = a + nh, n = 0, 1, 2,..., N. (27) 10 / 55

We then have q n q(t n ), n = 0, 1, 2,..., N, (28) to mean that q n is an approximation to the exact solution q(t) at the point t n. The numerical solution is given by the finite set of point values {q n } n=n n=0. See Fig. 1. Fig. 1. Illustration of the numerical approximation of an ODE on a regular mesh of (constant) steplength h. At any given time t n there is an error q(t n ) q n. 11 / 55

Errors At a point t n we define the error as e n = q(t n ) q n or e n = q n q(t n ). (29) See Fig. 1 Other forms of the discretisation error are the absolute error while the relative error is e n = q(t n ) q n, (30) r n = q(tn ) q n q(t n ). (31) In this context it is also useful to introduce the so-called big O notation g(t) = O(r(t)), as t t 0, (32) which means that there exists a positive constant K such that g(t) K r(t), as t t 0, (33) for any open interval that contains the point t 0. For example, we can write T (h) = O(h 2 ), as h 0. (34) 12 / 55

Taylor Methods. The Bases Consider the IVP (25) with initial condition q(t n ) = q n at t = t n. If q(t) is sufficiently smooth, then a Taylor series reads q(t n +h) = q(t n )+hq (1) (t n )+ 1 2 h2 q (2) (t n )+...+ 1 p! hp q (n) (t n )+O(h p+1 ). (35) By neglecting terms O(h p+1 ) we then write an exact relation between approximate values [ q n+1 = q n + h q (1) (t n ) + 1 2 h1 q (2) (t n ) +... + 1 ] p! hp 1 q (n) (t n ). (36) From here we write a numerical method, called a Taylor method, as q n+1 = q n + hφ(t n, q n, h), φ(t n, q n, h) = q (1) (t n ) + 1 2 h1 q (2) (t n ) +... + 1 p! hp 1 q (n) (t n ), (37) where φ(t n, q n, h) is the increment function, which is determined once (k) n 13 / 55

Example: Euler s method Euler s method is the simplest Taylor method. Only the first term of φ(t n, q n, h) is used, while the leading term q (1) (t n ) is found from the ODE in (25) at time t = t n, that is q (1) (t n ) = s(t n, q n ). The method then reads q n+1 = q n + hφ(t n, q n }, h), φ(t n, q n, h) = s(t n, q n (38) ). Euler s method is very simple but inaccurate (first-order accurate). The model IVP. We introduce the model initial-value problem q } (t) = βq(t), t [0, b], q(0) = q 0 (39). Here s(t, q(t)) = βq(t) where, unless otherwise stated, β 0. As an example, solve by hand the model IVP (39) with b = 5, β = 1, q 0 = 1 using Euler s method, with h = 1. We obtain: q n+1 = q n + hs(t n, q n ) = q n + hβq n = (1 + βh)q n = 2q n. (40) 14 / 55

Table 1 shows the Euler results, the exact solution and the error. The numerical solution is very inaccurate and error grows very rapidly, as time increases. n t n q n (Numerical, Euler) Exact Error 0 0 1.0 1.0 0.0 1 1 2.0 2.7183 0.7183 2 2 4.0 7.3891 3.3891 3 3 8.0 20.0855 12.0855 4 4 16.0 54.5981 38.5981 5 5 32.0 148.4132 116.4132 Table 1. Model IVP solved with Euler s method with h = 1. A high-order Taylor method for the model IVP is easily constructed after finding the high-order derivatives in (37) as Then the Taylor method of order p is q (p) (t) = β p q, p. (41) q n+1 = P (r)q n, r βh, P (r) = (1 + r + 1 2 r2 +... + 1 p! rp ). (42) 15 / 55

Taylor s second-order method Solve the model IVP (39) with b = 5, β = 1, q 0 = 1 using Taylor s second-order method, with h = 1. We have q n+1 = (1 + r + 1 2 r2 )q n = 5 2 qn, r = βh. (43) Table 2 shows the Taylor second-order results, the exact solution and the error. Note that thought still inaccurate, Taylor s second order method is visibly better than the first order Euler method, see Table 1. n t n q n (Numerical, 2nd order Taylor) Exact Error 0 0 1.0 1.0 0.0 1 1 2.5 2.7183 0.2183 2 2 6.25 7.3891 1.1391 3 3 15.625 20.0855 4.4605 4 4 39.0625 54.5981 15.5356 5 5 97.6563 148.4132 50.7569 Table 2. Model IVP solved with Taylor s second order method with h = 1. 16 / 55

High-order Taylor methods for non-linear IVPs Taylor s method (37) for non-linear IVPs (10) requires the computation of derivatives q (p) (t) in (37). This is done using the ODE in (10) and we have q (1) (t) = s(t, q), q (2) (t) = d dts(t, q),... =... (44) q (p) (t) = dp 1 s(t, q). dtp 1 Exercise: Verify that and q (2) (t) = d s s(t, q(t)) = dt t + s s q q (3) (t) = d2 dt 2 s(t, q(t)) = 2 s t 2 +2s 2 s t q +s2 2 s q 2 + s t s q +s (45) ( ) s 2. (46) q High-order Taylor methods for non-linear ODEs may require the use of symbolic manipulators to find the high-order derivatives. 17 / 55

Some Theoretical Notions Definition: Consistency I. A method of the form (37) is said to be consistent (consistent with the ODE in (10)) if lim φ(t, q, h) = φ(t, q, 0) = s(t, q). (47) h 0 The Euler method (38) is consistent, as φ(t, q, 0) = s(t, q). Operators. To define other concepts we first re-write the differential equation in operator form as L e (q(t)) = q s(t, q) = 0, (48) where L e is the (exact) differential operator. Analogously, we can define the numerical method (37) in terms of a numerical operator L a, as L a (q n ) = q n+1 [q n + hφ(t n, q n, h)] = 0. (49) 18 / 55

Definition: Local truncation error and accuracy The (normalized) local truncation error τ n at the point t n is defined as τ n = 1 h L a(q(t n )). (50) Note that τ n involves the numerical operator L a evaluated at the exact point value q(t n ) at time t n. In general, τ n 0. Since q(t) is the exact solution, the error is due exclusively to L a and so τ n measures the error due to the numerical scheme. In the literature, there are other forms of defining the local truncation error. For the Euler method applied to the model IVP (39) we have L a (q n ) = q n+1 [q n + hβq n ] = 0. (51) Therefore, applying the definition of local truncation error, Taylor expanding q(t) about t n and using the exact differential operator we have 19 / 55

τ n = 1 h {L a(q(t n ))}, = 1 { h q(t n+1 ) [q(t n ) + hβq(t n )] } {, } = 1 h q(t n ) + hq (1) + 1 2 h2 q (2) +... + 1 p! hp q (p) q(t n ) hβq(t n ), { } 1 2 h2 q (2) +... + 1 p! hp q (p), = 1 h = 1 2 hq(2) +... + 1 p! hp 1 q (p). (52) As q(t) is the exact solution (smooth) of the ODE we used Taylor series expansion for q(t n+1 ) Also, L e (q(t n )) = q (t n ) s(t n, q(t n )) = 0 (53) The leading term of the local truncation error is τ n = 1 2 hq(2) (54) and thus Euler s method is first-order accurate (the error contains h to tyne power 1). The leading term of τ n includes q (2), which could be large. 20 / 55

Definition: accuracy. If the leading term of the local truncation error has the form τ n = Kh p, (55) where K (constant) is independent of h, the method is p-th order accurate. Alternative definition of accuracy. The order of accuracy p > 0 of a method is the largest value of p for which the following inequality holds e n (t, q; h) Ch p, (t, q), 0 < h H, (56) where e n (t, q; h) is the error of the numerical solution, which depends on the output time t and on the time step h. The constant C is independent of h but could depend on H. This is also written thus e n = O(h p ), as h 0. (57) Definition: consistency II. A scheme is said to be consistent if lim h 0 τ n = 0, (58) for which a sufficient condition is that the scheme be at least first order accurate. 21 / 55

Empirical determination of accuracy: convergence rate Definition (56) can be used to empirically determine the order of accuracy p of a method, if one knows an exact solution to compute an error. We write e n (t n, q n ; h) = Ch p, (59) where C is constant and p, the accuracy, is to be determined. We carry out the following steps: 22 / 55

1 Fix an output time T out = b 2 Consider a sequence of meshes with steps and associated integers h 1 > h 2 > > h k > h k+1 > > h K N 1 < N 2 < < N k < N k+1 < < N K with T out = b = a + N l h l for l = 1, 2,..., K. 3 For two successive meshes with steps h k, h k+1 respectively, define e N k (T out, q N k ; h k ) = Ch p k ; en k+1 (T out, q N k+1 ; h k+1 ) = Ch p k+1. (60) 4 Obtain an estimate for the empirical order of accuracy p as p k+1 = ln( en k+1 (T out,q N k+1 ;h k+1 ) ) e N k (T out,q N k ;h k ) ln( h k+1 h k ). (61) One expects p k+1 to converge asymptotically to the actual p, for sufficiently large k. A fixed ratio between successive meshes is often used, e.g. h k /h k+1 = 2. 23 / 55

Characteristic polynomial and stability For the model IVP (39) a one-step method can be written as q n+1 = P (r)q n, r βh, (62) where P (r) is termed the characteristic polynomial. Since the exact solution of the model IVP is q(t) = q 0 e βt, the accuracy of (62) depends on how well P (r) approximates e βt. It can also be shown that perturbations introduced in the initial condition q 0 obey the same equation (62) and therefore if P (r) > 1 errors will grow without bound. This motivates the following Definition: interval of absolute stability. For a one-step method of the form (62) the interval of absolute stability is defined as R = {r βh / P (r) < 1}. (63) Example: Euler method. The Euler method applied to the model IVP (39) can be written as q n+1 = (1 + r)q n, r = βh, (64) 24 / 55

so that P (r) = 1 + r. Application of the absolute stability condition (63) implies the two inequalities 1 1 + r 1. (65) The left inequality 1 1 + r implies 2 r. The right inequality 1 + r 1 implies r 0. Thus we obtain the absolute stability condition 2 r 0, (66) or R = [ 2, 0]. We convert this condition into a restriction on the time step h. We obtain h 2 β. (67) For large values of β this condition could be very restrictive, requiring the use of very small time steps h to reach a desired output time. This restriction is very severe when solving a class of problems called stiff problems. 25 / 55

Runge-Kutta Methods Popular numerical methods for ODEs are Runge-Kutta methods q n+1 = q n + hφ(t n, q n, h) (68) The increment function φ is obtained by appropriate evaluation of the function s(t, q(t)) in the interval [t n, t n+1 ] The attractive feature of Runge-Kutta methods is that one can construct high order methods which are still relatively simple Second-order Runge-Kutta method. One first expresses the increment function as a weighted average of two functions k 1 and k 2 as follows φ(t n, q n, h) = α 1 k 1 + α 2 k 2, k 1 = s(t n, q n ), k 2 = s(t n + α 3 h, q n + α 4 hk 1 ). (69) 26 / 55

α i, i = 1,..., 4 are constants to be found α 1 and α 2 determine the weights for the functions k 1 and k 2, while α 3 and α 4 determine the positions within the time interval [t n, t n+1 ] at which the slope s(t, q(t)) is to be evaluated, in order to obtain the expected accuracy The constants are found by requiring that the sought method reproduces Taylor s method of second order, which after using (45) in (37), becomes q n+1 = q n + hφ(t n, q n, h) φ(t n, q n, h) = s(t n, q n ) + h 2 q(2) (t n ) = s(t n, q n ) + 1 2 h t s(tn, q n ) + 1 2 hs(tn, q n ) q s(tn, q n ). (70) Taylor expanding the function k 2 = s(t n + α 3 h, q n + α 4 hk 1 ) about (t n, q n ) one has k 2 = s(t n + α 3 h, q n + α 4 hk 1 ), = s(t n, q n ) + α 3 h t s(tn, q n ) + α 4 hk 1 q s(tn, q n ). } (71) 27 / 55

Substituting (71) into (69) we obtain α 1 + α 2 = 1, α 2 α 3 = 1/2, α 2 α 4 = 1/2. (72) There are three equations for four unknowns and therefore there is an infinite number of solutions that depend on a single parameter. One possible well-known solution is α 3 = α 4 = 1, α 1 = α 2 = 1/2, (73) which gives the second-order Runge-Kutta method q n+1 = q n + hφ(t n, q n, h), φ(t n, q n, h) = 1 2 (k 1 + k 2 ), k 1 = s(t n, q n ), k 2 = s(t n + h, q n + hk 1 ). (74) 28 / 55

Higher-order Runge-Kutta methods The above procedure can also be used to construct schemes of higher order of accuracy. A third-order Runge-Kutta scheme is q n+1 = q n + hφ(t n, q n, h), φ(t n, q n, h) = 1 9 h(2k 1 + 3k 2 + 4k 3 ), k 1 = s(t n, q n ), (75) k 2 = s(t n + 1 2 h, qn + 1 2 hk 1), k 3 = s(t n + 3 4 h, qn + 3 4 hk 2). A fourth-order Runge-Kutta scheme is q n+1 = q n + hφ(t n, q n, h), φ(t n, q n, h) = 1 6 h(k 1 + 2k 2 + 2k 3 + k 2 ), k 1 = s(t n, q n ), k 2 = s(t n + 1 2 h, qn + 1 2 hk 1), k 3 = s(t n + 1 2 h, qn + 1 2 hk 2), k 4 = s(t n + h, q n + hk 3 ). This scheme is very popular in computational practice. (76) 29 / 55

TVD Runge-Kutta methods This new class of methods of the Runge-Kutta type has been proposed to solve hyperbolic partial differential equations in combination with the semi-discrete approach, whereby space and time discretizations are treated as two separate procedures. Consider an ODE (or a system of ODEs) written in the form q (t) = L(q). (77) A two-stage, second-order TVD Runge-Kutta scheme is q n+1/2 = q n + hl(q n ), q n+1 = 1 2 [qn + (q n+1/2 + hl(q n+1/2 ))]. } (78) The scheme can also be written as the usual one-step formula (37) q n+1 = q n + hφ(t n, q n, h), φ(t n, q n, h) = 1 2 (k 1 + k 2 ), (79) k 1 = L(q n ), k 2 = L(q n + hl(q n )). 30 / 55

A third order TVD Runge-Kutta method is the following three-stage scheme q n+1/3 = q n + hl(q n ), q n+2/3 = 3 4 qn + 1 4 qn+1/3 + 1 4 hl(qn+1/3 ), q n+1 = 1 3 qn + 2 3 qn+2/3 + 2 3 hl(qn+2/3 ). Written as the usual one-step formula (37) the scheme reads q n+1 = q n + hφ(t n, q n, h), φ(t n, q n, h) = 1 6 h(k 1 + k 2 + 4k 3 ), k 1 = L(q n ), k 2 = L(q n + hk 1 ), k 3 = L(q n + 1 4 h(k 1 + k 2 )). (80) (81) 31 / 55

Other methods Other methods can be constructed by adopting a quadrature approach. Consider the IVP ODE: q (t) = s(t, q(t)), t [t n, t n+1 ], (82) IC: q(t n ) = q n. Exact integration of the ODE gives t n+1 q n+1 = q n + s(t, q(t))dt. (83) t n By using appropriate methods to evaluate the integral we obtain familiar numerical methods. Euler explicit. The Euler (explicit) method, studied previously as a special case of Taylor s method, can also be constructed from (83) by evaluating the integrand s(t, q(t)) at time t = t n. The resulting integral leads to q n+1 = q n + hs(t n, q n ). (84) 32 / 55

Euler implicit. If s(t, q(t)) is evaluated at t = t n+1 we obtain q n+1 = q n + hs(t n+1, q n+1 ). (85) Note that now the unknown q n+1 appears also as argument of s(t, q), which in general is a non-linear function. Example. For the model IVP (39) the unknown q n+1 can be obtained explicitly as ( ) 1 q n+1 = q n. (86) 1 βh For a general, non-linear IVP (25) one needs to solve iteratively, at every time step, a non-linear algebraic equation f(x) hs(γ, x) x + δ = 0, (87) where x = q n+1 ; γ and δ are constants. Implicit methods pose no restriction on the size of the time step (unconditional stability), but for time accuracy one may have to limit the size of h in practical calculations. Implicit methods are, however, mandatory for stiff problems. 33 / 55

The trapezoidal method. A second-order accurate implicit method is obtained from (83) by evaluating the integral using the trapezoidal rule. We obtain q n+1 = q n + 1 2 h[s(tn, q n ) + s(t n+1, q n+1 )]. (88) Other methods can also be constructed from (83) by approximating the integrand function s(t, q(t)) by a polynomial P (t) of degree p that interpolates a sequence of p + 1 points with (t k, q k ), k = n p, n p + 1,..., n, (89) P (t k ) = s(t k, q k ), k = n p, n p + 1,..., n. (90) The Adams-Bashforth method. Let us consider the first-degree interpolating the polynomial with p(t) = a + bt, (91) p(t n 1 ) = s(t n 1, q n 1 ) s n 1, p(t n ) = s(t n, q n ) s n. (92) 34 / 55

Evaluating the polynomial at the points t n 1 and t n gives a linear system for the a and b, whose solution is b = sn s n 1 h The integral in (83) gives t n+1, a = s n sn s n 1 t n. (93) h t n (a + bt)dt = ah + 1 2 b(tn+1 t n )(t n+1 + t n ) The final scheme then reads = 1 2 h[3s(tn, q n ) s(t n 1, q n 1 )]. (94) q n+1 = q n + 1 2 h[3s(tn, q n ) s(t n 1, q n 1 )]. (95) This is the Adams-Bashforth method, a three-step, second order method. This is one example of a large family of methods for ODEs, called Multi-Step Methods, which can be constructed by generalizing the above integral formulation. 35 / 55

Choosing the optimal time step Given the total time of a simulation T out (the output time) we have h = (T out a)/n T out = a + Nh, (96) where t = a is the initial time and T out = b is the final time The error at t = T out can be written as e N = q(t out ) q N. (97) But since the computer representation of q N is finite we have instead a value q N, and an associated error ē N. That is q N = q N + ē N, so that the full error is R N = q(t out ) q N ) = q(t out ) q N + q N q N ) e N + ē N. (98) Fig. 2 shows behaviour of errors. e N decreases (linearly) as the mesh is refined. However, ē N increases as the mesh is refined. The total error R N decreases initially as the mesh is refined, up to a point h = h opt, where the error is the minimum. 36 / 55

Further refinement of the mesh increases the error R N, due to the component ē N. The point h = h opt is called the optimal time step. The choice of h = h opt is usually made by performing a series of numerical experiments on model problems with exact solution. We remark that modern numerical software for ODEs use variable (adaptive) time steps. Fig. 2. Illustration of the behaviour of errors as the time step h varies. There is an optimal value of the time step h opt for which the error R n has a minimum. 37 / 55

Methods for Systems of ODEs We consider the IVP for systems of ODEs, that is EDO: Q (t) = S(t, Q(t)), t [a, b] CI: Q(a) = Q 0. The methods for the scalar IVP can be generalized to system IVPs, whereby each equation is a vector equation. For example, the explicit Euler method for a system reads } (99) Q n+1 = Q n + hφ(t n, Q n, h), φ(t n, Q n, h) = S(t n, Q n ), (100) while the fourth-order Runge-Kutta method is Q n+1 = Q n + hφ(t n, Q n, h) φ(t n, Q n, h) = 1 6 [K 1 + 2K 2 + 2K 3 + K 4 ], K 1 = S(t n, Q n ), K 2 = S(t n + 1 2 h, Qn + 1 2 hk 1), K 3 = S(t n + 1 2 h, Qn + 1 2 hk 2), K 4 = S(t n + h, Q n + hk 3 ). (101) 38 / 55

Stiff Problems Stiff problems are a special class of problems for ODEs. These can be characterized by the fact that the solution may be decomposed as follows q(t) = q slow (t) + q fast (t), (102) where q slow (t) changes very slowly with time and q fast (t) changes very rapidly with time; q slow (t) is called the slow component and q fast (t) is called the fast component, also called the transient. The transient usually has the form q fast (t) = e γt, where γ is a negative constant with γ large. It might in fact be possible to identify a point t trans such that for t t trans the solution q(t) is fundamentally given by q slow (t), with q fast (t) being almost negligible. However, from the numerical point of view, the strongest stability restriction could come from the transient q fast (t), whose contribution to the solution, as said, could be virtually negligible. 39 / 55

Examples of stiff problems Stiff problem 1. Consider the IVP The solution is ODE: q (t) = βq, t > 0, β = 25, IC: q(0) = 1. } (103) q(t) = 0 + e 25t, q slow (t) = 0, q fast (t) = e 25t. (104) Note that for t = t trans = 1, q fast (t) 1.3 10 11, that is the solution is virtually that given by the slow component q slow (t) = 0. From the numerical point of view, it is the negligible, transient component the one that poses the strongest stability restriction. For the explicit Euler method, the time step must satisfy the rather severe stability restriction h 2 = 0.08. (105) 25 For example, to complete a simulation with output time t out = 1000 would require at least 12500 time steps. 40 / 55

Stiff problem 2. Consider the IVP ODE: q (t) = βq βsint + cost, β < 0, IC: q(0) = 1. The solution is given by } (106) q(t) = sint + e βt, q slow (t) = sint, q fast (t) = e βt. (107) Exercise. Plot the solution and each component of the solution for t [0, 4π] for β = 25. Stiff problem 3. Let us consider the second-order linear IVP ODE: q (t) + c 1 q (t) + c 2 q(t) = 0, t 0, (108) IC: q(0) = 1, q (0) = 1. The general solution of the ODE is q(t) = Ae β 1t + Be β 2t, β 1 = 1 2 ( c 1 c 2 1 4c 2), β 2 = 1 2 ( c 1 + c 2 1 4c 2). (109) 41 / 55

Imposing the initial condition one obtains q(0) = A + B = 1, q (0) = β 1 A + β 2 B = 1, (110) and the solution for the coefficients A and B is found to be A = β 2 1 β 2 β 1, B = 1 β 1 β 2 β 1. (111) Taking, for instance, c 1 = 26/5, c 2 = 1 one obtains β 1 = 1/5, β 2 = 1/4 and the general solution is q(t) = 5 4 e 1 5 t 1 4 e 5t. (112) Note that the solution contains a component of the form e βt, with β large. Plot the solution and each component of the solution for t [0, 4]. 42 / 55

Remarks on stiff problems In practice one does not know in advance the form of the solution of a given problem involving ODEs and therefore it is not always possible to identify the stiff character of the problem Often, however, the origin of the ODEs can be helpful in this regard. For example, problems involving chemical reactions are usually stiff Mathematically one can determine the stiff character of a problem (99) by analysing the eigenvalues of the Jacobian matrix A = S(t, Q) Q. (113) A practical advise is this: If one does not know the character of the problem, assume it is stiff. 43 / 55

Boundary Value Problems (BVP) We solve the two-point boundary value problem for a second-order ODE ODE: q (t) = s(t, q(t), q } (t)), t (a, b), (114) BCs: q(a) = α, q(b) = β, where α and β are prescribed real values. The second order ODE may be linear or non-linear, depending on the function s. Boundary conditions (BCs) are imposed at both ends of the interval. In a BVP the solution must pass through the points (a, α) and (b, β), as illustrated in Fig. 3. The linear version of the BVP (114) is ODE: q (t) = c(t)q (t) + d(t)q(t) + e(t), t (a, b), BCs: q(a) = α, q(b) = β, (115) where the coefficients c(t), d(t) and e(t) are prescribed functions of time, or constants. } 44 / 55

Theorem: existence and uniqueness of the BVP Theorem: Suppose that the function of three arguments s(x, y, z) in (114) is defined and continuous in the set D = { (x, y, z) : < a x b <, (y, z) R 2}. (116) Assume in addition that: y s(x, y, z) and z s(x, y, z) are also continuous in D y s(x, y, z) > 0, (x, y, z) D there exists a constant K > 0 such that z s(x, y, z) K, (x, y, z) D. Then the two-point BVP (114) has a unique solution. Proof: Omitted. The linear BVP. Sufficient conditions for existence and uniqueness for BVP (115) are: (i) c(t), d(t) and e(t) are continuous on [a, b]; (ii) d(t) > 0 on [a, b] and (iii) e(t) bounded on [a, b]. In what follows we study two types of numerical methods for solving two-point boundary values problems. 45 / 55

Fig. 3. Boundary value problem. The solution is prescribed at the two points t = a and t = b. Fig. 4. Shooting method. One solves the IVP (117) for many values of the parameter g until the numerical solution hits the target β 46 / 55

The Shooting Method The shooting method associates to the original BVP (114) the IVP ODE: q (t) = s(t, q(t), q } (t)), t (a, b), ICs: q(a) = α, q (a) = g, (117) g is an initial guess for the slope of the curve q(t) at t = a. The associated IVP (117) is solved by any of the methods for IVPs studied previously For a given value of g one obtains a numerical solution to the IVP (117) whose value at the end point b will not necessarily be equal to β; see Fig. 3. The shooting method works by shooting many times with different values of g until the numerical solution hits the target β. See Fig. 4. 47 / 55

A practical shooting method First note that for any g k one can compute a corresponding numerical solutions {q (k) i }, i = 1, 2,..., N, to the IVP (117). In this manner we define the numerical function A practical scheme is defined as follows: B(g k ) = q (k) N = β k. (118) Step 1: For given guessed slopes g 1, g 2 compute two solutions to IVP (117), with corresponding end points (g 1, B(g 1 )), (g 2, B(g 2 )). Step 2: Fit a line through P 1 = (g 1, B(g 1 )) and P 2 = (g 2, B(g 2 )), i.e. { } B(g2 ) B(g 1 ) B(g) = B(g 2 ) + (g g 2 ). (119) g 2 g 1 Step 3: A third value g 3 is obtained by requiring B(g 3 ) = β, that is { } g 2 g 1 g 3 = g 2 + (β B(g 2 )). (120) B(g 2 ) B(g 1 ) 48 / 55

Step 4: A general iterative scheme is { } g k g k 1 g k+1 = g k + (β B(g k )), k = 2,..., K. B(g k ) B(g k 1 ) (121) Step 5: The process is stopped if B(g k ) β T OL (122) and the last solution to IVP (117) is accepted as the solution to the original BVP (114). Here T OL > 0 is a small prescribed tolerance, e.g. T OL = 10 6. Remarks: The solution of the BVP (117) {q (K) i }, i = 1, 2,..., N, where K is the index for the last solution process, is the solution of BVP (114). This can be a costly computational procedure. One way to reduce the expense is to start the computations on a coarse mesh, that is small N. Then, as one begins to approach the correct value of β, one can refine the mesh to obtain an accurate final solution. 49 / 55

The Method of Finite Differences for BVPs The continuous domain [a, b] is partitioned into a finite number N of intervals to form a mesh or grid with N + 1 points. Let us consider the regular mesh case of equally spaced points t n with a = t 0 < t 1 < t 2 <... < t n <... < t N = b (123) with t n = a + nh ; n = 0,..., N ; h = b a N. (124) For any function f(t), with t [a, b], one can define point values f n = f(t n ). (125) If f(t) is sufficiently smooth, from Taylor s theorem we can write the truncated series f(t n + h) = f(t n ) + hf (t n ) + 1 2 h2 f (2) (t n ) +... + 1 p! hp f (p) (t n ). (126) Also f(t n h) = f(t n ) hf (t n )+ 1 2 h2 f (2) (t n )+...+( 1) p 1 p! hp f (p) (t n ). (127) 50 / 55

By neglecting terms O(h 2 ) in both series and substructing (127) from (126) we obtain a second-order approximation f (t n ) to the first derivative of f(t) at the point t = t n f (t n ) = f n+1 f n 1. (128) 2h Similarly, a second order approximation to the second-order derivative is f (t n ) = f n+1 2f n + f n 1 h 2. (129) Example: Let us consider the simplified linear BVP ODE: q (t) = d(t)q(t) + e(t), t (a, b), BCs: q(a) = α, q(b) = β. (130) Case 1: N = 2. The mesh has three points t 0, t 1, t 2 and values of q(t) at the boundary points t 0 and t 2 are prescribed. The only unknown value is that at the interior point t 1. The finite difference approximation to the second derivative of q(t) is q (t 1 ) = q 2 2q 1 + q 0 h 2. (131) 51 / 55

Then, the discrete form of the ODE in (130) is q 2 2q 1 + q 0 h 2 = d 1 q 1 + e 1. (132) But the prescribed boundary values are q 0 = α and q 2 = β. Substitution of these gives the solution for the single unknown as q 1 = α + β h2 e 1 2 + h 2 d 1. (133) Case 2: N = 3. Of the four resulting points t 0, t 1, t 2, t 3, values of q(t) at the boundary points t 0 and t 3 are prescribed. The only unknown values are those at t 1 and t 2. Finite difference approximations to the second derivative of q(t) and evaluation of the functions on the right hand side at points t 1 and t 2 give, respectively, and q 2 2q 1 + q 0 h 2 = e 1 + d 1 q 1 (134) q 3 2q 2 + q 1 h 2 = e 2 + d 2 q 2. (135) 52 / 55

These equations, after applying the boundary conditions q 0 = α and q 3 = β, can be written as and (2 + h 2 d 1 )q 1 q 2 = α h 2 e 1 (136) q 1 + (2 + h 2 d 2 )q 2 = β h 2 e 2. (137) This pair of algebraic equations can be written as a linear system AX = B, or [ 2 + h 2 ] [ ] [ d 1 1 q1 α h 1 2 + h 2 = 2 ] e 1 d 2 q 2 β h 2. (138) e 2 The exact solution of this system is and q 1 = (α h2 e 1 )(2 + h 2 e 2 ) + h 2 e 2 β (2 + h 2 d 1 )(2 + h 2 d 2 ) 1 q 2 = (β h2 e 2 )(2 + h 2 e 1 ) + h 2 e 1 α (2 + h 2 d 1 )(2 + h 2 d 2 ) 1 (139). (140) 53 / 55

The general case We apply the finite difference approximation to the ODE at all interior points n = 1,..., N 1 and at the boundary points 0 and N we apply boundary conditions. At the points t 1, t n and t N 1 we obtain the the following algebraic equations (2 + h 2 d 1 )q 1 q 2 = h 2 e 1 + α,... =... q n 2 + (2 + h 2 d n )q n q n+1 = h 2 e n,... =... q N 2 + (2 + h 2 d N 1 )q N 1 = h 2 e N 1 + β. (141) This is a linear system of algebraic equations of the form AX = B with unknowns X = [q 1, q 2,..., q N 1 ]. The coefficient matrix A is a tri-diagonal matrix. The numerical solution of a general tri-diagonal linear system can be obtained by applying Thomas algorithm. 54 / 55

References Boyce W E and DiPrima R C. Elementary differential equations and boundary value problems. Sixth Edition, John Wiley and Sons, 1997 Lambert J D. Numerical methods for ordinary differential systems. The initial value problem. John Wiley and Sons, 1997 Shampine L F, Gladwell I and Thompson S. Solving ODEs with MATLAB. Cambridge University Press, 2003. 55 / 55

Exercises for ODEs 1. For each of the following ordinary differential equations q (t) + q(t) = 0, q(t) = cost, q(t) = sint, q (t) = βq βsint + cost, q(t) = cost, q(t) = sint + e βt, q (t) = 3γ γ 2 t 2 + q 2 (t), q(t) = cost, q(t) = sint, q (2) (t) q(t) = 0, q(t) = e t, q(t) = cosht, t 2 q (t) 3tq (t) + 4q(t) = 0, q(t) = t 2, q(t) = t 2 ln(t), q (4) (t) + 4q (3) (t) + 3q(t) = t, q(t) = t/3, q(t) = e t + t/3, t 2 q (t) + 5tq (t) + 4q(t) = 0, q(t) = t 2, q(t) = t 2 ln(t), t > 0, show that the given functions are solutions (or otherwise) of the corresponding equation. Where appropriate, state the role of the parameters and the valid domain for the independent variable. 2 For each of the following ordinary differential equations t 2 q (t) 3tq (t) + 4q(t) = 0, q (3) (t) + 2e t q (2) (t) + q(t)q (t) = t 4, q (4) (t) + 4q (3) (t) + 3q(t) = t, 55 / 55

(a) state the order of the ODE and (b) state whether the ODE is linear or non-linear. 3. For each first-order ODE of the form q (t) = s(t, q(t)) q (t) = βq(t), β 0, constant, q (t) = 3γ γ 2 t 2 + q 2 (t), q (t) = βq βsint + cost, β < 0, verify that the conditions of existence and uniqueness for the IVP are satisfied. 4. Write each of the following high-order ODEs as a corresponding systems of first-order ODEs (a) q (t) + q(t) = 0, (b) q (t) + αq (t) + βq(t) + γ = 0, (c) q (m) (t) + c 1 q (m 1) (t) + c 2 q (m 2) (t) +... + c m q(t) + c m+1 = 0, c i constants. 5. Consider the model IVP ODE: q (t) = βq(t), t [0, 1], IC: q(0) = 1. } 55 / 55

Assuming β = 1 solve the IVP by hand using a time step of h = 1/2, up to the output time T out = 1, using the following methods (a) the explicit Euler method, (b) the implicit Euler method, (c) the second order Runge-Kutta method, (d) the second order Adams-Bashforth method, (e) the second order TVD Runge-Kutta method. For each of the methods display the numerical results on a table (see Tables 1 and 2, as an example), along with the exact solution, the absolute error and the relative error. Write a few comments on your results. 6. Repeat the above calculations of problem 5 with a time step h = 1/4. Write a few comments on your results. 7. For the methods (a) to (c) in problem 5 (a) calculate the local truncation error for the IVP for the ODE q (t) = s(t, q(t)), (b) find the interval of absolute stability for the IVP for the model ODE q (t) = βq(t), β 0, constant. 55 / 55

8. Consider the Taylor method of order p applied to the model IVP for ODE q (t) = βq(t), β 0, constant. Attempt to find (using approximate methods, plots, etc.) the interval of absolute stability for p = 3,..., 10. Compare the intervals of absolute stability for different values of p and write some comments on your results. 9. Consider the two-point boundary value problem (BVP) ODE: q (t) = s(t, q(t), q (t)) e tq sinq, t [1, 2], BCs: q(1) = 0, q(2) = 0. Verify that this problem has a unique solution (or otherwise) by analyzing the conditions of the theorem of existence and uniqueness. 10. Consider the linear BVP ODE: q (t) = s(t, q(t), q (t)) c(t)q + d(t)q + e(t), t [a, b], BCs: q(a) = α, q(b) = β. Consider now the associated IVPs ODE: q (t) = c(t)q + d(t)q + e(t), t [a, b], ICs: q(a) = α, q (a) = 0. } 55 / 55

and ODE: q (t) = c(t)q + d(t)q, t [a, b], ICs: q(a) = 0, q (a) = 1. Assuming that these two associated IVPs have solutions q 1 (t) and q 2 (t), respectively, show that if q 2 (b) is non-zero then ( ) β q1 (b) q(t) = q 1 (t) + q 2 (t) q 2 (b) is the unique solution to the original BVP. Remark: The technique is called the linear shooting method, whereby it is sufficient to solve two associated linear initial value problems and form a linear combination of the solutions to obtain an approximate solution to the original two-point boundary value problem. 11. Consider the two-point BVP ODE: q (t) = q + 2q + cost, t [0, 1 2 π], ICs: q(0) = 3/10, q( 1 2π) = 1/10. } 55 / 55

(a) Verify that the exact solution of the BVP is q(t) = 1 10 (sint + 3cost). (b) Apply the linear shooting method by solving by hand each linear IVP using the second-order Runge-Kutta method with step 1 4 π. (c) Apply the linear shooting method by solving by hand each linear IVP using the second-order Runge-Kutta method with step 1 8 π. (d) Apply the linear shooting method by solving by hand each linear IVP using the second-order Runge-Kutta method with step 1 16 π. 12. Apply the finite difference method to solve the two-point BVP ODE: q (t) = q + 2q + cost, t [0, 1 2 π], ICs: q(0) = 3/10, q( 1 2π) = 1/10. (a) Discretize [0, 1 2π] with three points (one internal point) and find the explicit solution for q(t) at the point t 1 = 1 4π, by hand. (b) Compare the solution obtained in (a) with that obtained from the linear shooting method, at t 1 = 1 4 π. (c) Discretize [0, 1 2π] with five points (three internal points) and find the explicit solution for q(t) at the points t 1, t 2 and t 3. } 55 / 55

(d) Compare the solution obtained in (c) with that obtained from the linear shooting method with step h = 1 8 π at the points t1, t 2 and t 3. (e) Discretize [0, 1 2π] with N + 1 points (N 1 internal points) and formulate the problem for finding the solution for q(t) at the points t 1, t 2,..., t N 1, in terms of a system of linear algebraic equations of the form AX = B, where A is the coefficient matrix, X is the vector of unknowns and B is a vector of known values, that include the boundary conditions. (f) Write (or use) a computer program to solve the general problem and test the program against the results obtained in (a) and (c) for the appropriate mesh. 13. Apply the finite difference method to solve the two-point BVP ODE: q (t) = c(t)q } + d(t)q + e(t), t [a, b], ICs: q(a) = α, q(b) = β. (a) Discretize [a, b] with N + 1 points (N 1 internal points) and formulate the problem for finding the solution for q(t) at the points t 1, t 2,..., t N 1, in terms of a system of linear algebraic equations of the form AX = B. 55 / 55

(b) Investigate the use of available software to solve the linear system of equations. Consider the structure of the coefficient matrix A. (c) Write (or use) a computer program to solve the general problem and test the program against known results. 55 / 55