Numerical Methods - Initial Value Problems for ODEs

Numerical Methods - Initial Value Problems for ODEs Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 1 / 43

Outline 1 Initial Value Problems & ODEs 2 Single Step Methods Euler s Method Taylor Series Method of Order n Runge-Kutta Method Adaptive Runge-Kutta-Fehlber Method 3 Multistep Methods Adams-Bashforth Explicit Methods Adams-Mouton Implicit Methods Predictor-Corrector Methods 4 Convergence and Stability Convergence Stability Function 5 Higher Order ODE Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 2 / 43

Initial Value Problems & ODEs Definition (Ordinary Differential Equation (ODE)) An ordinary differential equation is an equation that involves one or more derivatives of an univariate function. The solutions for an ODE are differ from each other by a constant. Definition (Initial Value Problem) A solution to the initial value problem (IVP) dy dt = f(t, y), with y(t 0) = y 0 on an interval [t 0, b] is a differential function y = φ(t) such that φ(t 0 ) = y 0 and φ (t) = f(t, φ(t)) for all t [t 0, b] The solution of IVP is unique. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 3 / 43

Well-posed Problem Theorem (Well-posed problem) Suppose that D = {(t, y) a t b and < y < } and that f(t, y) is continuous on D. If f satisfies a Lipschitz condition on D, then the initial value problem y(t) = f(t, y), a t b, y(a) = y 0, has a unique solution. Definition (Lipschitz condition) A function f(t, y) is said to satisfy a Lipschitz condition in the variable y on a set D R 2 if there exist a constant L > 0 such that f(t, y 1 ) f(t, y 2 ) < L y 1 y 2, whenever (t, y 1 ) and (t, y 2 ) are in D. The constant L is called a Lipschitz constant for f. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 4 / 43

Well-posed Problem Theorem Suppose f(t, y) is defined on a convex set D R 2. If there exists a constant L > 0, such that f (t, y) L, for all (t, y) D, y then f satisfies a Lipschitz condition on D in the variable y with Lipschitz constant L. Example Show that there is a unique solution to the initial value problem y (t) = 1 + t sin(yt), 0 t 2, y(0) = 0. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 5 / 43

Euler s Method Euler s method is the simplest numerical method for solving well-posed IVP: dy dt = f(t, y), a t b, y(a) = α. First partition / discretise the interval into N + 1 equally spaced mesh points: a = t 0 < t 1 < t 2 < < t N = b, t i = t 0 + ih, i N and h = (b a)/n. Consider the two adjecent mesh points [t i, t i+1 ], from the Taylor s series, y(t i+1 ) = y(t i + h) = y(t i ) + hy (t i ) + h2 2 y (ξ), ξ [t i, t i+1 ]. Write y i as the approximation to the actual solution y(t i ) and substitute y (t i ) = f(t i, y(t i )), we have the Euler s method iteration rule: y i+1 = y i + hf(t i, y i ). The initial starting point of the Euler s method is given by the initial condition y 0 = y(a) = α. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 8 / 43

Euler s Method The local discretisation error ɛ i = y(t i + h) y(t i ) hf(t i, y(t i )) is O(h 2 ). However, the global discretisation error E i = y(t i ) y i is O(h). Example Solve the IVP y = (t y)/2 with y(0) = 1 over 0 t 3 with Euler method. ANSWER: MATLAB code : nm06_euler_driver.m (Analytic solution is 3e t/2 2 + t) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 9 / 43

Taylor Series Method of Order n Theorem Suppose f(t, y) is continuous and satisfies a Lipschitz condition in variable y, and consider the IVP y = f(t, y), a t b, y(a) = α. For the mesh points t i+1 = t i + h, the Taylor series method approximate the solution y(t i+1 ) with the formula: y i+1 = y 0 + d 1 h + d 2 2! h2 + + d n n! hn, for i = 0, 1, 2,..., N, where d i = y (i) (t) evaluated at t i. Note that the global error for the Taylor series method is O(h n ). Example Solve the IVP y = (t y)/2 with y(0) = 1 over 0 t 3 with Taylor series method of order 2. ANSWER: MATLAB code : nm06_taylor2.m Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 11 / 43

Runge-Kutta Method of Order 2 The methods tried to imitate the Taylor series method without requiring analytic differentiation of the ODE. In Euler method: y i+1 = y i + hf(t i, y i ), the slope f(t, y) is evaluated at the start of the interval t i, ie a forward difference scheme. Intuitively, for better accuracy, we could evaluate f(t, y) at the midpoint by using the Euler method first to obtain y i+h/2, then evaluate f(t i+h/2, y i+h/2 ) to get a symmetrical scheme. Because of the symmetry, the local error is reduced by an order (in the step size) and the method is now a second-order method called Runge-Kutta Method of order 2 (RK2) or the midpoint method. The RK2 algorithm: k 1 = hf(t i, y i ) y i+1/2 = y i + k 1 /2 k 2 = hf(t i+1/2, y i+1/2 ) y i+1 = y i + k 2 Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 13 / 43

Runge-Kutta Method In general the Runge-Kutta method could be written in the form of y i+1 = y i + w 1 k 1 + w 2 k 2 + w 3 k 3 + w 4 k 4... k 1 = hf(t i, y i ) k 2 = hf(t i + a 1 h, y i + b 1 k 1 ) k 3 = hf(t i + a 2 h, y i + b 2 k 1 + b 3 k 2 ) k 4 = hf(t i + a 3 h, y i + b 4 k 2 + b 5 k 2 + b 6 k 3 ). =., where the constants a i and b i are determined by comparing with the corresponding order of Taylor series method. For example, the 2nd order Taylor series method gives: y(t i+1 ) = y(t i ) + hf(t i, y(t i )) + h2 2 f (t i, y(t i )) + h3 3! f (ξ i, y(ξ i )), and we wish to find the corresponding w 1, w 2, a 1 and b 1 for k 1, k 2 and y i+1 = y i + w 1 k 1 + w 2 k 2. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 14 / 43

Runge-Kutta Method From the 2nd order Taylor series method gives: y i+1 = y i + hf(t i, y i ) + h2 2 f (t i, y i ) + O(h 3 ) = y i + hf(t i, y i ) + h2 2 [f t(t i, y i ) + f(t i, y i )f y (t i, y i )] + O(h 3 ). Also, we know k 1 = hf(t i, y i ) and expand k 2 in term of Taylor series k 2 = hf(t i + a 1 h, y i + b 1 k 1 ) = h [f(t i, y i ) + a 1 hf t (t i, y i ) + b 1 k 1 f y (t i, y i )] + O(h 3 ) = h [f(t i, y i ) + a 1 hf t (t i, y i ) + b 1 hf(t i y i )f y (t i, y i )] + O(h 3 ) Substituting k 1 and k 2 into y i+1 = y i + w 1 k 1 + w 2 k 2 get y i+1 = y i + h(w 1 + w 2 )f(t i, y i ) + w 2 h 2 [a 1 f t (t i, y i ) + b 1 f(t i, y i )f y (t i, y i )] Comparing the coefficients we get: w 1 + w 2 = 1, a 1 w 2 = 1 2, b 1w 2 = 1 2. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 15 / 43

Runge-Kutta Method Note the there are 3 equations for the four unknown w 1, w 2, a 1 and b 1, therefore we have one degree of freedom in solving the coefficients. Choose w 1 = 0, w 2 = 1, a 1 = b 1 = 1 2 k 1 = hf(t i, y i ) we have the mid-point method: k 2 = hf(t i + h/2, y i + k 1 /2) y i+1 = y i + k 2 It is possible to choose an optimal value for a 1 such that the error O(h 3 ) is minimized. The value chosen is a 1 = 2 3 and thus w 1 = 1 4, w 2 = 3 4, b 1 = 2 3 : k 1 = hf(t i, y i ) k 2 = hf(t i + 2 3 h, y i + 2 3 k 1) y i+1 = y i + 1 4 k 1 + 3 4 k 2 Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 16 / 43

Runge-Kutta Method of Order 4 Similarly, the Runge-Kutta Method of Order 4 (RK4) algorithm: k 1 = hf(t i, y i ) k 2 = hf(t i + h 2, y i + k 1 2 ) k 3 = hf(t i + h 2, y i + k 2 2 ) k 4 = hf(t i + h, y i + k 3 ) y i+1 = y i + 1 6 (k 1 + 2k 2 + 2k 3 + k 4 ) Note that RK2 has global truncation error of O(h 2 ), while RK4 has a global truncation error of O(h 4 ). RK4 is a common, optimal and reliable numerical integrator, especially if it is used together with an adaptive step-size control. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 17 / 43

Adaptive Runge-Kutta-Fehlberg (RKF45) Method The RKF45 algorithm: k 1 = hf(t i, y i ) k 2 = hf(t i + 1 4 h, y i + 1 4 k 1) k 3 = hf(t i + 3 8 h, y i + 3 32 k 1 + 9 32 k 2) k 4 = hf(t i + 12 13 h, y i + 1932 2197 k 1 7200 2197 k 2 + 7296 2197 k 3) k 5 = hf(t i + h, y i + 439 216 k 1 8k 2 + 3680 513 k 3 845 4104 k 4) k 6 = hf(t i + 1 2 h, y i 8 27 k 1 + 2k 2 3544 2565 k 3 + 1859 4104 k 4 11 40 k 5) ỹ i+1 = y i + 25 216 k 1 + 1408 2565 k 3 + 2197 4104 k 4 1 5 k 5 y i+1 = y i + 16 135 k 1 + 6656 12825 k 3 + 28561 56430 k 4 9 50 k 5 + 2 55 k 6 Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 19 / 43

Adaptive Runge-Kutta-Fehlberg (RKF45) Method The RKF45 algorithm gives two estimates for y(t i ), ie: 4th order estimate : ỹ i+1; and 5th order estimate : y i+1. The difference between the two estimates gives the local truncation error. ie. ɛ = ỹ i+1 y i+1 O(h 5 ). A simple adaptive scheme: Choose an acceptable error ɛ 0. Suppse we did a calculation with step size h c and error ɛ c, then a new step-size that will produce error ɛ 0 is h 0 = h c(ɛ 0/ɛ c) 1/5. Hence, If ɛ c ɛ 0, accept the calculation with current step-size h c, but change the next step size to h 0. If ɛ c > ɛ 0, reject y i+1 and repeat the calculation with step-size h 0. The Matlab command ode45 implement a variant of RK45 with adaptive step control. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 20 / 43

Multistep Methods Multistep methods make use of the information from several previous mesh points to compute the value at the new mesh point. Consider y (t) = f(t, y), integrate over [t i, t i+1 ] and we have: y(t i+1 ) = y(t i ) + ti+1 t i f(t, y(t)) dt. As y(t i+1 ) is unknown, we cannot evaluate the integral explicitly. Instead, we rely on interpolating the integrand with a polynomial. For example, let say we know the value of (t i, y(t i )) and we are approximating f(t, y) with interpolating polynomial of degree 0 (ie a horizontal line), then f(t, y) = f(t i, y(t i )) + (t t i )f (τ i, y(τ i )) where τ i [t i, t i+1 ]. Integrating over [t i, t i+1 ] and let h = t i+1 t i, gives y(t i+1 ) = y(t i ) + hf(t i, y(t i )) + h2 2 f (ξ i, y(ξ)), Which is the one-step Euler method y i+1 = y i + hf(t i, y i ). Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 22 / 43

Adams-Bashforth Explicit Methods Continue along the idea, now we use interpolating polynomial through the two points (t i, y(t i )) and (t i 1, y(t i 1 )) for f(t, y): ti+1 { y(t i+1 ) = y(t i ) + f(t i, y(t i )) + (t t i ) f(t i, y(t i )) f(t i 1, y(t i 1 )) t i h + (t t } i)(t t i 1 ) f (τ i, y(τ i )) dt 2! = y(t i ) + h 2 {3f(t i, y(t i )) f(t i 1, y(t i 1 ))} + 5 12 h3 f (ξ i, y(ξ i )) Two-step Adams-Bashforth method: y i+1 = y i + h 2 [3f(t i, y i ) f(t i 1, y i 1 )] + 5 12 h3 f (ξ i, y(ξ i )) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 24 / 43

Adams-Bashforth Explicit Methods One-step Adams-Bashforth: y i+1 = y i + hf(t i, y i ) + h2 2 f (ξ i, y(ξ i )) Two-step Adams-Bashforth: y i+1 = y i + h 2 [3f(t i, y i ) f(t i 1, y i 1 )] + 5 12 h3 f (ξ i, y(ξ i )) Three-step Adams-Bashforth: y i+1 = y i + h 12 [23f(t i, y i ) 16f(t i 1, y i 1 ) + 5f(t i 2, y i 2 )] + 3 8 h4 f (ξ i, y(ξ i )) Four-step Adams-Bashforth: y i+1 = y i + h 24 [55f(t i, y i ) 59f(t i 1, y i 1 ) + 37f(t i 2, y i 2 ) 9f(t i 3, y i 3 )] + 251 720 h5 f (4) (ξ i, y(ξ i )) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 25 / 43

Adams-Moulton Implicit Methods Now we also include (t i+1, y(t i+1 )) as additional interpolation node, then we have the Implicit formula. For example, use (t i+1, y(t i+1 )) and (t i, y(t i )) ti+1 { y(t i+1 ) = y(t i ) + f(t i, y(t i )) t t i+1 t t i + f(t i+1, y(t i+1 )) t i t i t i+1 t i+1 t i + (t t } i)(t t i 1 ) f (τ i, y(τ i )) dt 2! = y(t i ) + h 2 {f(t i, y(t i )) f(t i+1, y(t i+1 ))} 1 12 h3 f (ξ i, y(ξ i )) One-step Adams-Moulton implicit method: y i+1 = y i + h 2 [f(t i, y i ) f(t i+1, y i+1 )] 1 12 h3 f (ξ i, y(ξ i )) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 27 / 43

Adams-Moulton Implicit Methods One-step Adams-Moulton: y i+1 = y i + h 2 [f(t i+1, y i+1 ) + f(t i, y i )] 1 12 h3 f (ξ i, y(ξ i )) Two-step Adams-Moulton: y i+1 = y i + h 12 [5f(t i+1, y i+1 )+8f(t i, y i ) f(t i 1, y i 1 )] 1 24 h4 f (ξ i, y(ξ i )) Three-step Adams-Moulton: y i+1 = y i + h 24 [9f(t i+1, y i+1 ) + 19f(t i, y i ) 5f(t i 1, y i 1 ) +f(t i 2, y i 2 )] 19 720 h5 f (4) (ξ i, y(ξ i )) Four-step Adams-Moulton: y i+1 = y i + h 720 [251f(t i+1, y i+1 ) + 646f(t i, y i ) 264f(t i 1, y i 1 ) +106f(t i 2, y i 2 ) 19f(t i 3, y i 3 )] 3 160 h6 f (5) (ξ i, y(ξ i )) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 28 / 43

Predictor-Corrector Methods One weakness the Adams-Moulton implicit formula is it not always possible to algebraically re-arrangethe formula to make y(t i+1 ) explicit. However, combine with Adams-Bashforth explicit formula to form a predictor-corrector pairs. The simplest example will be the so-called leapfrog algorithm: Predictor (1-step AB) : p i+1 = y i + hf(t i, y i) Corrector (1-step AM) : y i+1 = y i + h [f(ti+1, pi+1) + f(ti, yi)] 2 Another commonly used method will be the 4th Order Adams-Bashforth-Moulton methods: Predictor : p i+1 = y i + h [55fi 59fi 1 + 37fi 2 9fi 3] 24 Corrector : y i+1 = y i + h [9fi+1 + 19fi 5fi 1 + fi 2], where 2 f i+1 = f(t i+1, p i+1) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 30 / 43

Convergence Stability of ODE scheme depends on the nature of IVP. Eg, Euler scheme diverges for y = λy, y(0) = α, but converges for y = λy. For convergent solution curves, the local errors at each step are reduced over t, and accumulative global error may be less than the sum of the local errors. Theorem For an initial value problem: y = f(t, y), y(0) = α if f y > δ for some positive δ, then the solution curve diverges. if f y < δ, then the solution curve converges. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 33 / 43

Stability Even if the IVP is convergent, it could still be unstable due to large h. Example Consider IVP y = (t y)/2, y(0) = 1. We know that f y (t) = 1/2 for all t [0, 50], thus the Euler method should be convergent. However, the numerical solution y(t) by using Euler method for for h = 2.5 is convergent but diverge for h = 5. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 35 / 43

Stability Consider a linear (or linearized) ODE: y = λy, and discretized, say by Euler method (of course could be other method). Then, we have y i+1 = y i hλy i. The absolute stability function is defined as Q(hλ) = y i+1 y i. In the case for Euler method, we have Q(hλ) = (1 hλ). If the amplification factor Q(hλ) < 1, then we are guaranteed that the sequence {y i } will not grow without bound, and hence stable. Assuming that λ is real, then for the Euler method to be stable we need 1 < 1 λh < 1 or 0 < λh < 2, ie h < 2/λ in order for the Euler method to be stable. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 36 / 43

Stability The following figure shows an IVP (y = 10y, y(0) = 1) solved by Euler method with different values of step size h. Note that the numerical solution curves become unstable when h 2/c = 0.10. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 37 / 43

Higher Order ODE & System of ODEs Higher order ODE can be solved numerically by turning into a system of first order ODEs. Consider IVP of order n: y (n) = f(t, y, y,..., y (n 1) ), y(0) = α 0, y (0) = α 1,..., y (n 1) = α n 1. Define new variables x 1, x 2,..., x n : x 1 = y, x 2 = y,..., x n = y (n 1). Now the IVP is equivalent to x 1 = x 2, x 1 (0) = α 0 x 2 = x 3, x 2 (0) = α 1. x n = f(t, x 1, x 2,..., x n ), x n = α n 1 In vector notation: X = F(t, X), X(0) = A, where X = [x 1, x 2,..., x n ] T, F = [x 2, x 3,..., f] T and A = [α 0, α 1,..., α n 1 ]. Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 39 / 43

System of First Order ODEs Solve the system of first order ODEs numerically are very much like solving a single first order ODEs. For example, consider to solve X (t) = F(t, X), X(0) = A with a Runge-Kutta method of order 4, we have Discretise the time interval [a, b] into n subdivisions with h = (b a)/n. RK4 iteration formula: X i+1 = X i + h [k1 + 2k2 + 2k3 + k4], where 6 k 1 = hf(t, X) k 2 = hf(t + 1 2 h, X + 1 2 k1) k 3 = hf(t + 1 2 h, X + 1 2 k2) k 4 = hf(t + h, X + k 3) Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 40 / 43

System of ODEs (Example) Example Solve y = 4y, y(0) = 0, y (0) = 2. ANSWER: MATLAB code : nm06_system.m Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 41 / 43

System of ODEs (Example) Example (Lorenz system) Solve the Lorenz system: x = σ(y x) y = x(ρ z) y z = xy βz with the values σ = 3, ρ = 26.5 and β = 1. ANSWER: MATLAB code : nm06_lorenz.m Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 42 / 43

THE END Y. K. Goh (UTAR) Numerical Methods - Initial Value Problems for ODEs 2013 43 / 43