Prt IB Numericl Anlysis Bsed on lectures by G. Moore Notes tken by Dexter Chu Lent 2016 These notes re not endorsed by the lecturers, nd I hve modified them (often significntly) fter lectures. They re nowhere ner ccurte representtions of wht ws ctully lectured, nd in prticulr, ll errors re lmost surely mine. Polynomil pproximtion Interpoltion by polynomils. Divided differences of functions nd reltions to derivtives. Orthogonl polynomils nd their recurrence reltions. Lest squres pproximtion by polynomils. Gussin qudrture formule. Peno kernel theorem nd pplictions. [6] Computtion of ordinry differentil equtions Euler s method nd proof of convergence. Multistep methods, including order, the root condition nd the concept of convergence. Runge-Kutt schemes. Stiff equtions nd A-stbility. [5] Systems of equtions nd lest squres clcultions LU tringulr fctoriztion of mtrices. Reltion to Gussin elimintion. Column pivoting. Fctoriztions of symmetric nd bnd mtrices. The Newton-Rphson method for systems of non-liner lgebric equtions. QR fctoriztion of rectngulr mtrices by Grm-Schmidt, Givens nd Householder techniques. Appliction to liner lest squres clcultions. [5] 1
Contents IB Numericl Anlysis Contents 0 Introduction 3 1 Polynomil interpoltion 4 1.1 The interpoltion problem...................... 4 1.2 The Lgrnge formul........................ 4 1.3 The Newton formul......................... 5 1.4 A useful property of divided differences.............. 7 1.5 Error bounds for polynomil interpoltion............. 8 2 Orthogonl polynomils 12 2.1 Sclr product............................ 12 2.2 Orthogonl polynomils....................... 13 2.3 Three-term recurrence reltion................... 14 2.4 Exmples............................... 15 2.5 Lest-squres polynomil pproximtion.............. 16 3 Approximtion of liner functionls 18 3.1 Liner functionls.......................... 18 3.2 Gussin qudrture......................... 19 4 Expressing errors in terms of derivtives 22 5 Ordinry differentil equtions 26 5.1 Introduction.............................. 26 5.2 One-step methods.......................... 27 5.3 Multi-step methods.......................... 30 5.4 Runge-Kutt methods........................ 34 6 Stiff equtions 37 6.1 Introduction.............................. 37 6.2 Liner stbility............................ 37 7 Implementtion of ODE methods 40 7.1 Locl error estimtion........................ 40 7.2 Solving for implicit methods..................... 41 8 Numericl liner lgebr 42 8.1 Tringulr mtrices.......................... 42 8.2 LU fctoriztion........................... 43 8.3 A = LU for specil A........................ 46 9 Liner lest squres 50 2
0 Introduction IB Numericl Anlysis 0 Introduction Numericl nlysis is the study of lgorithms. There re mny problems we would like lgorithms to solve. In this course, we will tckle the problems of polynomil pproximtion, solving ODEs nd solving liner equtions. These re ll problems tht frequently rise when we do (pplied) mths. In generl, there re two things we re concerned with ccurcy nd speed. Accurcy is prticulrly importnt in the cses of polynomil pproximtion nd solving ODEs, since we re trying pproximte things. We would like to mke good pproximtions of the solution with reltively little work. On the other hnd, in the cse of solving liner equtions, we re more concerned with speed our solutions will be exct (up to numericl errors due to finite precision of clcultions), but we would like to solve it quickly. We might hve to del with huge systems, nd we don t wnt the computtion time to grow too quickly. In the pst, this ws n importnt subject, since they hd no computers. The lgorithms hd to be implemented by hnd. It ws thus very importnt to find some prcticl nd efficient methods of computing things, or else it would tke them forever to clculte wht they wnted. So we wnted quick lgorithms tht give resonbly ccurte results. Nowdys, this is still n importnt subject. While we hve computers tht re much fster t computtion, we still wnt our progrms to be fst. We would lso wnt to get relly ccurte results, since we might be using them to, sy, send our rocket to the Moon. Moreover, with more computtionl power, we might scrifice efficiency for some other desirble properties. For exmple, if we re solving for the trjectory of prticle, we might wnt the solution to stisfy the conservtion of energy. This would require some much more complicted nd slower lgorithms tht no one would hve considered in the pst. Nowdys, with computers, these lgorithms become more fesible, nd re becoming incresingly more populr. 3
1 Polynomil interpoltion IB Numericl Anlysis 1 Polynomil interpoltion Polynomils re nice. Writing down polynomil of degree n involves only n + 1 numbers. They re esy to evlute, integrte nd differentite. So it would be nice if we cn pproximte things with polynomils. For simplicity, we will only del with rel polynomils. Nottion. We write P n [x] for the rel liner vector spce of polynomils (with rel coefficients) hving degree n or less. It is esy to show tht dim(p n [x]) = n + 1. 1.1 The interpoltion problem The ide of polynomil interpoltion is tht we re given n + 1 distinct interpoltion points {x i } n i=0 R, nd n + 1 dt vlues {f i} n i=0 R. The objective is to find p P n [x] such tht p(x i ) = f i for i = 0,, n. In other words, we wnt to fit polynomil through the points (x i, f i ). There re mny situtions where this my come up. For exmple, we my be given n + 1 ctul dt points, nd we wnt to fit polynomil through the points. Alterntively, we might hve complicted function f, nd wnt to pproximte it with polynomil p such tht p nd f gree on t lest tht n + 1 points. The nive wy of looking t this is tht we try polynomil nd then solve the system of equtions p(x) = n x n + n 1 x n 1 + + 0, f i = p(x i ) = n x n i + n 1 x n 1 i + + 0. This is perfectly respectble system of n + 1 equtions in n + 1 unknowns. From liner lgebr, we know tht in generl, such system is not gurnteed to hve solution, nd if the solution exists, it is not gurnteed to be unique. Tht ws not helpful. So our first gol is to show tht in the cse of polynomil interpoltion, the solution exists nd is unique. 1.2 The Lgrnge formul It turns out the problem is not too hrd. You cn probbly figure it out yourself if you lock yourself in room for few dys (or hours). In fct, we will come up with two solutions to the problem. The first is vi the Lgrnge crdinl polynomils. These re simple to stte, nd it is obvious tht these work very well. However, prcticlly, this is not the best wy to solve the problem, nd we will not tlk bout them much. Insted, we use this solution s proof of existence of polynomil interpoltions. We will then develop our second method on the ssumption tht polynomil interpoltions exist, nd find better wy of computing them. 4
1 Polynomil interpoltion IB Numericl Anlysis Definition (Lgrnge crdinl polynomils). The Lgrnge crdinl polynomils with respect to the interpoltion points {x i } n i=0 re, for k = 0,, n, l k (x) = n i=0,i k x x i x k x i. Note tht these polynomils hve degree exctly n. The significnce of these polynomils is we hve l k (x i ) = 0 for i k, nd l k (x k ) = 1. In other words, we hve l k (x j ) = δ jk. This is obvious from definition. With these crdinl polynomils, we cn immeditely write down solution to the interpoltion problem. Theorem. The interpoltion problem hs exctly one solution. Proof. We define p P n [x] by Evluting t x i gives p(x j ) = p(x) = n f k l k (x). k=0 n f k l k (x j ) = k=0 n f k δ jk = f j. So we get existence. For uniqueness, suppose p, q P n [x] re solutions. Then the difference r = p q P n [x] stisfies r(x j ) = 0 for ll j, i.e. it hs n + 1 roots. However, non-zero polynomil of degree n cn hve t most n roots. So in fct p q is zero, i.e. p = q. While this works, it is not idel. If we one dy decide we should dd one more interpoltion point, we would hve to recompute ll the crdinl polynomils, nd tht is not fun. Idelly, we would like some wy to reuse our previous computtions when we hve new interpoltion points. 1.3 The Newton formul The ide of Newton s formul is s follows for k = 0,, n, we write p k P k [x] for the polynomil tht stisfies k=0 p k (x i ) = f i for i = 0,, k. This is the unique degree-k polynomil tht stisfies the first k + 1 conditions, whose existence (nd uniqueness) is gurnteed by the previous section. Then we cn write p(x) = p n (x) = p 0 (x) + (p 1 (x) p 0 (x)) + + (p n (x) p n 1 (x)). Hence we re done if we hve n efficient wy of finding the differences p k p k 1. 5
1 Polynomil interpoltion IB Numericl Anlysis We know tht p k nd p k 1 gree on x 0,, x k 1. So p k p k 1 evlutes to 0 t those points, nd we must hve k 1 p k (x) p k 1 (x) = A k (x x i ), i=0 for some A k yet to be found out. Then we cn write p(x) = p n (x) = A 0 + n k=1 (x x i ). A k k 1 This formul hs the dvntge tht it is built up grdully from the interpoltion points one-by-one. If we stop the sum t ny point, we hve obtined the polynomil tht interpoltes the dt for the first k points (for some k). Conversely, if we hve new dt point, we just need to dd new term, insted of re-computing everything. All tht remins is to find the coefficients A k. For k = 0, we know A 0 is the unique constnt polynomil tht interpoltes the point t x 0, i.e. A 0 = f 0. For the others, we note tht in the formul for p k p k 1, we find tht A k is the leding coefficient of x k. But p k 1 (x) hs no degree k term. So A k must be the leding coefficient of p k. So we hve reduced our problem to finding the leding coefficients of p k. The solution to this is known s the Newton divided differences. We first invent new nottion: A k = f[x 0,, x k ]. Note tht these coefficients depend only on the first k interpoltion points. Moreover, since the lbelling of the points x 0,, x k is rbitrry, we don t hve to strt with x 0. In generl, the coefficient f[x j,, x k ] is the leding coefficient of the unique q P k j [x] such tht q(x i ) = f i for i = j,, k. While we do not hve n explicit formul for wht these coefficients re, we cn come up with recurrence reltion for these coefficients. Theorem (Recurrence reltion for Newton divided differences). For 0 j < k n, we hve i=0 f[x j,, x k ] = f[x j+1,, x k ] f[x j,, x k 1 ] x k x j. Proof. The key to proving this is to relte the interpolting polynomils. Let q 0, q 1 P k j 1 [x] nd q 2 P k j stisfy q 0 (x i ) = f i i = j,, k 1 q 1 (x i ) = f i q 2 (x i ) = f i i = j + 1,, k i = j,, k We now clim tht q 2 (x) = x x j x k x j q 1 (x) + x k x x k x j q 0 (x). 6
1 Polynomil interpoltion IB Numericl Anlysis We cn check directly tht the expression on the right correctly interpoltes the points x i for i = j,, k. By uniqueness, the two expressions gree. Since f[x j,, x k ], f[x j+1,, x k ] nd f[x j,, x k 1 ] re the leding coefficients of q 2, q 1, q 0 respectively, the result follows. Thus the fmous Newton divided difference tble cn be constructed x i f i f[, ] f[,, ] f[,, ] x 0 f[x 0 ] f[x 0, x 1 ] x 1 f[x 1 ] f[x 0, x 1, x 2 ]. f[x 1, x 2 ].. x 2 f[x 2 ] f[x 2, x 3, x 4 ] f[x 0, x 1,, x n ] f[x 2, x 3 ]... x 3 f[x 3 ]........ x n f[x n ] From the first n columns, we cn find the n + 1th column using the recurrence reltion bove. The vlues of A k cn then be found t the top digonl, nd this is ll we relly need. However, to compute this digonl, we will need to compute everything in the tble. In prctice, we often need not find the ctul interpolting polynomil. If we just wnt to evlute p(ˆx) t some new point ˆx using the divided tble, we cn use Horner s scheme, given by S <- f[x 0,..., x n ] for k = n - 1,..., 0 S <- (^x - x k )S + f[x 0,..., x k ] end This only tkes O(n) opertions. If n extr dt point {x n+1, f n+1 } is dded, then we only hve to compute n extr digonl f[x k,, x n+1 ] for k = n,, 0 in the divided difference tble to obtin the new coefficient, nd the old results cn be reused. This requires O(n) opertions. This is less strightforwrd for Lgrnge s method. 1.4 A useful property of divided differences In the next couple of sections, we re interested in the error of polynomil interpoltion. Suppose the dt points come from f i = f(x i ) for some complicted f we wnt to pproximte, nd we interpolte f t n dt points {x i } n i=1 with p n. How does the error e n (x) = f(x) p n (x) depend on n nd the choice of interpoltion points? At the interpoltion point, the error is necessrily 0, by definition of interpoltion. However, this does not tell us nything bout the error elsewhere. Our ultimte objective is to bound the error e n by suitble multiples of the derivtives of f. We will do this in two steps. We first relte the derivtives to the divided differences, nd then relte the error to the divided differences. The first prt is n esy result bsed on the following purely clculus lemm. 7
1 Polynomil interpoltion IB Numericl Anlysis Lemm. Let g C m [, b] hve continuous mth derivtive. Suppose g is zero t m + l distinct points. Then g (m) hs t lest l distinct zeros in [, b]. Proof. This is repeted ppliction of Rolle s theorem. We know tht between every two zeros of g, there is t lest one zero of g C m 1 [, b]. So by differentiting once, we hve lost t most 1 zeros. So fter differentiting m times, g (m) hs lost t most m zeros. So it still hs t lest l zeros. Theorem. Let {x i } n i=0 [, b] nd f Cn [, b]. Then there exists some ξ (, b) such tht f[x 0,, x n ] = 1 n! f (n) (ξ). Proof. Consider e = f p n C n [, b]. This hs t lest n + 1 distinct zeros in [, b]. So by the lemm, e (n) = f (n) p (n) n must vnish t some ξ (, b). But then p (n) n = n!f[x 0,, x n ] constntly. So the result follows. 1.5 Error bounds for polynomil interpoltion The ctul error bound is not too difficult s well. It turns out the error e = f p n is like the next term in the Newton s formul. This vgue sentence is mde precise in the following theorem: [, b] nd f C[, b]. Let x [, b] be non- Theorem. Assume {x i } n i=0 interpoltion point. Then where e n ( x) = f[x 0, x 1,, x n, x]ω( x), ω(x) = n (x x i ). i=0 Note tht we forbid the cse where x is n interpoltion point, since it is not cler wht the expression f[x 0, x 1,, x n, x] mens. However, if x is n interpoltion point, then both e n ( x) nd ω( x) re zero, so there isn t much to sy. Proof. We think of x = x n+1 s new interpoltion point so tht p n+1 (x) p n (x) = f[x 0,, x n, x]ω(x) for ll x R. In prticulr, putting x = x, we hve p n+1 ( x) = f( x), nd we get the result. Combining the two results, we find Theorem. If in ddition f C n+1 [, b], then for ech x [, b], we cn find ξ x (, b) such tht e n (x) = 1 (n + 1)! f (n+1) (ξ x )ω(x) Proof. The sttement is trivil if x is n interpoltion point pick rbitrry ξ x, nd both sides re zero. Otherwise, this follows directly from the lst two theorems. 8
1 Polynomil interpoltion IB Numericl Anlysis This is n exct result, which is not too useful, since there is no esy constructive wy of finding wht ξ x should be. Insted, we usully go for bound. We introduce the mx norm This gives the more useful bound Corollry. For ll x [, b], we hve f(x) p n (x) g = mx t [,b] g(t). 1 (n + 1)! f (n+1) ω(x) Assuming our function f is fixed, this error bound depends only on ω(x), which depends on our choice of interpoltion points. So cn we minimize ω(x) in some sense by picking some clever interpoltion points = {x i } n i=0? Here we will hve n fixed. So insted, we put s the subscript. We cn write our bound s 1 f p (n + 1)! f (n+1) ω. So the objective is to find tht minimizes ω. For the moment, we focus on the specil cse where the intervl is [ 1, 1]. The generl solution cn be obtined by n esy chnge of vrible. For some mgicl resons tht hopefully will become cler soon, the optiml choice of comes from the Chebyshev polynomils. Definition (Chebyshev polynomil). The Chebyshev polynomil of degree n on [ 1, 1] is defined by T n (x) = cos(nθ), where x = cos θ with θ [0, π]. So given n x, we find the unique θ tht stisfies x = cos θ, nd then find cos(nθ). This is in fct polynomil in disguise, since from trigonometric identities, we know cos(nθ) cn be expnded s polynomil in cos θ up to degree n. Two key properties of T n on [ 1, 1] re (i) The mximum bsolute vlue is obtined t ( ) πk X k = cos n for k = 0,, n with T n (X k ) = ( 1) k. (ii) This hs n distinct zeros t ( ) 2k 1 x k = cos 2n π. for k = 1,, n. When plotted out, the polynomils look like this: 9
1 Polynomil interpoltion IB Numericl Anlysis T 4 (x) 1 1 1 x 1 All tht relly mtters bout the Chebyshev polynomils is tht the mximum is obtined t n + 1 distinct points with lternting sign. The exct form of the polynomil is not relly importnt. Notice there is n intentionl clsh between the use of x k s the zeros nd x k s the interpoltion points we will show these re indeed the optiml interpoltion points. We first prove convenient recurrence reltion for the Chebyshev polynomils: Lemm (3-term recurrence reltion). The Chebyshev polynomils stisfy the recurrence reltions T n+1 (x) = 2xT n (x) T n 1 (x) with initil conditions Proof. T 0 (x) = 1, T 1 (x) = x. cos((n + 1)θ) + cos((n 1)θ) = 2 cos θ cos(nθ). This recurrence reltion cn be useful for mny things, but for our purposes, we only use it to show tht the leding coefficient of T n is 2 n 1 (for n 1). Theorem (Miniml property for n 1). On [ 1, 1], mong ll polynomils 1 p P n [x] with leding coefficient 1, 2 T n 1 n minimizes p. Thus, the 1 minimum vlue is 2. n 1 Proof. We proceed by contrdiction. Suppose there is polynomil q n P n with leding coefficient 1 such tht q n < 1 2 n 1. Define new polynomil r = 1 2 n 1 T n q n. This is, by ssumption, non-zero. Since both the polynomils hve leding coefficient 1, the difference must 1 hve degree t most n 1, i.e. r P n 1 [x]. Since 2 T n 1 n (X k ) = ± 1 2, nd n 1 q n (X n ) < 1 2 by ssumption, r lterntes in sign between these n + 1 points. n 1 But then by the intermedite vlue theorem, r hs to hve t lest n zeros. This is contrdiction, since r hs degree n 1, nd cnnot be zero. 10
1 Polynomil interpoltion IB Numericl Anlysis Corollry. Consider w = n (x x i ) P n+1 [x] i=0 for ny distinct points = {x i } n i=0 [ 1, 1]. Then min ω = 1 2 n. This minimum is chieved by picking the interpoltion points to be the zeros of T n+1, nmely ( ) 2k + 1 x k = cos 2n + 2 π, k = 0,, n. Theorem. For f C n+1 [ 1, 1], the Chebyshev choice of interpoltion points gives f p n 1 1 2 n (n + 1)! f (n+1). Suppose f hs s mny continuous derivtives s we wnt. Then s we increse n, wht hppens to the error bounds? The coefficients involve dividing by n exponentil nd fctoril. Hence s long s the higher derivtives of f don t blow up too bdly, in generl, the error will tend to zero s n, which mkes sense. The lst two results cn be esily generlized to rbitrry intervls [, b], nd this is left s n exercise for the reder. 11
2 Orthogonl polynomils IB Numericl Anlysis 2 Orthogonl polynomils It turns out the Chebyshev polynomils is just n exmple of more generl clss of polynomils, known s orthogonl polynomils. As in liner lgebr, we cn define sclr product on the spce of polynomils, nd then find bsis of orthogonl polynomils of the vector spce under this sclr product. We shll show tht ech set of orthogonl polynomils hs three-term recurrence reltion, just like the Chebyshev polynomils. 2.1 Sclr product The sclr products we re interested in would be generliztion of the usul sclr product on Eucliden spce, x, y = n x i y i. We wnt to generlize this to vector spces of functions nd polynomils. We will not provide forml definition of vector spces nd sclr products on n bstrct vector spce. Insted, we will just provide some exmples of commonly used ones. Exmple. i=1 (i) Let V = C s [, b], where [, b] is finite intervl nd s 0. Pick weight function w(x) C(, b) such tht w(x) > 0 for ll x (, b), nd w is integrble over [, b]. In prticulr, we llow w to vnish t the end points, or blow up mildly such tht it is still integrble. We then define the inner product to be f, g = b w(x)f(x)d(x) dx. (ii) We cn llow [, b] to be infinite, e.g. [0, ) or even (, ), but we hve to be more creful. We first define f, g = b w(x)f(x)g(x) dx s before, but we now need more conditions. We require tht b w(x)xn dx to exist for ll n 0, since we wnt to llow polynomils in our vector spce. For exmple, w(x) = e x on [0, ), works, or w(x) = e x2 on (, ). These re sclr products for P n [x] for n 0, but we cnnot extend this definition to ll smooth functions since they might blow up too fst t infinity. We will not go into the technicl detils, since we re only interested in polynomils, nd knowing it works for polynomils suffices. (iii) We cn lso hve discrete inner product, defined by f, g = m w j f(ξ j )g(ξ j ) j=1 12
2 Orthogonl polynomils IB Numericl Anlysis with {ξ j } m j=1 distinct points nd {w j} m j=1 > 0. Now we hve to restrict ourselves lot. This is sclr product for V = P m 1 [x], but not for higher degrees, since sclr product should stisfy f, f > 0 for f 0. In prticulr, we cnnot extend this to ll smooth functions. With n inner product, we cn define orthogonlity. Definition (Orthogonlilty). Given vector spce V nd n inner product,, two vectors f, g V re orthogonl if f, g = 0. 2.2 Orthogonl polynomils Definition (Orthogonl polynomil). Given vector spce V of polynomils nd inner product,, we sy p n P n [x] is the nth orthogonl polynomil if In prticulr, p n, p m = 0 for n m. p n, q = 0 for ll q P n 1 [x]. We sid the orthogonl polynomil, but we need to mke sure such polynomil hs to be unique. It is clerly not unique, since if p n stisfies these reltions, then so does λp n for ll λ 0. For uniqueness, we need to impose some scling. We usully do so by requiring the leding polynomil to be 1, i.e. it is monic. Definition (Monic polynomil). A polynomil p P n [x] is monic if the coefficient of x n is 1. In prctice, most fmous trditionl polynomils re not monic. They hve different scling imposed. Still, s long s we hve some scling, we will hve uniqueness. We will not mess with other sclings, nd stick to requiring them to be monic since this is useful for proving things. Theorem. Given vector spce V of functions nd n inner product,, there exists unique monic orthogonl polynomil for ech degree n 0. In ddition, {p k } n k=0 form bsis for P n[x]. Proof. This is big induction proof over both prts of the theorem. We induct over n. For the bse cse, we pick p 0 (x) = 1, which is the only degree-zero monic polynomil. Now suppose we lredy hve {p n } n k=0 stisfying the induction hypothesis. Now pick ny monic q n+1 P n+1 [x], e.g. x n+1. We now construct p n+1 from q n+1 by the Grm-Schmidt process. We define p n+1 = q n+1 n k=0 This is gin monic since q n+1 is, nd we hve p n+1, p m = 0 q n+1, p k p k, p k p k. for ll m n, nd hence p n+1, p = 0 for ll p P n [x] = p 0,, p n. 13
2 Orthogonl polynomils IB Numericl Anlysis To obtin uniqueness, ssume both p n+1, ˆp n+1 P n+1 [x] re both monic orthogonl polynomils. Then r = p n+1 ˆp n+1 P n [x]. So r, r = r, p n+1 ˆp n+1 = r, p n+1 r, ˆp n+1 = 0 0 = 0. So r = 0. So p n+1 = ˆp n 1. Finlly, we hve to show tht p 0,, p n+1 form bsis for P n+1 [x]. Now note tht every p P n+1 [x] cn be written uniquely s p = cp n+1 + q, where q P n [x]. But {p k } n k=0 is bsis for P n[x]. So q cn be uniquely decomposed s liner combintion of p 0,, p n. Alterntively, this follows from the fct tht ny set of orthogonl vectors must be linerly independent, nd since there re n + 2 of these vectors nd P n+1 [x] hs dimension n + 2, they must be bsis. In prctice, following the proof nively is not the best wy of producing the new p n+1. Insted, we cn reduce lot of our work by mking clever choice of q n+1. 2.3 Three-term recurrence reltion Recll tht for the Chebyshev polynomils, we obtined three-term recurrence reltion for them using specil properties of the cosine. It turns out these recurrence reltions exist in generl for orthogonl polynomils. We strt by picking q n+1 = xp n in the previous proof. We now use the fct tht xf, g = f, xg. This is not necessrily true for rbitrry inner products, but for most sensible inner products we will meet in this course, this is true. In prticulr, it is clerly true for inner products of the form f, g = w(x)f(x)g(x) dx. Assuming this, we obtin the following theorem. Theorem. Monic orthogonl polynomils re generted by with initil conditions where p k+1 (x) = (x α k )p k (x) β k p k 1 (x) p 0 = 1, p 1 (x) = (x α 0 )p 0, α k = xp k, p k p k, p k, β k = p k, p k p k 1, p k 1. Proof. By inspection, the p 1 given is monic nd stisfies p 1, p 0 = 0. 14
2 Orthogonl polynomils IB Numericl Anlysis Using q n+1 = xp n in the Grm-Schmidt process gives p n+1 = xp n p n+1 = xp n n k=0 n k=0 xp n, p k p k, p k p k p n, xp k p k, p k p k We notice tht p n, xp k nd vnishes whenever xp k hs degree less thn n. So we re left with = xp n xp n, p n p n, p n p n p n, xp n 1 p n 1, p n 1 p n 1 = (x α n )p n p n, xp n 1 p n 1, p n 1 p n 1. Now we notice tht xp n 1 is monic polynomil of degree n so we cn write this s xp n 1 = p n + q. Thus p n, xp n 1 = p n, p n + q = p n, p n. Hence the coefficient of p n 1 is indeed the β we defined. 2.4 Exmples The four fmous exmples re the Legendre polynomils, Chebyshev polynomils, Lguerre polynomils nd Hermite polynomils. We first look t how the Chebyshev polynomils fit into this frmework. Chebyshev is bsed on the sclr product defined by f, g = 1 1 1 f(x)g(x) dx. 1 x 2 Note tht the weight function blows up mildly t the end, but this is fine since it is still integrble. This links up with T n (x) = cos(nθ) for x = cos θ vi the usul trigonometric substitution. We hve T n, T m = = π 0 π 0 1 cos(nθ) cos(mθ) sin θ dθ 1 cos2 θ cos(nθ) cos(mθ) dθ = 0 if m n. The other orthogonl polynomils come from sclr products of the form f, g = s described in the tble below: b w(x)f(x)g(x) dx, 15
2 Orthogonl polynomils IB Numericl Anlysis Type Rnge Weight Legendre [ 1, 1] 1 Chebyshev [ 1, 1] 1 1 x 2 Lguerre [0, ) e x Hermite (, ) e x2 2.5 Lest-squres polynomil pproximtion If we wnt to pproximte function with polynomil, polynomil interpoltion might not be the best ide, since ll we do is mke sure the polynomil grees with f t certin points, nd then hope it is good pproximtion elsewhere. Insted, the ide is to choose polynomil p in P n [x] tht minimizes the error. Wht exctly do we men by minimizing the error? The error is defined s the function f p. So given n pproprite inner product on the vector spce of continuous functions, we wnt to minimize This is usully of the form f p, f p = f p 2 = f p, f p. b w(x)[f(x) p(x)] 2 dx, but we cn lso use lterntive inner products such s f p, f p = m w j [f(ξ i ) p(ξ i )] 2. j=1 Unlike polynomil interpoltion, there is no gurntee tht the pproximtion grees with the function nywhere. Unlike polynomil interpoltion, there is some gurntee tht the totl error is smll (or s smll s we cn get, by definition). In prticulr, if f is continuous, then the Weierstrss pproximtion theorem tells us the totl error must eventully vnish. Unsurprisingly, the solution involves the use of the orthogonl polynomils with respect to the corresponding inner products. Theorem. If {p n } n k=0 re orthogonl polynomils with respect to,, then the choice of c k such tht n p = c k p k minimizes f p 2 is given by nd the formul for the error is k=0 c k = f, p k p k 2, f p 2 = f 2 n k=0 f, p k 2 p k 2. 16
2 Orthogonl polynomils IB Numericl Anlysis Note tht the solution decouples, in the sense tht c k depends only on f nd p k. If we wnt to tke one more term, we just need to compute n extr term, nd not redo our previous work. Also, we notice tht the formul for the error is positive term f 2 subtrcting lot of squres. As we increse n, we subtrct more squres, nd the error decreses. If we re lucky, the error tends to 0 s we tke n. Even though we might not know how mny terms we need in order to get the error to be sufficiently smll, we cn just keep dding terms until the computed error smll enough (which is something we hve to do nywy even if we knew wht n to tke). Proof. We consider generl polynomil We substitute this in to obtin p = f p, f p = f, f 2 n c k p k. k=0 n c k f, p k + k=0 n c 2 k p k 2. Note tht there re no cross terms between the different coefficients. We minimize this qudrtic by setting the prtil derivtives to zero: k=0 0 = c k f p, f p = 2 f, p k + 2c k p k 2. To check this is indeed minimum, note tht the Hessin mtrix is simply 2I, which is positive definite. So this is relly minimum. So we get the formul for the c k s s climed, nd putting the formul for c k gives the error formul. Note tht our constructed p P n [x] hs nice property: for k n, we hve f p, p k = f, p k p, p k = f, p k f, p k p k 2 p k, p k = 0. Thus for ll q P n [x], we hve f p, q = 0. In prticulr, this is true when q = p, nd tells us f, p = p, p. Using this to expnd f p, f p gives f p 2 + p 2 = f 2, which is just glorified Pythgors theorem. 17
3 Approximtion of liner functionls IB Numericl Anlysis 3 Approximtion of liner functionls 3.1 Liner functionls In this chpter, we re going to study pproximtions of liner functions. Before we strt, it is helpful to define wht liner functionl is, nd look t certin exmples of these. Definition (Liner functionl). A liner functionl is liner mpping L : V R, where V is rel vector spce of functions. In generlly, liner functionl is liner mpping from vector spce to its underlying field of sclrs, but for the purposes of this course, we will restrict to this specil cse. We usully don t put so much emphsis on the ctul vector spce V. Insted, we provide formul for L, nd tke V to be the vector spce of functions for which the formul mkes sense. Exmple. (i) We cn choose some fixed ξ R, nd define liner functionl by L(f) = f(ξ). (ii) Alterntively, for fixed η R we cn define our functionl by L(f) = f (η). In this cse, we need to pick vector spce in which this mkes sense, e.g. the spce of continuously differentible functions. (iii) We cn define L(f) = b f(x) dx. The set of continuous (or even just integrble) functions defined on [, b] will be sensible domin for this liner functionl. (iv) Any liner combintion of these liner functions re lso liner functionls. For exmple, we cn pick some fixed α, β R, nd define L(f) = f(β) f(α) β α [f (β) + f (α)]. 2 The objective of this chpter is to construct pproximtions to more complicted liner functionls (usully integrls, possibly derivtives point vlues) in terms of simpler liner functionls (usully point vlues of f itself). For exmple, we might produce n pproximtion of the form L(f) N i f(x i ), where V = C p [, b], p 0, nd {x i } N i=0 [, b] re distinct points. i=0 18
3 Approximtion of liner functionls IB Numericl Anlysis How cn we choose the coefficients i nd the points x i so tht our pproximtion is good? We notice tht most of our functionls cn be esily evluted exctly when f is polynomil. So we might pproximte our function f by polynomil, nd then do it exctly for polynomils. More precisely, we let {x i } N i=0 [, b] be rbitrry points. Then using the Lgrnge crdinl polynomils l i, we hve f(x) i=0 N f(x i )l i (x). Then using linerity, we cn pproximte ( N ) N L(f) L f(x i )l i (x) = L(l i )f(x i ). So we cn pick i=0 i = L(l i ). Similr to polynomil interpoltion, this formul is exct for f P N [x]. But we could do better. If we cn freely choose { i } N i=0 nd {x i} N i=0, then since we now hve 2n + 2 free prmeters, we might expect to find n pproximtion tht is exct for f P 2N+1 [x]. This is not lwys possible, but there re cses when we cn. The most fmous exmple is Gussin qudrture. 3.2 Gussin qudrture The objective of Gussin qudrture is to pproximte integrls of the form L(f) = b i=0 w(x)f(x) dx, where w(x) is weight function tht determines sclr product. Trditionlly, we hve different set of nottions used for Gussin qudrture. So just in this section, we will use some funny nottion tht is inconsistent with the rest of the course. We let f, g = b w(x)f(x)g(x) dx be sclr product for P ν [x]. We will show tht we cn find weights, written {b n } ν k=1, nd nodes, written {c k} ν k=1 [, b], such tht the pproximtion b w(x)f(x) dx ν b k f(c k ) is exct for f P 2ν 1 [x]. The nodes {c k } ν k=1 will turn out to be the zeros of the orthogonl polynomil p ν with respect to the sclr product. The im of this section is to work this thing out. We strt by showing tht this is the best we cn chieve. k=1 19
3 Approximtion of liner functionls IB Numericl Anlysis Proposition. There is no choice of ν weights nd nodes such tht the pproximtion of b w(x)f(x) dx is exct for ll f P 2ν[x]. Proof. Define Then we know q(x) = b ν (x c k ) P ν [x]. k=1 w(x)q 2 (x) dx > 0, since q 2 is lwys non-negtive nd hs finitely mny zeros. However, So this cnnot be exct for q 2. ν b k q 2 (c n ) = 0. k=1 Recll tht we initilly hd the ide of doing the pproximtion by interpolting f t some rbitrry points in [, b]. We cll this ordinry qudrture. Theorem (Ordinry qudrture). For ny distinct {c k } ν k=1 [, b], let {l k} ν k=1 be the Lgrnge crdinl polynomils with respect to {c k } ν k=1. Then by choosing the pproximtion b k = b w(x)l k (x) dx, L(f) = b w(x)f(x) dx is exct for f P ν 1 [x]. We cll this method ordinry qudrture. ν b k f(c k ) This simple ide is how we generte mny clssicl numericl integrtion techniques, such s the trpezoidl rule. But those re quite inccurte. It turns out clever choice of {c k } does much better tke them to be the zeros of the orthogonl polynomils. However, to do this, we must mke sure the roots indeed lie in [, b]. This is wht we will prove now given ny inner product, the roots of the orthogonl polynomils must lie in [, b]. Theorem. For ν 1, the zeros of the orthogonl polynomil p ν re rel, distinct nd lie in (, b). We hve in fct proved this for prticulr cse in IB Methods, nd the sme rgument pplies. Proof. First we show there is t lest one root. Notice tht p 0 = 1. Thus for ν 1, by orthogonlity, we know b w(x)p ν (x)p 1 (x) dx = b k=1 w(x)p ν (x) dx = 0. 20
3 Approximtion of liner functionls IB Numericl Anlysis So there is t lest one sign chnge in (, b). We hve lredy got the result we need for ν = 1, since we only need one zero in (, b). Now for ν > 1, suppose {ξ j } m j=1 re the plces where the sign of p ν chnges in (, b) (which is subset of the roots of p ν ). We define q(x) = m (x ξ j ) P m [x]. j=1 Since this chnges sign t the sme plce s p ν, we know qp ν mintins the sme sign in (, b). Now if we hd m < ν, then orthogonlity gives q, p ν = b w(x)q(x)p ν (x) dx = 0, which is impossible, since qp ν does not chnge sign. Hence we must hve m = ν. Theorem. In the ordinry qudrture, if we pick {c k } ν k=1 to be the roots of p ν (x), then get we exctness for f P 2ν 1 [x]. In ddition, {b n } ν k=1 re ll positive. Proof. Let f P 2ν 1 [x]. Then by polynomil division, we get f = qp ν + r, where q, r re polynomils of degree t most ν 1. We pply orthogonlity to get b w(x)f(x) dx = b Also, since ech c k is root of p ν, we get ν b k f(c k ) = k=1 w(x)(q(x)p ν (x) + r(x)) dx = ν b k (q(c k )p ν (c k ) + r(c k )) = k=1 b w(x)r(x) dx. ν b k r(c k ). But r hs degree t most ν 1, nd this formul is exct for polynomils in P ν 1 [x]. Hence we know b w(x)f(x) dx = b w(x)r(x) dx = k=1 ν b k r(c k ) = k=1 ν b k f(c k ). To show the weights re positive, we simply pick s specil f. Consider f {l 2 k }ν k=1 P 2ν 2[x], for l k the Lgrnge crdinl polynomils for {c k } ν k=1. Since the qudrture is exct for these, we get 0 < b w(x)l 2 k(x) dx = ν b j l 2 k(c j ) = j=1 k=1 ν b j δ jk = b k. j=1 Since this is true for ll k = 1,, ν, we get the desired result. 21
4 Expressing errors in terms of derivtives IB Numericl Anlysis 4 Expressing errors in terms of derivtives As before, we pproximte liner functionl L by L(f) n i L i (f), i=0 where L i re some simpler liner functionls, nd suppose this is exct for f P k [x] for some k 0. Hence we know the error e L (f) = L(f) n i L i (f) = 0 whenever f P k [x]. We sy the error nnihiltes for polynomils of degree less thn k. How cn we use this property to generte formule for the error nd error bounds? We first strt with rther simple exmple. Exmple. Let L(f) = f(β). We decide to be silly nd pproximte L(f) by i=0 L(f) f(α) + β α (f (β) + f (α)), 2 where α β. This is clerly much esier to evlute. The error is given by nd this vnishes for f P 2 [x]. e L (f) = f(β) f(α) β α (f (β) + f (α)), 2 How cn we get more useful error formul? We cn t just use the fct tht it nnihiltes polynomils of degree k. We need to introduce something beyond this the k + 1th derivtive. We now ssume f C k+1 [, b]. Note tht so fr, everything we ve done works if the intervl is infinite, s long s the weight function vnishes sufficiently quickly s we go fr wy. However, for this little bit, we will need to require [, b] to be finite, since we wnt to mke sure we cn tke the supremum of our functions. We now seek n exct error formul in terms of f (k+1), nd bounds of the form e L (f) c L f (k+1) for some constnt c L. Moreover, we wnt to mke c L s smll s possible. We don t wnt to give constnt of 10 million, while in relity we cn just use 2. Definition (Shrp error bound). The constnt c L is sid to be shrp if for ny ε > 0, there is some f ε C k+1 [, b] such tht e L (f) (c L ε) f (k+1) ε. This mkes it precise wht we men by c L cnnot be better. This doesn t sy nything bout whether c L cn ctully be chieved. This depends on the prticulr form of the question. 22
4 Expressing errors in terms of derivtives IB Numericl Anlysis To proceed, we need Tylor s theorem with the integrl reminder, i.e. f(x) = f() + (x )f () + + (x )k f (k) () + 1 k! k! x (x θ) k f (k+1) (θ) dθ. This is not relly good, since there is n x in the upper limit of the integrl. Insted, we write the integrl s b (x θ) k +f (k+1) (θ) dθ, where we define (x θ) k + is new function defined by { (x θ) k (x θ) k x θ + = 0 x < θ. Then if λ is liner functionl tht nnihiltes P k [x], then we hve ( ) 1 b λ(f) = λ (x θ) k k! +f (k+1) (θ) dθ for ll f C k+1 [, b]. For our liner functionls, we cn simplify by tking the λ inside the integrl sign nd obtin λ(f) = 1 k! b λ((x θ) k +)f (k+1) (θ) dθ, noting tht λ cts on (x θ) k + C k 1 [, b] s function of x, nd think of θ s being held constnt. Of course, pure mthemticins will come up with liner functionls for which we cnnot move the λ inside, but for our liner functionls (point vlues, derivtive point vlues, integrls etc.), this is vlid, s we cn verify directly. Hence we rrive t Theorem (Peno kernel theorem). If λ nnihiltes polynomils of degree k or less, then for ll f C k+1 [, b], where λ(f) = 1 k! b K(θ)f (k+1) (θ) dθ Definition (Peno kernel). The Peno kernel is K(θ) = λ((x θ) k +). The importnt thing is tht the kernel K is independent of f. Tking suprem in different wys, we obtin different forms of bounds: b K(θ) dθ f (k+1) λ(f) 1 ( ) 1 b 2 k! K(θ) 2 dθ f (k+1). 2 K(θ) f (k+1) 1 23
4 Expressing errors in terms of derivtives IB Numericl Anlysis Hence we cn find the constnt c L for different choices of the norm. When computing c L, don t forget the fctor of 1 k!! By fiddling with functions bit, we cn show these bounds re indeed shrp. Exmple. Consider our previous exmple where e L (f) = f(β) f(α) β α (f (β) + f (α)), 2 with exctness up to polynomils of degree 2. We wlog ssume α < β. Then K(θ) = e L ((x θ) 2 +) = (β θ) 2 + (α θ) 2 + (β α)((β θ) + + (α θ) + ). Hence we get Hence we know for ll f C 3 [, b]. 0 θ α K(θ) = (α θ)(β θ) α θ β 0 β θ b. e L (f) = 1 2 β α (α θ)(β θ)f (θ) dθ Note tht in this prticulr cse, our function K(θ) does not chnge sign on [, b]. Under this extr ssumption, we cn sy bit more. First, we note tht the bound 1 b λ(f) K(θ) dθ k! f (k+1) cn be chieved by x k+1, since this hs constnt k + 1th derivtive. Also, we cn use the integrl men vlue theorem to get the bound ( ) λ(f) = 1 b K(θ) dθ f (k+1) (ξ), k! where ξ (, b) depends on f. These re occsionlly useful. Exmple. Continuing our previous exmple, we see tht K(θ) 0 on [, b], nd Hence we hve the bound b K(θ) dθ = 1 6 (β α)3. e L (f) 1 12 (β α)3 f, nd this bound is chieved for x 3. We lso hve e L (f) = 1 12 (β α)3 f (ξ) for some f-dependent vlue of some ξ (, b). 24
4 Expressing errors in terms of derivtives IB Numericl Anlysis Finlly, note tht Peno s kernel theorem sys if e L (f) = 0 for ll f P k [x], then we hve e L (f) = 1 k! b K(θ)f (k+1) (θ) dθ for ll f C k+1 [, b]. But for ny other fixed j = 0,, k 1, we lso hve e L (f) = 0 for ll f P j [x]. So we lso know e L (f) = 1 j! b K j (θ)f (j+1) (θ) dθ for ll f C j+1 [, b]. Note tht we hve different kernel. In generl, this might not be good ide, since we re throwing informtion wy. Yet, this cn be helpful if we get some less smooth functions tht don t hve k + 1 derivtives. 25
5 Ordinry differentil equtions IB Numericl Anlysis 5 Ordinry differentil equtions 5.1 Introduction Our next big gol is to solve ordinry differentil equtions numericlly. We will focus on differentil equtions of the form for 0 t T, with initil conditions y (t) = f(t, y(t)) y(0) = y 0. The dt we re provided is the function f : R R N R N, the ending time T > 0, nd the initil condition y 0 R n. Wht we seek is the function y : [0, T ] R N. When solving the differentil eqution numericlly, our gol would be to mke our numericl solution s close to the true solution s possible. This mkes sense only if true solution ctully exists, nd is unique. From IB Anlysis II, we know unique solution to the ODE exists if f is Lipschitz. Definition (Lipschitz function). A function f : R R N R N is Lipschitz with Lipschitz constnt λ 0 if f(t, x) f(t, ˆx) λ x ˆx for ll t [0, T ] nd x, ˆx R N. A function is Lipschitz if it is Lipschitz with Lipschitz constnt λ for some λ. It doesn t relly mtter wht norm we pick. It will just chnge the λ. The importnce is the existence of λ. A specil cse is when λ = 0, i.e. f does not depend on x. In this cse, this is just n integrtion problem, nd is usully esy. This is convenient test cse if our numericl pproximtion does not even work for these esy problems, then it s pretty useless. Being Lipschitz is sufficient for existence nd uniqueness of solution to the differentil eqution, nd hence we cn sk if our solution converges to this unique solution. An extr ssumption we will often mke is tht f cn be expnded in Tylor series to s mny degrees s we wnt, since this is convenient for our nlysis. Wht exctly does numericl solution to the ODE consist of? We first choose smll time step h > 0, nd then construct pproximtions y n y(t n ), n = 1, 2,, with t n = nh. In prticulr, t n t n 1 = h nd is lwys constnt. In prctice, we don t fix the step size t n t n 1, nd llow it to vry in ech step. However, this mkes the nlysis much more complicted, nd we will not consider vrying time steps in this course. If we mke h smller, then we will (probbly) mke better pproximtions. However, this is more computtionlly demnding. So we wnt to study the behviour of numericl methods in order to figure out wht h we should pick. 26
5 Ordinry differentil equtions IB Numericl Anlysis 5.2 One-step methods There re mny wys we cn clssify numericl methods. One importnt clssifiction is one-step versus multi-step methods. In one-step methods, the vlue of y n+1 depends only on the previous itertion t n nd y n. In multi-step methods, we re llowed to look bck further in time nd use further results. Definition ((Explicit) one-step method). A numericl method is (explicit) one-step if y n+1 depends only on t n nd y n, i.e. for some function φ h : R R N R N. y n+1 = φ h (t n, y n ) We will lter see wht explicit mens. The simplest one-step method one cn imgine is Euler s method. Definition (Euler s method). Euler s method uses the formul y n+1 = y n + hf(t n, y n ). We wnt to show tht this method converges. First of ll, we need to mke precise the notion of convergence. The Lipschitz condition mens there is unique solution to the differentil eqution. So we would wnt the numericl solution to be ble to pproximte the ctul solution to rbitrry ccurcy s long s we tke smll enough h. Definition (Convergence of numericl method). For ech h > 0, we cn produce sequence of discrete vlues y n for n = 0,, [T/h], where [T/h] is the integer prt of T/h. A method converges if, s h 0 nd nh t (hence n ), we get y n y(t), where y is the true solution to the differentil eqution. Moreover, we require the convergence to be uniform in t. We now prove tht Euler s method converges. We will only do this properly for Euler s method, since the lgebr quickly becomes tedious nd incomprehensible. However, the proof strtegy is sufficiently generl tht it cn be dpted to most other methods. Theorem (Convergence of Euler s method). (i) For ll t [0, T ], we hve lim y n y(t) = 0. h 0 nh t (ii) Let λ be the Lipschitz constnt of f. Then there exists c 0 such tht e n ch eλt 1 λ for ll 0 n [T/h], where e n = y n y(t n ). 27
5 Ordinry differentil equtions IB Numericl Anlysis Note tht the bound in the second prt is uniform. So this immeditely gives the first prt of the theorem. Proof. There re two prts to proving this. We first look t the locl trunction error. This is the error we would get t ech step ssuming we got the previous steps right. More precisely, we write y(t n+1 ) = y(t n ) + h(f, t n, y(t n )) + R n, nd R n is the locl trunction error. For the Euler s method, it is esy to get R n, since f(t n, y(t n )) = y (t n ), by definition. So this is just the Tylor series expnsion of y. We cn write R n s the integrl reminder of the Tylor series, R n = By some creful nlysis, we get where tn+1 t n (t n+1 θ)y (θ) dθ. R n ch 2, c = 1 2 y. This is the esy prt, nd tends to go rther smoothly even for more complicted methods. Once we hve bounded the locl trunction error, we ptch them together to get the ctul error. We cn write e n+1 = y n+1 y(t n+1 ) ) = y n + hf(t n, y n ) (y(t n ) + hf(t n, y(t n )) + R n ( ) = (y n y(t n )) + h f(t n, y n ) f(t n, y(t n )) R n Tking the infinity norm, we get e n+1 y n y(t n ) + h f(t n, y n ) f(t n, y(t n )) + R n e n + hλ e n + ch 2 = (1 + λh) e n + ch 2. This is vlid for ll n 0. We lso know e 0 = 0. Doing some lgebr, we get Finlly, we hve n 1 e n ch 2 (1 + hλ) j ch λ ((1 + hλ)n 1). j=0 (1 + hλ) e λh, since 1 + λh is the first two terms of the Tylor series, nd the other terms re positive. So (1 + hλ) n e λhn e λt. So we obtin the bound e n ch eλt 1. λ Then this tends to 0 s we tke h 0. So the method converges. 28
5 Ordinry differentil equtions IB Numericl Anlysis This works s long s λ 0. However, λ = 0 is the esy cse, since it is just integrtion. We cn either check this cse directly, or use the fct tht e λt 1 λ T s λ 0. The sme proof strtegy works for most numericl methods, but the lgebr will be much messier. We will not do those in full. We, however, tke note of some useful terminology: Definition (Locl trunction error). For generl (multi-step) numericl method y n+1 = φ(t n, y 0, y 1,, y n ), the locl trunction error is η n+1 = y(t n+1 ) φ n (t n, y(t 0 ), y(t 1 ),, y(t n )). This is the error we will mke t the (n + 1)th step if we hd ccurte vlues for the first n steps. For Euler s method, the locl trunction error is just the Tylor series reminder term. Definition (Order). The order of numericl method is the lrgest p 1 such tht η n+1 = O(h p+1 ). The Euler method hs order 1. Notice tht this is one less thn the power of the locl trunction error, since when we look t the globl error, we drop power, nd only hve e n h. Let s try to get little bit beyond Euler s method. Definition (θ-method). For θ [0, 1], the θ-method is ( ) y n+1 = y n + h θf(t n, y n ) + (1 θ)f(t n+1, y n+1 ). If we put θ = 1, then we just get Euler s method. The other two most common choices of θ re θ = 0 (bckwrd Euler) nd θ = 1 2 (trpezoidl rule). Note tht for θ 1, we get n implicit method. This is since y n+1 doesn t just pper simply on the left hnd side of the equlity. Our formul for y n+1 involves y n+1 itself! This mens, in generl, unlike the Euler method, we cn t just write down the vlue of y n+1 given the vlue of y n. Insted, we hve to tret the formul s N (in generl) non-liner equtions, nd solve them to find y n+1! In the pst, people did not like to use this, becuse they didn t hve computers, or computers were too slow. It is tedious to hve to solve these equtions in every step of the method. Nowdys, these re becoming more nd more populr becuse it is getting esier to solve equtions, nd θ-methods hve some huge theoreticl dvntges (which we do not hve time to get into). We now look t the error of the θ-method. We hve ( ) η = y(t n+1 ) y(t n ) h θy (t n ) + (1 θ)y (t n+1 ) We expnd ll terms bout t n with Tylor s series to obtin ( = θ 1 ) ( 1 h 2 y (t n ) + 2 2 θ 1 ) h 3 y (t n ) + O(h 4 ). 3 We see tht θ = 1 2 gives us n order 2 method. Otherwise, we get n order 1 method. 29
5 Ordinry differentil equtions IB Numericl Anlysis 5.3 Multi-step methods We cn try to mke our methods more efficient by mking use of previous vlues of y n insted of the most recent one. One common method is the AB2 method: Definition (2-step Adms-Bshforth method). The 2-step Adms-Bshforth (AB2) method hs y n+2 = y n+1 + 1 2 h (3f(t n+1, y n+1 ) f(t n, y n )). This is prticulr exmple of the generl clss of Adms-Bshforth methods. In generl, multi-step method is defined s follows: Definition (Multi-step method). A s-step numericl method is given by s ρ l y n+l = h l=0 s σ l f(t n+l, y n+l ). This formul is used to find the vlue of y n+s given the others. l=0 One point to note is tht we get the sme method if we multiply ll the constnts ρ l, σ l by non-zero constnt. By convention, we normlize this by setting ρ s = 1. Then we cn lterntively write this s y n+s = h s l=0 s 1 σ l f(t n+l, y n+l ) ρ l y n+l. This method is n implicit method if σ s 0. Otherwise, it is explicit. Note tht this method is liner in the sense tht the coefficients ρ l nd σ l pper linerly, outside the f s. Lter we will see more complicted numericl methods where these coefficients pper inside the rguments of f. For multi-step methods, we hve slight problem to solve. In one-step method, we re given y 0, nd this llows us to immeditely pply the one-step method to get higher vlues of y n. However, for n s-step method, we need to use other (possibly 1-step) method to obtin y 1,, y s 1 before we cn get strted. Fortuntely, we only need to pply the one-step method fixed, smll number of times, even s h 0. So the ccurcy of the one-step method t the strt does not mtter too much. We now study the properties of generl multi-step method. The first thing we cn tlk bout is the order: Theorem. An s-step method hs order p (p 1) if nd only if nd s ρ l = 0 l=0 s ρ l l k = k l=0 for k = 1,, p, where 0 0 = 1. l=0 s σ l l k 1 l=0 30