Physics Mster Course: Numericl Methods 2007 Hns Mssen Rdboud Universiteit Nijmegen Onderwijsinstituut Wiskunde, Ntuur- en Sterrenkunde Toernooiveld 1 6525 ED Nijmegen September 2007
1 Introduction In these notes we introduce some of the min methods of numericl nlysis: numericl solution of equtions, polynomil interpoltion nd pproximtion, numericl integrtion nd solution of ordinry differentil equtions References [BuF] Richrd L Burden, J Dougls Fires: Numericl Anlysis, PWS Publishing Compny, Boston, 1993 [Sto] J Stoer: Numerische Mthemtik 1, Springer, 1980 [Jeu] M de Jeu: Numerieke Wiskunde 1, 2003-2004, Dictt Mthemtisch Instituut, Universiteit Leiden, jnuri 2004 2 Solving Equtions In this chpter we tret severl methods for the solution of equtions in one vrible The most bsic method, repeted bisection, hs lredy been treted by Andres Gürtler in his introduction to Mtlb This method is esy to use nd prcticlly lwys works It hs the disdvntges, however, of being slow, nd not generlising esily to higher dimensions We shll now discuss two other common methods: itertion of function, nd the method of Newton-Rphson 21 Itertion Suppose we wish to solve the eqution cos(x) = x This cn be esily chieved on pocket clcultor s follows: Enter ny rel number (for instnce 0) Press the cosine button repetedly The number in disply will converge to the fixed point of the cosine function, ie the solution of our eqution How fr we re still removed from the solution (the error ) cn be estimted from the chnge in the disply upon pressing the button: in this exmple the chnge is lwys lrger thn the error 1
X 2 X 1 0 1 X 0 Theorem 1 Let g : [, b] [, b] be differentble with continuous derivtive Suppose tht there exists k < 1 such tht for ll x [, b]: g (x) k Then g hs unique fixed point p, nd for ny choice of strting point x 0 [, b] the sequence defined by x n+1 := g(x n ), (n N) converges to p We obtin the bove exmple by choosing g : [ 1, 1] [ 1, 1] : x cos(x) Indeed, for 1 x 1 we hve cos (x) = sin(x) sin(1) < 1 (Note tht for generl strting point) x 0 R outside [ 1, 1] the first itertion x 1 = cos(x 0 ) lredy lies in [ 1, 1]) We shll write the n-fold composition g g g g s g n Proof of Theorem 1 x nd y such tht Therefore, for ll n N, Now let X n denote the set of n-th itertes: By the men vlue theorem, for ll x, y [, b] there exists z between g(x) g(y) = g (z) x y k x y g n (x) g n (y) k n x y k n (b ) X n := { g n (x) x b } We clim tht s n increses, these sets shrink to point, which must then be the unique fixed point of g Indeed, let n := inf(x n ) nd b n := sup(x n ) Since X 0 X 1 X 2, the sequence 0, 1, 2, must be incresing, sy with limit p, nd the sequence b 0, b 1, b 2, must be decresing, sy with limit p But since b n n k n (b ), 2
p must be equl to p Now choose ny x 0 [, b] Then, since x n := g n (x 0 ) X n, n x n b n Hence the sequence x 0, x 1, x 2, x 3, tends to p This limit p must be fixed point since g is continuous: ( ) g(p) = g lim x n = lim g(x n) = lim x n+1 = p n n n Finlly, if q [, b] is fixed point of g, then q = g n (q) Hence So the fixed point p is unique q = lim n g n (q) = p Mny equtions cn be cst in the form of fixed point eqution eqution hs unique root in the intervl [1, 2] It cn be rewritten s For exmple, the x 3 + 4x 2 10 = 0 (1) 4x 2 = 10 x 3, so x = 1 2 10 x3 However, t first the function g : [1, 2] R : x 1 2 10 x3 does not meet our conditions, since it does not mp [1, 2] into itself, nd hs too lrge derivtive: g (x) = 3x 2 4 10 x 3, vrying from g (1) = 1 4 to g (2) = 2 3 2 However, when we restrict the function to [1, 3 2 ], it stisfies g (x) g ( 3 2 ) 0,66 So with ny strting point in [1, 3 ] convergence is ssured With some inventiveness we 2 cn improve this procedure further: we cn reformulte the eqution (1) s x 2 (x + 4) = 10, so x = 10 x + 4 It is not difficult to check tht the function g(x) := 10/(x + 4) mps the intervl [1, 2] into itself, nd hs quite smll derivtive there The convergence is ccordingly fster Roughly, the error in the pproximtion of p decreses by fctor g (p) ech step once we re close to p This mens tht we gin digit of ccurcy every 10 log g (p) steps Not knowing p in dvnce, we cn estimte this speed of convergence by 10 log k 3
Exercise 1 Clculte the root of eqution (1) to 9 deciml plces, using both itertion procedures described bove Keep count of the number of itertions you need in ech cse 22 The method of Newton-Rphson The following very powerful method for the clcultion of squre roots is built into the hrdwre of pocket clcultors Let c be positive number, nd define x 0 := c nd x n+1 := 1 ( x n + c ) 2 x n Then the sequence x 0, x 1, x 2, converges to c quite fst Once we re close to the squre root, the number of ccurte digits roughly doubles with every step! This is specil cse of the method of Newton-Rphson, which works s follows Theorem 2 Let f : [, b] R be twice differentible with f() 0 nd f(b) > 0 Suppose tht for ll x [, b] both f (x) nd f (x) re strictly positive nd bounded from bove Let x 0 [, b] be such tht f(x 0 ) > 0, nd define for n = 0, 1, 2, : x n+1 := x n f(x n) f (x n ) Then the sequence x 0, x 1, x 2, converges to the unique root of the eqution f(p) = 0 f(x) g(x) x We get the exmple bove, the lgorithm to clculte c, by choosing f : [ c, c] R : x x 2 c Clerly, f nd f re strictly positive on this intervl Moreover, the function g : [ c, c] [ c, c] : x x f(x) f (x) 4
cn lso be written s g(x) = x x2 c = 1 ( x + c ) 2x 2 x We cn prove tht the lgorithm works in this cse using Theorem 1! Indeed, for x c: g (x) = 1 ( 1 c ) [0, 1 2 x 2 2 ] It follows tht the sequence x 0 c, x 1 := g(x 0 ), x 2 := g(g(x 0 )), converges to the unique fixed point of g, which is c Proof of Theorem 2 Since f(x 0 ) 0, we hve x 0 p Suppose tht x n p for some n Then by the men vlue theorem, f(x n ) = f(p) + f (y)(x n p), nd since f(p) = 0 nd f is incresing, f(x n ) f (x n )(x n p), from which it follows tht f(x n+1 ) = x n f(x n) f (x n ) p By induction, the whole sequence x 0, x 1, x 2, lies bove p, so tht f(x n ) 0, nd the sequence is decresing So it hs limit l, which must be such tht l = l f(l)/f (l), hence f(l) = 0 nd l = p Theorem 2 nd its proof do not mke cler why the convergence is so fst We shll discuss this point seprtely Theorem 3 In the sitution of Theorem 2 there is positive constnt γ such tht for n N: x n+1 p γ x n p 2 This theorem sys tht, roughly speking, when we re getting close to p the number of ccurte digits doubles with every step Proof of Theorem 3 Now note tht We cn view the Newton-Rphson method s the itertion of g : [p, b] [p, b] : x x f(x) f (x) g (x) = f(x)f (x) f (x) 2 αf(x), for some properly chosen constnt α Also, f (x) β for some β It follows tht there re y nd z with p z y x such tht g(x) p = g (y) x p αf(y) x p = αf (z) x p 2 αβ x p 2 Exercise 2 Clculte the fixed point of the cosine function to 11 deciml plces using () bisection; (b) itertion; (c) Newton-Rphson Keep count of the number of steps required in ech method 5
Exercise 3 () Use the method of Newton-Rphson to find the solution ccurte to within 10 12 of the eqution (x 2) 2 log(x) = 0, for x [1, 2] nd for x [e, 4] (b) Use the method of Newton-Rphson to pproximte, to within 10 10, the vlue of x tht produces the point on the grph of y = 1 x tht is closest to (2, 1) 6
3 Polynomil Approximtion nd Interpoltion Polynomils re esily evluted by computers They re, moreover, esy to mnipulte: ddition, multipliction nd differentition re strightforwrd This mkes them idel tools for the pproximtion of functions nd the interpoltion of dt by smooth curves 31 Approximting continuous functions It is n encourging fct tht continuous functions cn be pproximted by polynomils rbitrrily well This is the content of the following Theorem Theorem 4 (Weierstrß) Let f : [, b] R be continuous nd let ε > 0 Then there exists polynomil p such tht for ll x [, b]: f(x) p(x) < ε For this stndrd theorem of functionl nlysis we shll give constructive proof with probbilistic flvour Proof Without loss of generlity we my ssume tht [, b] = [0, 1] For n N, let B n be the Bernstein polynomil of degree n bsed on f: B n (x) := n k=0 ( ) n ( k f x k n) k (1 x) n k, (0 x 1) We clim tht, for n lrge enough, B n pproximtes f s required Since f is continous function on closed intervl, it is uniformly continuous, nd hence there exists δ > 0 such tht for ll x, y [, b]: x y < δ = f(x) f(y) < 1 2 ε We rewrite our polynomil in probbilistic lnguge s follows Let us fix x [0, 1], nd let X be rndom vrible, hving binomil distribution with prmeters n nd x Then ( 1 ) B n (x) = E f( ) n X Now note tht E(X/n) = x nd Vr (X/n) = x(1 x)/n So by Chebyshev s inequlity [ ] 1 P n X x δ 1 x(1 x) δ2 n We choose n so lrge tht 1 nδ 2 < ε f, 7
where f denotes the mximl vlue of f on [, b] Then, since x(1 x) 1 4, (f( B n (x) f(x) = 1 ) ) E n X ( f(x) 1 ) E f n X f(x) ( ( 1 ) = E f n X ) [ ] f(x) 1 n X x < δ 1 P n X x < δ ( ( 1 ) + E f n X ) [ ] f(x) 1 n X x δ 1 P n X x δ 1 2 ε + 2 f ε 4 f = ε In prctice, the Bernstein polynomils re not very useful, since their convergence is slow They hve, however, the gret dvntge tht they re quite stble: if f is chnged only slightly, the Bernstein polynomils lso move little Their gret competitors, the Lgrnge polynomils, re highly sensitive to smll chnges in f Exercise 4 Clculte numericlly the Bernstein polynomil for the function f : [ 1, 1] R : x x How lrge vlue of n do you need in order tht f B n < 008? Compre this with the estimte in the proof of Theorem 4 32 Lgrnge s Interpoltion Formul Through two points in plne psses single line Accordingly, through two points (ξ 0, α 0 ) nd (ξ 1, α 1 ) in R 2 (with different x-coordintes: ξ 0 ξ 1 ), psses the grph of unique liner function This function cn be written in different wys: either s or s L 1 (x) = α 0 + (α 1 α 0 ) x ξ 0 ξ 1 ξ 0, L 1 (x) = α 0 x ξ 1 ξ 0 ξ 1 + α 1 x ξ 0 ξ 1 ξ 0 (Check tht these functions indeed pss through the given points, nd tht they re the sme) This is the liner Lgrnge polynomil through the two points The first formul is known s Newton s liner interpoltion formul nd the second s Lgrnge s liner interpoltion formul Through three points (ξ 0, α 0 ), (ξ 1, α 1 ), nd (ξ 2, α 2 ), psses the grph of unique qudrtic function L 2 It cn lterntively be written (in Newton s wy ) s L 2 (x) = α 0 + (α 1 α 0 ) x ξ 0 (x ξ 0 )(x ξ 1 ) + (α 2 L 1 (ξ 2 )) ξ 1 ξ 0 (ξ 2 ξ 0 )(ξ 2 ξ 1 ), or (in Lgrnge s wy ) s (x ξ 1 )(x ξ 2 ) L 2 (x) = α 0 (ξ 0 ξ 1 )(ξ 0 ξ 2 ) + α (x ξ 0 )(x ξ 2 ) 1 (ξ 1 ξ 0 )(ξ 1 ξ 2 ) + α (x ξ 0 )(x ξ 1 ) 2 (ξ 2 ξ 0 )(ξ 2 ξ 1 ) 8
(Check gin tht these functions indeed pss through the given points, nd tht they re the sme) In this wy we my continue First we follow Lgrnge s line; we shll come bck to Newton s recursive interpoltion procedure in the next section Lgrnge s pproch is bsed on liner lgebr: the set P n of ll polynomils of degree n cn be viewed s vector spce under ordinry ddition nd multipliction by sclrs The dimension of this spce is n + 1; bsis is formed by the set of functions 1, x, x 2,, x n If set of rel numbers ξ 0, ξ 1,, ξ n is given, useful lterntive bsis is the Lgrnge bsis {λ 0, λ 1,, λ n }, given by λ i (x) = k i x ξ k ξ i ξ k It is esily seen tht indeed these re polynomils of degree n nd tht { 1 if i = j, λ i (ξ j ) = δ ij := 0 if i j Theorem 5 (Lgrnge s interpoltion formul) Through n + 1 points (ξ 0, α 0 ), (ξ 1, α 1 ),, (ξ n, α n ) psses the grph of unique polynomil L n of degree n On the Lgrnge bsis it is given by n L n (x) = α i λ i (x) i=0 Proof Uniqueness: Suppose tht the grphs of polynomils p nd q of degree n pss through our n + 1 points: p(ξ i ) = α i, q(ξ i ) = α i, for i = 0, 1, n Now, nonvnishing polynomil of degree n cnnot hve more thn n zeroes Therefore the polynomil p q, being zero in n + 1 points, nmely ξ 0, ξ n, must vnish identiclly, ie p = q Existence: Clerly, for j = 0, 1,, n, L n (ξ j ) = So L n fulfills our requirements n α i λ i (ξ j ) = i=0 n α i δ ij = α j i=0 33 Newton s Interpoltion Method Suppose tht we hve fitted polynomil through number of dt points, nd tht someone does n extr mesurement, giving us new point If we were to use Lgrnge s interpoltion method described bove, we would hve to strt ll over gin, clculting n entirely new Lgrnge bsis Newton s method llows us to incorporte the new point, still using the results of previous clcultions The result is, of course, the sme Here is recipe 9
Procedure Let the polynomil L n of degree n pssing through the points (ξ 0, α 0 ), (ξ 1, α 1 ),, (ξ n, α n ) be given Let (ξ n+1, α n+1 ) be new point, with ξ n+1 ξ i for i = 0,, n Then the polynomil of degree n + 1 pssing through ll the old points nd the new point is given by L n+1 (x) := L n (x) + (α n+1 L n (ξ n+1 )) n i=0 x ξ i ξ n+1 ξ i (2) Indeed, the extr term chnges nothing in the points ξ 0,, ξ n, nd sets the vlue t ξ n+1 to α n+1 34 Approximtion errors We now return to function pproximtion Let f : [, b] R nd set of rel numbers ξ 0 < ξ 1 < < ξ n b be given We my pproximte f by the Lgrnge polynomil through the points (ξ 0, f(ξ 0 )), (ξ 1, f(ξ 1 )),, (ξ n, f(ξ n )): L n (x) := n f(ξ i )λ i (x) i=0 How good is this pproximtion? Tht depends on the smoothness of f In principle, the vlues of f outside the set {ξ 0, ξ 1,, ξ n } ber no reltion to the vlues in these points, but if f is few times differentible, nd bound is known on the derivtive, then L n my be proved to lie quite close to f everywhere It is this kind of result which we re fter in this section We strt with kind of n-th order men vlue theorem Theorem 6 Let f : [, b] R be n + 1 times differentible with continuous n + 1-st derivtive f (n+1) Let ξ 0 < ξ 1 < < ξ n b nd let L n denote the Lgrnge interpoltion of f in the points ξ 0, ξ 1,, ξ n Then for every x [, b] there exists q x [, b] such tht f(x) L n (x) = 1 (n + 1)! (x ξ 0)(x ξ 1 ) (x ξ n )f (n+1) (q x ) Proof Fix number x [, b], nd let L n+1 denote the Lgrnge interpoltion of f in ξ 0, ξ 1,, ξ n, nd x Differentiting (2), with ξ n+1 replced by x, n + 1 times, we find tht L (n+1) n+1 is constnt, nmely ( f(x) Ln (x) ) (n + 1)! (x ξ 0 ) (x ξ n ) We re done if we show tht this constnt vlue of L (n+1) n+1 is tken by f (n+1) in t lest one point We rgue s follows The difference f L n+1 hs (t lest) n + 2 zeroes: ξ 0, ξ 1,, ξ n, nd x By Rolle s Theorem, between every pir of zeroes of f L n+1 there must lie t lest one zero of f L n+1, hence f L n+1 must hve t lest n + 1 zeroes Continuing in this wy, we conclude tht f (n+1) L (n+1) n+1 indeed hs t lest one zero 10
Corollry 7 Under the conditions of Theorem 6 we hve f(x) L n (x) 1 (n + 1)! (x ξ0 )(x ξ 1 ) (x ξ n ) Sometimes we my be stisfied with the rough nd simple estimte: f L n (b )n+1 (n + 1)! f (n+1) f (n+1) (3) Wrning The bove results my seem to indicte tht Lgrnge polynomil pproximtions of smooth functions converge relibly nd quickly Does not the fctor 1/(n + 1)! gurntee this? The following exmple shows tht this suggestion is misleding Consider the function f(x) = 1 1+x on [ 5, 5] Let L 2 n denote the Lgrnge interpoltion of f in n + 1 equidistnt points 5 = ξ 0 < ξ 1 < < ξ n = 5 Then for every x ( 5, 5) the sequence L 1 (x), L 2 (x), L 3 (x), diverges Exercise 5 Show tht the estimte in Corollry 7 does not gurntee convergence of the sequence (L n ) n N in the bove exmple in ny point x of the intervl [ 5, 5] except the endpoints Explin why this is not counterexmple to Theorem 4 Exercise 6 Find the 4-th order Lgrnge polynomil L 4 for the function log : [1, 2] R with points of support 1, 5 4, 3 2, 7 4, nd 2 Evlute this polynomil in the point 11 8, nd the error of this pproximtion, ( 11 ) ( L4 8 log 11 ) 8 Compre this error to the error bound of Corollry 7 35 Piecewise Approximtion The exmple in Exercise 5 bove shows tht Lgrnge pproximtion of functions by polynomils of higher nd higher degree is not lwys good ide For this reson it is generl prctice to keep the degree low, sy just 1, 2 or 3, nd to pproximte functions piecewise This method will be the bsis of the next Chpter, numericl integrtion 11
4 Numericl Integrtion The gol of this chpter is to describe methods for numericl pproximtion of the integrl f(x)dx, given some more or less smooth function f : [, b] R The theory of Riemnn integrtion gives such method: clculting Riemnn sums However, the convergence properties of this method re poor Under mild smoothness ssumptions on f considerbly better procedures re vilble Our strtegy will be to first pproximte f by polynomil p n, nd then to integrte p n This my be done on the whole intervl [, b], leding to diversity of qudrture rules (rules for clculting res, ie integrls) Or it my be done fter subdivision of the intervl, nd piecewise pproximtion by polynomils, leding to composite qudrture rules 41 Bsic Qudrture Rules We strt with two simple qudrture rules Trpezium rule If f : [, b] R is twice differentible with continuous second derivtive, then the expression T (f) := 1 2 (b )( f() + f(b) ) pproximtes the integrl of f with n ccurcy given by T (f) (b )2 f(x)dx f 12 Midpoint rule f(x) T(f) b If f : [, b] R is twice differentible with continuous second derivtive, then the expression M(f) := (b ) f ( ) +b 2 pproximtes the integrl of f with n ccurcy given by M(f) (b )2 f(x)dx f 24 f(x) M(f) Before we proceed to prove these sttements, we note tht the pprently more primitive Midpoint Rule is fctor 2 better then the Trpezium Rule! Wht is more, the figure below indictes tht, if the Trpezium Rule yields number tht is too lrge, the Midpoint Rule usully comes out too smll, nd conversely This suggests the, often much better, pproximtion described below b 12
Simpson Rule If f : [, b] R is four times differentible with continuous fourth derivtive, then the expression ) ( ( S(f) := 3( 1 T (f)+2m(f) = 1 6 f() + 4f +b ) ) 2 + f(b) pproximtes the integrl of f with n ccurcy given by S(f) (b )5 f(x)dx f (4) 2880 f(x) T(f) M(f) b 42 Qudrture Rules in Generl We shll prove the results in the preceeding section by lemm, bsed on Theorems 5 nd Corollry 7 of Chpter 3 The lemm sys tht ny set of n + 1 points, chosen in some intervl, leds to qudrture rule which is exct for ll polynomils of degree n or less Lemm 8 Let f : [, b] R be n + 1 times differentible with continuous n + 1-st derivtive f (n+1) Let ξ 0 < ξ 1 < < ξ n b nd let λ 0, λ 1,, λ n be the Lgrnge bsis ssocited to these points Then the expression n I n (f) := c j f(ξ j ) where c j := λ j (x)dx j=0 defines n pproximtion to the integrl of f stisfying I n(f) f(x)dx 1 f (n+1) (x ξ 0 ) (x ξ n ) dx (n + 1)! Proof Let L n be the Lgrnge polynomil of degree n pssing through the points (ξ 0, f(ξ 0 )),, (ξ n, f(ξ n )) It follows from Theorem 5 tht n n L n (x)dx = f(ξ j ) λ j (x)dx = c j f(ξ j ) = I n (f) Hence I n(f) f(x)dx = j=0 By Theorem 7 the sttement follows L n (x)dx j=0 f(x)dx L n (x) f(x) dx Proof of the Trpezium Rule In the bove lemm put n = 1, ξ 0 =, ξ 1 = b, nd note tht x b b c 0 = b dx = 1 2 (b ) nd c x 1 = b dx = 1 (b ) 2 13
So in this cse, I 2 (f) = 1 2( b )(f() + f(b) ) = T (f) Moreover, chnging to the vrible u := (x )/(b ): Hence by the lemm T (f) 1 (x )(x b) dx = (b ) 3 u(1 u) du = 1 6 (b )3 f dx 1 2 f 0 (x )(x b) dx = 1 12 f (b ) 3 Proof of the Midpoint Rule into Lemm 8 we obtin M(f) fdx f Here we hve n = 0, ξ 0 = 1 ( + b) Substituting these dt 2 1 x ξ 0 dx = (b ) 2 f u 1 2 du = 1 4 (b )2 f 0 However, we get much better estimte by observing tht, if we rise n to 1, nd pick n extr point ξ 1 ξ 0 somewhere in the intervl [, b], our Lgrnge polynomil chnges, ccording to (2), by L 1 L 0 = cst (x ξ 0 ) It follows tht the qudrture rule does not chnge t ll: I 1 (f) I 0 (f) = cst Applying Lemm 8 to this sitution, we obtin: M(f) fdx 1 2 f (x ξ 0 )dx = 0 (x ξ 0 )(x ξ 1 ) dx This holds for ll ξ 1 ξ 0 = 1 2 ( + b), but we my tke the limit ξ 1 ξ 0 : 1 M(f) fdx 1 2 f (x ξ 0 ) 2 dx = (b ) 3 (u 1 2 )2 du = 1 24 f (b ) 3 0 Proof of the Simpson Rule This time, tke n = 3, ξ 0 =, ξ 1 = 1 2 ( + b), ξ 2 = b, nd ξ 3 rbitrry The Lgrnge qudrture rule I 2 pplied to the points ξ 0, ξ 1 nd ξ 2 only, is precisely the Simpson Rule: S(f) = I 2 (f) Indeed, (x )(x b) 1 c 1 = dx = 4(b ) u(1 u) du = 2 (ξ 1 )(ξ 1 b) 3 (b ), 0 14
nd c 0 = c 2 = 1 6 (b ) since c 0 + c 1 + c 2 = I 2 (1) = b nd c 0 = c 2 by symmetry When we dd the fourth point ξ 3, the Lgrnge polynomil chnges from L 2 to L 3, where (see (2)), L 3 (x) L 2 (x) = cst (x )(x 1 ( + b))(x b), 2 so tht the Lgrnge qudrture does not chnge t ll: I 3 I 2 = ( L3 (x) L 2 (x) ) dx = 0 Hence I 3 does not depend on ξ 3, nd we my tke ξ 3 ξ 1 in our ppliction of Lemm 8: S(f) fdx 1 1 f (4) (b ) 5 u(1 u)(u 1 (b 4! 2 )2 )5 du = f (4) 2880 0 43 Composite Qudrture Rule nd Order The Lgrngin qudrture rule I n of Lemm 8 is bsed on n + 1 points, nd yields the exct integrl for polynomil functions of degree n or less However, we hve seen tht it my s well be exct for ll polynomil functions of n even higher degree Definition The order of qudrture rule is the lowest vlue of k for which the rule is exct on ll polynomils of degree less thn k, but gives wrong result for some polynomil of degree k For exmple, the Trpezium nd Midpoint rules hve order 2, nd the Simpson rule hs order 4 We hve seen t the end of Chpter 3 tht, in order to increse the ccurcy of qudrture rule, it my not be wise to increse the order indefinitely: f (n) my go up too fst Insted, we fix qudrture of low order, but we chop the intervl [, b] into more nd more, usully equl, prts This leds to composite qudrture rules A modern refinement is dptive qudrture, where the mesh of the prtition is mde to depend on the function f: where f (p) is lrge, the subdivision is mde finer We now list three obvious composite qudrture rules, with their ccurcy, s they follow from the bsic rules given in the previous sections Composite Trpezium Rule Let f : [, b] R be twice continuously differentble, nd let n N Define h := (b )/n nd let T n be the qudrture rule given by Then T n (f) := h ( 1 2 f() + f( + h) + f( + 2h) + + f( + (n 1)h) + 1 2 f(b)) T n(f) fdx 1 12 f (b )3 n 2 Composite Midpoint Rule Let f : [, b] R be twice continuously differentble, nd let n N Define h := (b )/n nd let M n be the qudrture rule given by M n (f) := h ( f( + 1 2 h) + f( + 3 2n 1 2h) + + f( + 2 h) ) 15
Then M n(f) fdx 1 24 f (b )3 n 2 Composite Simpson Rule Let f : [, b] R be four times continuously differentble, nd let n N, h := (b )/n Let S n be the qudrture rule given by S n (f) := (f()+4f(+ 1 6 h 1 2 h)+2f(+h)+4f(+ 3 2n 1 h)+ +2f(+(n 1)h)+4f(+ h)+f(b) 2 2 ) Then S n(f) fdx 1 2880 f (b )5 n 4 Remrk The increse in ccurcy of composite qudrture rule goes s n p, where p is the order of the elementry qudrture rule Exercise 7 Clculte π to 6 digits ccurcy by evluting the integrl 1 () using the composite Trpezium Rule; (b) using the composite Midpoint Rule; (c) using the composite Simpson Rule 0 4 1 + x 2 dx, Wht is in ech cse the miniml vlue of n (number of subintervls) you need? 44 Gussin Qudrture As we hve seen, the Midpoint rule, which is bsed on single point only, is nevertheless of order 2 The Trpezium Rule is bsed on two points, nd lso hs order 2 However, it turns out to be possible to find different pir of points in the intervl, so tht the order of the liner ( trpezium-like ) Lgrnge qudrture goes up to 4 In the sme wy, the Simpson rule, which is bsed on three points, is of order 4, but by clever rerrngement of the supporting points the order cn be rised to 6 Generlly, for ech n N there exists (unique) set of n points inside [, b], such tht the Lgrnge qudrture bsed on those points is of order 2n No higher order is chieveble on n points We shll not prove these sttements here, but refer to the literture Let us just note tht the supporting points used by Guß re the zeroes of the n-th Legendre orthogonl polynomil on [, b] Exercise 8 Determine the weights c 0, c 1 nd c 2 of the Lgrnge qudrture rule on the 3 intervl [ 1, 1], bsed on the points ξ 0 := 5, ξ 3 1 := 0, nd ξ 2 := 5 Wht is the order p of this qudrture rule? (Prove your nswer) Give shrp upper bound for the error of this rule in terms of f (p) nd b 16
5 Ordinry Differentil Equtions In this chpter we shll tret numericl methods for the solution of ordinry differentil equtions, ie differentil equtions in one vrible In sense this is generlistion of the previous chpter on numericl integrtion We re given continuous function f : [, b] R R nd rel number y 0, nd we re looking for solutions ϕ : [, b] R of the differentil eqution ϕ (x) = f(x, ϕ(x)) (4) with initil condition ϕ() = y 0 The feture tht complictes this sitution considerbly in comprison with Chpter 4 is the dependence of f on the second coordinte, ie on ϕ itself Let us first note tht eqution (4) cn lterntively be written s n integrl eqution: ϕ(x) = y 0 + x f(u, ϕ(u)) du (5) Now the new vlue ϕ(x) is expressed in terms of erlier vlues ϕ(u), nd in order to evlute it we my pply the integrtion techniques of Chpter 4 to the right hnd side, keeping in mind tht we should only use dt tht hve lredy been clculted 51 Euler s Method The first ide tht comes to mind is to use first order qudrture This leds to the oldest nd most primitive method, pioneered by Euler For positive integer n we keep the stndrd nottion of lst chpter: h := b n nd x i := + ih, (i = 0,, n) Proposition 9 Let f : [, b] R R be differentible with continuous nd bounded derivtives Define sequence w 0, w 1, w 2,, w n by w 0 := y 0 nd w i+1 := w i + hf(x i, w i ) Then w i is n pproximtion of y i := ϕ(x i ) stisfying where y i w i b 2n M e(xi )L 1 L L := f y nd M := f x + f f y, Of ll the detils in this sttement the most importnt ones re the simple rule by which the Euler pproximtions re produced nd the fctor 1/n in the estimte, which shows tht convergence to the true solution is very slow For the proof we need the following lemm 17
Lemm 10 Let 0, 1, 2, be sequence of positive numbers stisfying For some α, γ > 0 Then for ll i N: This cn be by induction i+1 γ i + β i γ i 0 + β γi 1 γ 1 Proof of Proposition 9 From the generl theory of differentil equtions we know tht (4) llows unique solution, provided tht f is Lipschitz continuous in y for ll x This is gurnteed by the boundedness of f y Note furthermore tht ϕ (x) = df(x, ϕ(x)) dx = f f (x, ϕ(x)) + (x, ϕ(x)) f(x, ϕ(x)), x y so tht ϕ (x) M By Tylor s Theorem we hve, for some ξ (x i, x i+1 ), Therefore so tht ϕ(x i+1 ) = ϕ(x i ) + hϕ (x i ) + 1 2 h2 ϕ (ξ) y i+1 w i+1 = ( y i w i ) + h ( f(xi, y i ) f(x i, w i ) ) + 1 2 h2 ϕ (ξ), y i+1 w i+1 y i w i (1 + hl) + 1 2 h2 M The sttement follows from Lemm 10 if we observe tht (1 + hl) i e (x i )L 52 Runge Kutt Methods of Second Order By using higher order qudrture the pproximtion method cn be inmproved The reltion y i+1 = y i + cn be pproximted by the Anstz where xi +h x i f(u, ϕ(u)) du w i+1 = w i + c 0 k 0 + c 1 k 1, k 0 := hf(x i, w i ) nd k 1 := hf(x i + ξh, w i + ξk 0 ) If we choose the vlues of c 0, c 1 s the weights of Lgrnge qudrture supported by the points 0 nd ξ [0, 1], c 0 = 1 1 2ξ nd c 1 = 1 2ξ, 18
then the bove scheme yields pproximtions w i to ϕ(x i ) tht re correct to second order: w i ϕ(x i ) C n 2 In this second order Runge-Kutt scheme ξ is free prmeter Some prticulr vlues hve cquired specil nmes: ξ = 1 : ξ = 2 3 : ξ = 1 2 : Improved Euler Method; Heun s Method; Midpoint Method 53 Runge Kutt Method of Fourth Order The most populr Runge-Kutt scheme, for mny the Runge Kutt Method, is the following 4-th order method for the solution of eqution (4), to be executed n times on the intervl [, b]: k 0 = hf(x i, w i ) ; k 1 = hf(x i + 1 2 h, w i + 1 2 k 0) ; k 2 = hf(x i + 1 2 h, w i + 1 2 k 1) ; k 3 = hf(x i + h, w i + k 2 ) ; w i+1 = w i + 1 6( k0 + 2k 1 + 2k 2 + k 3 ) It is one of lrge fmily of fourth order methods, which in its turn is prt of hierrchy of methods of ll orders This prticulr one combines reltively simple Simpson-type clcultion bsed on only 4 function evlutions, with fst convergence: w i ϕ(x i ) C n 4 Exercise 9 Consider the differentil eqution ϕ (x) = x ϕ(x) on the intervl [0, 4] with initil condition ϕ(0) = 3 compre it with the following pproximtions: () Euler s Method; (b) Heun s Method; (c) the Midpoint Method; (d) the 4-th order Runge-Kutt scheme Clculte the exct solution nd How mny itertions do you need in ech cse in order to pproximte ϕ(4) to within 10 3? 19
6 Gussin Elimintion nd Mtrix Fctoristion In this chpter we consider solution strtegies for equtions of the form Ax = b, where A is rel n n mtrix nd both b nd the unknown x re column vectors of lenght n The solution exists nd is unique if nd only if A is regulr, ie det(a) = 0 There is vriety of wys to solve this eqution, exctly or pproximtely Crmer s rule, for one, yields n nswer in principle, but is in prctice only useful for smll vlues of n, since the number of opertions required grows s n! Gussin elimintion is much more efficient method, tking roughly 2 3 n3 opertions We shll discuss Gussin elimintion nd some of its refinements in specil cses It is direct method in the sense tht, prt from rounding off errors, it is exct For lrge n, it my be profitble to use n pproximte, itertive method, which in ech step uses bout n 2 opertions This mkes sense if the number of itertions required is of the order of n or less 61 LU-fctoristion We recll tht mtrix L is clled lower tringulr if it is of the form l 11 0 L =, l n1 l nn ie if ll nonzero entries lie on or below the digonl Similrly, we cll mtrix U upper digonl if ll nonzero entries lie on or bove the digonl Clerly, the determinnt of such mtrices is the product of their digonl entries, so tht they re regulr iff these re ll nonzero All regulr lower tringulr mtrices form group L; the regulr upper tringulr mtrices form group U We shll now show tht Gussin elimintion (without row permuttion) is ctully fctoristion into lower nd n upper tringulr mtrix By E ij we denote the n n-mtrix hving 1 t the ij-position, nd 0 everywhere else Theorem 11 (LU-fctoristion) Let A be regulr n n mtrix Suppose tht A cn be trnsformed into n upper tringulr mtrix U by Gussin elimintion Then with A = LU, L = 1l + λ ij E ij, i>j where λ ij is the multiplier used when clening the j-th column with respect to the i-th row (nd 1l denotes the identity mtrix) Proof Let A (1) := A be the originl mtrix (1) 11 (1) 1n (1) n1 (1) nn 20
Let λ i1 := (1) i1 /(1) 11 for 2 i n Then sweeping clen the first column in the downwrd direction is described by the eqution A (2) := (1l λ n1 E n1 ) (1l λ 31 E 31 )(1l λ 21 E 21 )A (1) = Subsequently we define λ i2 := (2) i2 /(2) 22 for 3 i n nd A (3) := (1l λ n2 E n2 ) (1l λ 42 E 42 )(1l λ 32 E 32 )A (2) = (2) 11 (2) 12 (2) 1n 0 (2) 22 (2) 2n 0 (2) n2 (2) nn (3) 11 (3) 12 (3) 13 (3) 1n 0 (3) 22 (3) 23 (3) 2n 0 0 (3) 32 (3) 3n 0 0 (3) n3 (3) nn If this cn be continued without ever hitting on division by zero (ie (k) kk = 0 t some stge k), we eventully rrive t n upper tringulr mtrix U := A (n), which is relted to A by the eqution U = L n 1 L n 2 L 1 A, where L k := (1l λ nk E nk ) (1l λ k+1,k E k+1,k ), (1 k n 1) The rules of clcultion in the group L now yield the fctoristion A = LU, where L = L 1 1 L 1 2 L 1 n 1 is given by the expression in the theorem In generl it my hppen tht in the bove clcultion pivot element (k) kk is found to be zero In tht cse Gussin elimintion requires tht permuttion of rows is pplied in order to put nonzero element in its plce This permuttion of rows is relised by permuttion mtrix P, ie mtrix hving precisely one 1 in every row nd in every column, nd zeroes everywhere else In other words: n rbitrry regulr mtrix A stisfies P A = LU, so it cn be cst in the form A = P LU, where L L, U U nd P P, the group of permuttion mtrices The Mtlb commnd [L,U,P]=lu(A) returns lower tringulr mtrix L, upper tringulr mtrix U, nd permuttion mtrix P such tht P A = LU Remrk Suppose no exchnges of rows re needed in the elimintion process Then A cn be fctorised s A = LU with L L, U U Let us see in how fr this fctoristion is unique If D is regulr digonl mtrix, then L := LD nd U := D 1 U form nother decompostion of A In this wy decompositions cn be constructed hving rbitrry digonl entries for L Conversely, if second decomposition A = L U exists, then LU = L U, so tht U(U ) 1 = L 1 L These products must be t the sme time lower tringulr nd upper tringulr, hence they re given by digonl mtrix D, nd we find L = LD nd U = D 1 U This D is fixed by, for instnce, the requirement tht L hs digonl entries 1 21
Proposition 12 Let A be regulr mtrix Then the following re equivlent 1 A cn be trnsformed by Gussin elimintion into upper tringulr form without exchnge of rows; 2 A = LU for some L L, U U; 3 the determinnt of ech min minor of A is nonzero We omit the proof The decomposition of A s LU costs pproximtely 2 3 n3 opertions 62 Pivoting In numericl nlysis it is sometimes dvisble to pply row exchnge not only if pivot element (k) kk is zero, but lso if it is smll Indeed, division by these smll numbers my cuse uncceptble errors in the clcultion A populr strtegy to void division by smll numbers s much s possible, is column pivoting, ie serching for the lrgest entry in ech column (in bsolute vlue), nd interchnging it with the row whose turn it is to serve s pivot An more rdicl pproch is totl pivoting, looking for the lrgest entry in the whole block still to be treted, nd then interchnging two rows nd two columns in order to put this entry into the next pivot position Ech method hs its dvntges nd disdvntges, nd it is not known if there exists strtegy tht serves best under ll conditions 63 Specil cses of LU-fctoristion Theorem 13 (Sylvester) A symmetric mtrix is strictly positive definite iff the determinnt of ech min minor is strictly positive In prticulr, every strictly positive definite mtrix A hs n LU fctoristion In fct, it hs mny The following theorem sttes, however, tht one of these is prticulrly nice Theorem 14 (Cholesky decomposition) Let A be strictly positive definite n n mtrix Then there exists unique lower tringulr mtrix L such tht A = LL nd l ii > 0 for i = 1,, n We do not prove these theorems from liner lgebr here The Cholesky decomposition is found in pproximtely 1 3 n2 opertions 64 Itertive Methods If n is very lrge, even Gussin elimintion my be too much work In tht cse we my hve to resort to pproximtion of the solution by itertion This requires tht we write our solution s the fixed point of some contrction One such method is tht of Jcobi Definition A complex n n mtrix is clled strictly digonlly dominted if ii > n j=1,j i ij for i = 1, n 22
Lemm 15 A strictly digonlly dominted mtrix is regulr nd hs no zeroes on its digonl Proof The second point is obvious For the first, suppose tht Ax = 0 Then in prticulr (Ax) i = 0, ie ii x i = n j=1,j i ijx j Tking bsolute vlues we find tht ii x n j=1,j i ij x This contrdicts the ssumption unless x = 0, nd therefore x = 0 Theorem 16 (Jcobi Method) Let A be strictly digonlly dominted Write A = D + O, with D digonl nd O off-digonl (ie O ii = 0 for ll i) For given b nd strting point x 0 the sequence x 0, x 1, x 2,, defined by x k+1 := D 1 ( Ox k + b ) converges to the solution of the eqution Ax = b Moreover, if we define ij ϑ := mx i ii, j i then the error in the pproximtion x k stisfies x k x ϑ x k x k 1 1 ϑ Proof The mtrix D 1 O is contrction with norm ϑ, which is less thn 1 by the ssumption of digonl dominnce By similr rgument s in Theorem 1 it follows tht the sequence converges to the fixed point of the mp x D 1( Ox + b ), which is our solution Moreover we hve x k x x j+1 x j ϑ j+1 k x k x k 1 ϑ = x k x k 1 1 ϑ j=k j=k The following refinement of the Jcobi method widens the clss of dmissible A s in useful wy, speeding up the convergence for the A s we lredy hd We stte the theorem without proof Theorem (Guß-Seidel) Suppose tht A is strictly digonlly dominted or strictly positive definite For given b nd strting point x 0 the sequence x 0, x 1, x 2,, defined by x k+1 i = 1 ii i 1 j=1 ij x k+1 j converges to the solution of the eqution Ax = b n j=i+1 ij x k j + b i 23
Exercise 10 Suppose A = P LU, where P is permuttion mtrix, L lower tringulr mtrix with ones on the digonl, nd U upper tringulr () Count the number of opertions needed to compute the fctoristion A = P LU for given mtrix A (b) Count the number of opertions needed to clculte det(a) by fctoring (c) Compute det(a) nd count the number of opertions when A = 0 2 1 4 1 3 1 2 1 3 4 0 0 1 1 1 2 1 2 3 4 2 0 5 1 1 1 3 0 2 1 1 2 1 2 0 Exercise 11 Let A be the mtrix given in exercise 10 Find the solution of the eqution Ax = b, where b = (2, 3, 0, 1, 1, 3) T, in two wys: () By fctoristion; (b) by premultiplying the eqution on both sides with A, nd using Guß-Seidel itertion on the positive definite mtrix B := A A Wht is the number of itertions needed for n ccurcy of 10 3 in ech component? 24