An Introduction to Numerical Analysis. Martin Lotz

Transcription

1 An Introduction to Numericl Anlysis Mrtin Lotz SCHOOL OF MATHEMATICS, THE UNIVERSITY OF MANCHESTER, ALAN TURING BUILDING, OXFORD ROAD, MANCHESTER M39PL E-mil ddress: URL:

2

3 Contents Chpter Introduction Computtionl Complexity Accurcy Chpter Interpoltion 5 Lgrnge Interpoltion 6 Interpoltion Error 7 3 Convergence 9 4 An lterntive form 5 Newton s divided differences Chpter 3 Integrtion nd Qudrture 5 3 The Trpezium Rule 5 3 Simpson s Rule 7 33 Composite integrtion rules 9 Chpter 4 Numericl Liner Algebr 3 4 The Jcobi nd Guss Seidel methods 4 4 Vector nd Mtrix Norms 6 43 Convergence of Itertive Algorithms 3 44 The Condition Number 36 Chpter 5 Nonliner Equtions 4 5 The bisection method 4 5 Newton s method Fixed-point itertions Rtes of convergence 48 Newton s method in the complex plne Newton s method in two dimensions 49 iii

4

5 CHAPTER Introduction Clssicl mthemticl nlysis owes its existence to the need to model the nturl world The study of functions nd their properties, of differentition nd integrtion, hs it origins in the ttempt to describe how things move nd behve With the rise of technology it becme incresingly importnt to get ctul numbers out of formule nd equtions This is where numericl nlysis comes into the scene: to develop methods to mke mthemticl models bsed on continuous mthemtics effective In prctice, one often cnnot simply plug numbers into formule nd get ll the exct results Most problems require n infinite number of steps to solve, but one only hs finite mount of time vilble; most numericl dt lso requires n infinite mount of storge (just try to store π on computer!), but piece of pper or computer only hs so much spce These re some of the resons tht led us to work with pproximtions An lgorithm is sequence of instructions to be crried out by computer (mchine or humn), in order to solve problem There re two guiding principles to keep in mind when designing nd nlysing numericl lgorithms () Computtionl complexity: lgorithms should be fst; () Accurcy: solutions should be good The first spect is due to limited time; the second due to limited spce In wht follows, we discuss these two spects in some more detil Computtionl Complexity An importnt considertion in the design of numericl lgorithms is efficiency; we would like to perform computtions s fst s possible Considerble speed-ups re possible by clever lgorithm design tht ims to reduce the number of rithmetic opertions needed to perform tsk The mesure of computtion time we use is the number of bsic (floting point) rithmetic opertions (+,,,/) needed to solve problem, s function of the input size The input size will be typiclly the number of vlues we need to specify the problem EXAMPLE (Horner s Algorithms) Tke, for exmple, the problem of evluting polynomil p n (x) = + x + x + + n x n for some x R nd given,, n A nive strtegy would be s follows: () Compute x, x,, x n, () Multiply k x k for k =,,n, (3) Add up ll the terms If ech of the x k is computed individully from scrtch, the overll number of multiplictions is n(n+)/ This cn be improved to n multiplictions by computing the powers x k, k n, itertively An even smrter wy, tht lso uses less intermedite storge, cn be derived by observing tht the polynomil cn be written in the following form: p n (x) = + x( + x + 3 x + + n x n ) = + xp n (x) The polynomil in brckets hs degree n, nd once we hve evluted it, we only need one dditionl multipliction to hve the vlue of p(x) In the sme wy, p n (x) cn be written s p n (x) = + xp n (x) for In discrete mthemtics nd combintorics, pproximtion lso becomes necessity, lbeit for different reson, nmely computtionl complexity Mny combintoril problems re clssified s NP-hrd, which mkes them computtionlly intrctble

6 INTRODUCTION polynomil p n (x) of degree n, nd so on This suggests the possibility of recursion, leding to Horner s Algorithm This lgorithm computes sequence of numbers b n = n b n = n + x b n b = + x b, where b turns out to be the vlue of the polynomil evluted t x In prctice, one would not compute sequence but overwrite the vlue of single vrible t ech step The following MATLAB nd Python code illustrtes how the lgorithm cn be implemented Note tht MATLAB encodes the coefficients,, n s vector with entries (),, (n + ) function p = horner(,x) n = length(); p = (n); for k=n : : p = (k)+x*p; end MATLAB def horner(polynomil,x): result = for coefficient in polynomil: result = result*x+coefficient return result Python This lgorithm only requires n multiplictions Horner s Method is the stndrd wy of evluting polynomils on computers Here we re less concerned with the precise numbers, but with the order of mgnitude Thus we will not cre so much whether computtion procedure uses 5n opertions (n is the input size) or n, but we will cre whether the lgorithm needs n 3 s opposed to n log(n) rithmetic opertions to solve problem To conveniently study the performnce of lgorithms we use the big-o nottion Given two functions f (n) nd g (n) tking integer rguments, we sy tht f (n) O(g (n)) or f (n) = O(g (n)), if there exists constnt C > nd n >, such tht f (n) < C g (n) for ll sufficiently lrge n > n For exmple, n log(n) = O(n ) nd n n O(n 3 ) EXAMPLE (*) Consider the problem of multiplying mtrix with vector: Ax = b, where A is n n n mtrix, nd x nd b re n-vectors Normlly, the number of multiplictions needed is n, nd the number of dditions n(n ) (verify this!) However, there re some mtrices, for exmple the one with the n-th roots of unity i j = e πi j /n s entries, for which there re lgorithms (in this cse, the Fst Fourier Trnsform) tht cn compute the product Ax in O(n logn) opertions This exmple is of gret prcticl importnce, but will not be discussed further t the moment An interesting nd chllenging field is lgebric complexity theory, which dels with lower bounds on the number of rithmetic opertions needed to perform certin computtionl tsks It lso sks questions such s whether Horner s method nd other lgorithms re optiml, tht is, cn t be improved upon Accurcy In the erly 9th century, CF Guss, one of the most influentil mthemticins of ll time nd pioneer of numericl nlysis, developed the method of lest squres in order to predict the reppernce of the recently discovered steroid Ceres He ws well wre of the limittions of numericl computing In his own words: Since none of the numbers which we tke out from logrithmic nd trigonometric tbles dmit of bsolute precision, but re ll to certin extent pproximte only, the results of ll clcultions performed by the id of these numbers cn only be pproximtely true From: CF Guss, Theori motus corporum coelestium in sectionibus conicis solem mbientium, 89

7 ACCURACY 3 Mesuring errors To mesure the qulity of pproximtions, we use the concept of reltive error Given quntity x nd computed pproximtion ˆx, the bsolute error is given by E bs ( ˆx) = x ˆx, while the reltive error is given s x ˆx E rel ( ˆx) = x The benefit of working with reltive errors is cler: they re scle invrint On the other hnd, bsolute error cn be meningless For exmple, n error of one hour is irrelevnt when estimting the ge of Stn the Tyrnnosurus rex t Mnchester Museum, but it is crucil when determining the time of lecture Tht is becuse in the former one hour corresponds to reltive error is of the order, while in the ltter it is of the order Floting point nd significnt figures Nowdys, the estblished wy of representing rel numbers on computers is using floting-point rithmetic In the double precision version of the IEEE stndrd for floting-point rithmetic, number is represented using 64 bits 3 A number is written x = ±f e, where f is frction in [,], represented using 5 bits, nd e is the exponent, using bits (wht is the remining 64th bit used for?) Two things re worth noticing bout this representtion: there re lrgest possible numbers, nd there re gps between representble numbers The lrgest nd smllest numbers representble in this form re of the order of ± 38, enough for most prcticl purposes A bigger concern re the gps, which mens tht the results of mny computtions lmost lwys hve to be rounded to the closest floting-point number Throughout this course, when going through clcultions without using computer, we will usully use the terminology of significnt figures (sf) nd work with 4 significnt figures in bse For exmple, in bse, 3 equls 73 to 4 significnt figures To count the number of significnt figures in given number, strt with the first non-zero digit from the left nd, moving to the right, count ll the digits therefter, counting finl zeros if they re to the right of the deciml point For exmple, 48, 4, 48, 4 nd 4 ll hve 5 significnt figures (sf) In rounding or trunction of number to n sf, the originl is replced by the closes number with with n sf An pproximtion ˆx of number x is sid to be correct to n significl figures if both ˆx nd x round to the sme n sf number 4 Note tht finl zeros to left of the deciml point my or my not be significnt: the number 4 hs lest 4 significnt figures, but without ny more informtion there is no wy of knowing whether or not ny more figures re significnt When 397 is rounded to 5 significnt figures to give 4, n explntion tht this hs 5 significnt figures is required This could be mde cler by writing it in scientific nottion: 4 6 To sy tht = 48 to 5 sf mens tht the exct vlue of becomes 48 fter rounding to 5 sf: tht is to sy, EXAMPLE 3 Suppose we wnt to find the solution to the qudrtic eqution The two solutions to this problem re given by x + bx + c = () x = b + b 4c, x = b b 4c In principle, to find x nd x one only needs to evlute the expressions for given,b,c Assume, however, tht we re only llowed to compute to four significnt figures, nd consider the prticulr eqution x + 397x + 3 = Using the formul, we hve, lwys rounding to four significnt figures, =,b = 397,c = 3, b = 5769 = 576 (to 4 sf),4c = 5 (to 4 sf), 3 A bit is binry digit, tht is, either or 4 This definition is not without problems, see for exmple the discussion in Section of Nichols J Highm, Accurcy nd Stbility of Numericl Algorithms, SIAM

8 4 INTRODUCTION b 4c = = 575 (to 4 sf), b 4c = 3969 Hence, the computed solutions (to 4 significnt figures) re given by The exct solutions, however, re x = 5, x = 3969 x = 3748, x = The solution x is completely wrong, t lest if we look t the reltive error: x x = 568 x While the ccurcy cn be incresed by incresing the number of significnt figures during the clcultion, such effects hppen ll the time in scientific computing nd the possibility of such effects hs to be tken into ccount when designing numericl lgorithms Note tht it mkes sense, s in the bove exmple, to look t errors in reltive sense An error of one mile is certinly negligible when deling with stronomicl distnces, but not so when mesuring the length of rce trck By nlysing wht cuses the error it is sometimes possible to modify the method of clcultion in order to improve the result In the present exmple, the problems re being cused by the fct tht b b 4c, nd therefore b + b 4c = cuses wht is clled ctstrophic cncelltion A wy out is provided by the observtion tht the two solutions re relted by () x x = c When b >, the clcultion of x ccording to () shouldn t cuse ny problems, in our cse we get 3969 to four significnt figures We cn then use () to derive x = c/(x ) = 37 3 Sources of errors As we hve seen, one cn get round numericl ctstrophes by choosing clever method for solving problem, rther thn incresing precision So fr we hve considered errors introduced due to rounding opertions There re other sources of errors: () Overflow () Errors in the model (3) Humn or mesurements errors (4) Trunction or pproximtion errors The first is rrely n issue, s we cn represent numbers of order 38 on computer The second two re importnt fctors, but fll outside the scope of this lecture The third hs to do with the fct tht mny computtions re done pproximtely rther thn exctly For computing the exponentil, for exmple, we might use method tht gives the pproximtion e x + x + x As it turns out, mny prcticl methods give pproximtions to the true solution

9 CHAPTER Interpoltion How do we represent continuous function on computer? If f is polynomil of degree n, f (x) = p n (x) = + x + + n x n then we only need to store the n + coefficients,, n Mybe surprisingly, polynomils cn represent lrge vriety of curves, s the following figure illustrtes In fct, one cn pproximte n rbitrry continuous function on bounded intervl by polynomil THEOREM (Weierstrss) For ny f C ([,],R) nd ny ε > there exists polynomil p(x) such tht mx f (x) p(x) ε x In this lecture we begin by studying the problem of polynomil interpoltion Given pirs (x j, y j ) R, j n, with distinct x j, we would like to find polynomil p of smllest possible degree such tht () p(x j ) = y j, j n FIGURE The interpoltion problem EXAMPLE Let h = /n, x =, nd x i = i h for i n The x i subdivide the intervl [,] into segments of equl length h Now let y i = i h/ for i n Then the points (x i, y i ) ll lie on the line p (x) = x/, s is esily verified It is lso esy to see tht p is the unique polynomil of degree t most tht goes through these points In fct, we will see tht it is the unique polynomil of degree t most n tht psses through these points! 5

10 6 INTERPOLATION We will first describe the method of Lgrnge interpoltion, which lso helps to estblish the existence nd uniqueness of n interpoltion polynomil stisfying () We then discuss the qulity of pproximting polynomils by interpoltion, the question of convergence, s well s other methods such s Newton interpoltion Lgrnge Interpoltion The next lemm shows tht it is indeed possible to find polynomil of degree t most n stisfying () We denote by P n the set of ll polynomils of degree t most n Note tht this lso includes polynomils of degree smller thn n, nd in prticulr constnts, since we llow coefficients such s n in the representtion + x + + n x n to be zero LEMMA Let x, x,, x n be distinct rel numbers Then there exist polynomils L k P n such tht { j = k, L k (x j ) = j k Moreover, the polynomil is in P n nd stisfies p n (x j ) = y j for j n p n (x) = n L k (x)y k PROOF Clerly, if L k exists, then it is polynomil of degree n with roots t x j for j k Hence, L k (x) = C k (x x j ) = C k (x x ) (x x j )(x x j + ) (x x n ) j k for constnt C k To determine C k, set x = x k Then L k (x k ) = = C k j k(x k x j ) nd therefore C k = k= j k(x k x j ) Note tht we ssumed the x j to be distinct, otherwise we would hve to divide by zero nd cuse disster We therefore get the representtion j k(x x j ) L k (x) = j k(x k x j ) This proves the first clim Now set p n (x) := n y k L k (x) k= Then p ( x j ) = n k= y kl k (x j ) = y j L j (x j ) = y j Since p n (x) is liner combintions of the vrious L k, it lives in P n This completes the proof We hve shown the existence of n interpolting polynomil We next show tht this polynomil is uniquely determined The importnt ingredient is the Fundmentl Theorem of Algebr, version of which sttes tht A polynomil of degree n with complex coefficients hs exctly n complex roots THEOREM (Lgrnge Interpoltion Theorem) Let n Let x j, j n, be distinct rel numbers nd let y j, j n, be ny rel numbers Then there exists unique p n (x) P n such tht () p n (x j ) = y j, j n PROOF The cse n = is cler, so let us ssume n In Lemm we constructed polynomil p n (x) of degree t most n stisfying the conditions (), proving the existence prt For the uniqueness, ssume tht we hve two such polynomils p n (x) nd q n (x) of degree t most n stisfying the interpolting property () The gol is to show tht they re the sme By ssumption, the difference p n (x) q n (x) is polynomil of degree t most n tht tkes on the vlue p n (x j ) q n (x j ) = y j y j = t the n + distinct x j, j n By the Fundmentl Theorem of Algebr, non-zero polynomil of degree n cn hve no more thn n distinct rel roots, from which it follows tht p n (x) q n (x), or p n (x) = q n (x) This concludes the proof

11 INTERPOLATION ERROR 7 DEFINITION Given n + distinct rel numbers x j, j n, nd n + rel numbers y j, j n, the polynomil n p n (x) = L k (x)y k k= is clled the Lgrnge interpoltion polynomil of degree n corresponding to the dt points (x j, y j ), j n If the y k re the vlues of function f, tht is, if f (x k ) = y k, k n, then p n (x) is clled the Lgrnge interpoltion polynomil ssocited to f nd x,, x n (*) A different tke on the uniqueness problem cn be rrived t by trnslting the problem into liner lgebr one For this, note tht if p n (x) = + x + + n x n, then the polynomil evlution problem t the x j, j n, cn be written s mtrix vector product: y x x n y y n = x x n x n xn n or y = X If the mtrix X is invertible, then the interpolting polynomil is uniquely determined by the coefficient vector = X y The mtrix X is invertible if nd only if det(x ) The determinnt of X is the well-known Vndermonde determinnt: x x n x x n det(x ) = det = (x j x i ) j >i x n xn n Clerly, this determinnt is different from zero if nd only if the x j re ll distinct, which shows the importnce of this ssumption EXAMPLE Consider the function f (x) = e x on the intervl [,], with interpoltion points x =, x =, x = The Lgrnge bsis functions re given by n, L (x) = (x x )(x x ) (x x )(x x ) = x(x ), L (x) = x, L (x) = x(x + ) The Lgrnge interpoltion polynomil is therefore given by p (x) = x(x )e + ( x )e + x(x + )e = + x sinh() + x (cosh() ) Interpoltion Error If the dt points (x j, y j ) come from function f (x), tht is, if f (x j ) = y j, then the Lgrnge interpolting polynomil cn look very different from the originl function It is therefore of interest to hve some control over the interpoltion error f (x) p n (x) Clerly, without ny further ssumption on f the difference cn be rbitrry We will therefore restrict to function f tht re sufficiently smooth Wht this mens is codified in the following definition DEFINITION Define C k ([,b],r) to be the set of functions f : [,b] R tht hve k continuous derivtives on [,b] EXAMPLE 3 All polynomils belong to C k ([,b],r) for ll bounded intervls [,b] nd ny integer k However, f (x) = /x C ([, ], R), s f (x) for x nd the function is therefore not continuous there Now tht we hve estblished the existence nd uniqueness of the interpoltion polynomil, we would like know how well it pproximtes the function

12 8 INTERPOLATION FIGURE Lgrnge interpoltion of e x t,, THEOREM 3 Let n nd ssume f C n+ ([,b],r) Let p n (x) P n be the Lgrnge interpoltion polynomil ssocited to f nd distinct x j, j n Then for every x [,b] there exists ξ = ξ(x) (,b) such tht (3) f (x) p n (x) = f (n+) (ξ) π n+ (x), (n + )! where π n+ (x) = (x x ) (x x n ) For the proof of Theorem 3 we need the following consequence of Rolle s Theorem LEMMA Let f C n ([,b],r), nd suppose f vnishes t n + points x,, x n Then there exists ξ (,b) such tht the n-th derivtive f (n) (x) stisfies f (n) (ξ) = PROOF By Rolle s Theorem, for ny two x i, x j there exists point in between where f vnishes, therefore f vnishes t (t lest) n points Repeting this rgument, it follows tht f (n) vnishes t some point ξ (,b) PROOF (Theorem 3) Assume x x j for j n (otherwise the theorem is clerly true) Define the function ϕ(t) = f (t) p n (t) f (x) p n(x) π n+ (t) π n+ (x) This function vnishes t n + distinct points, nmely t = x j, j n, nd x Assume n > (the cse n = is left s n exercise) By Lemm, the function ϕ (n+) hs zero ξ (,b), while the (n + )-st derivtive of p n vnishes (since p n is polynomil of degree n) We therefore hve from which we get This completes the proof = ϕ (n+) (ξ) = f (n+) (ξ) f (x) p n(x) (n + )!, π n+ (x) f (x) p n (x) = f (n+) (ξ) π n+ (x) (n + )!

13 3 CONVERGENCE 9 Theorem 3 contins n unspecified number ξ Even though we cn t find this loction in prctise, the sitution is not too bd s we cn sometimes upper-bound the n + -st derivtive of f on the intervl [,b] COROLLARY Under the conditions s in Theorem 3, where so tht f (x) p n (x) M n+ (n + )! π n+(x), M n+ = mx x b f (n+) (x) PROOF While we don t hve the ξ in the expression 3, by ssumption we do hve the bound This completes the proof f (n+) (ξ) M n+, f (x) p n (x) f (n+) (ξ)π n+ (x) (n + )! M π n+ (x) n+ (n + )! EXAMPLE 4 Suppose we would like to pproximte f (x) = e x by n interpolting polynomil p P t points x, x [,] tht re seprted by distnce h = x x Wht h should we choose to chieve From Corollry, we get p (x) e x 5, x x x p (x) e x M π (x), where M = mx x x x f () (x) e (becuse x, x [,] nd f () (x) = e x ) nd π (x) = (x x )(x x ) To find the mximum of π (x), first write x = x + θh, x = x + h for θ [,] Then π (x) = θh(h θh) = h θ( θ) By tking derivtives with respect to θ we find tht the mximum is ttined t θ = / Hence, π (x) h ( ) = h 4 We conclude tht p (x) e x h e 8 In order to chieve tht this flls below 5 we require tht h 8 5 /e = This gives informtion on how big the spcing of points needs to be for liner interpoltion to chieve certin ccurcy 3 Convergence For given set of points x,, x n nd function f, we hve bound on the interpoltion error Is it possible to mke the error smller by dding more interpoltion points, or by modifying the distribution of these points? The nswer to this question cn depend on two things: the clss of functions considered, nd the spcing of the points Let p n (x) denote the Lgrnge interpoltion polynomil of degree n for f t the points x,, x n The question we sk is whether lim mx p n(x) f (x) n x b Perhps surprisingly, the nswer is negtive, s the following fmous exmple, known s the Runge Phenomenon, shows EXAMPLE 5 Consider the intervl [,b] nd let x j = + j (b ), j n, n be n + uniformly spced points on [,b] Consider the function f (x) = + 5x

14 INTERPOLATION on the intervl [,] This function is smooth nd it ppers unlikely to cuse ny trouble However, when interpolting t vrious equispced points for incresing n, we see tht the interpoltion error seems to increse The reson for this phenomenon lies in the behviour of the complex function z /( + z ) The problem is not one of the interpoltion method, but hs to do with the spcing of the points EXAMPLE 6 Let us revisit the function /( + 5x ) nd try to interpolte it t Chebyshev points: x j = cos(j π/n), j n Clculting the interpoltion error for this exmple shows completely different result s the previous exmple In fct, plotting the error nd compring it with the cse of equispced points shows tht choosing the interpoltion points in clever wy cn be huge benefit Equl spcing Chebyshev spcing FIGURE 3 Interpoltion error for equispced nd Chebyshev points To summrise, we hve the following two observtions

15 5 NEWTON S DIVIDED DIFFERENCES () To estimte the difference f (x) p n (x) we need ssumptions on the function f, for exmple, tht it is sufficiently smooth () The loction of the interpoltion points x j, j n, is crucil Equispced points cn led to unplesnt results! 4 An lterntive form The representtion s Lgrnge interpoltion polynomil p n (x) = n L k (x)f (x k ) k= hs some drwbcks On the one hnd, it requires O(n ) opertions to evlute Besides this, dding new interpoltion points requires the reclcultion of the Lgrnge bsis polynomils L k (x) Both of these problems cn be remedied by rewriting the Lgrnge interpoltion formul Provided x x j for j n, the Lgrnge interpoltion polynomil cn be written s (4) p(x) = n w k k= x x f (x k k ) n w k, k= x x k where w k = / j k(x k x j ) re clled the brycentric weights Once the weights hve been computed, the evlution of this form only tkes O(n) opertions, nd updting it with new weights lso only tkes O(n) opertions To derive this formul, define L(x) = n k= (x x k) nd note tht p(x) = L(x) n k= w k/(x x k )f (x k ) Noting lso tht = n k= L k(x) = L(x) n k= w k/(x x k ) nd dividing by this intelligent one, Eqution (4) follows Finlly, it cn be shown tht the problem of computing the brycentric Lgrnge interpoltion is numericlly stble t points such s Chebyshev points 5 Newton s divided differences While the interpoltion polynomil of degree t most n for function f nd n+ points x,, x n is unique, it cn pper in different forms The one we hve seen so fr is the Lgrnge form, where the polynomil is given s liner combintion of the Lgrnge bsis functions: p(x) = n L k (x)y k, k= or some modifictions of this form, such s the brycentric form (see Lecture 5) A different wy of representing the interpoltion polynomil is by writing it in the form (5) p(x) = + (x x ) + + n (x x ) (x x n ) Provided we hve the coefficients,, n, evluting the polynomil only requires n multiplictions using Horner s Method Moreover, it is esy to dd new points: if x n+ is dded, the coefficients,, n don t need to be chnged We ll describe method of computing the coefficients,, n using divided differences The divided differences ssocited to the function f nd distinct x,, x n R re defined recursively s f [x i ] := f (x i ), f [x i, x i+ ] := f [x i+] f [x i ] x i+ x i, f [x i, x i+,, x i+k ] : = f [x i+, x i+,, x i+k ] f [x i, x i+,, x i+k ] x i+k x i

16 INTERPOLATION The divided differences cn be computed from divided difference tble, where we move from one column to the next by pplying the rules bove (here we use the shorthnd f i := f (x i )): x f f [x, x ] x f f [x, x, x ] f [x, x ] f [x, x, x, x 3 ] x f f [x, x, x 3 ] f [x, x 3 ] x 3 f 3 From this tble we lso see tht dding new pir (x n+, f n+ ) would require n updte of the tble tht tkes O(n) opertions THEOREM 4 Let x,, x n be distinct points Then the interpoltion polynomil for f t points x i,, x i+k is given by p i,k (x) = f [x i ] + f [x i, x i+ ](x x i ) + f [x i, x i+, x i+ ](x x i )(x x i+ ) + + f [x i,, x i+k ](x x i ) (x x i+k ) In prticulr, the coefficients in Eqution (5) re given by the divided differences k = f [x,, x k ], nd the interpoltion polynomil p n (x) cn therefore be written s p n (x) = p,n (x) = f [x ] + f [x, x ](x x ) + f [x, x, x ](x x )(x x ) + + f [x,, x n ](x x ) (x x n ) Before going into the proof, observe tht the divided difference f [x,, x n ] is the highest order coefficient, tht is, the coefficient of x n, of the interpoltion polynomil x n This observtion is crucil in the proof PROOF The proof is by induction on k For the cse k = we hve p i, (x) = f [x i ] = f (x i ), which is the unique interpoltion polynomil of degree t (x i, f (x i ), so the clim is true in this cse Assume the sttement holds for k >, which mens tht the interpoltion polynomil p i,k (x) for the pirs (x i, f (x i )),,(x i+k, f (x i+k )) is given s in the theorem We cn now choose vlue k+ such tht (6) p i,k+ (x) = p i,k (x) + k+ (x x i )(x x i+k ) interpoltes f t x i,, x i+k+ In fct, note tht p i,k+ (x j ) = f (x j ) for i j i + k, so tht we only require k+ to be chosen so tht p i,k+ (x i+k+ ) = f (x i+k+ ) Moreover, note tht k+ is the coefficient of highest order of p i,k+ (x), tht is, we cn write p i,k+ (x) = k+ x k+ + lower order terms The only thing tht needs to be shown is tht k+ = f [x i,, x i+k+ ] For this, we define new polynomil q(x), show tht it coincides with p i,k+ (x) by being the unique interpoltion polynomil of f t x i,, x i+k+, nd then show tht the highest order coefficient of q(x) is precisely f [x i,, x i+k+ ] Define (7) q(x) = (x x i )p i+,k (x) (x x i+k+ )p i,k (x) x i+k+ x i This polynomil hs degree k +, just like p i,k+ (x) Moreover: q(x i ) = p i,k (x i ) = f (x i ) q(x i+k+ ) = p i+,k (x i+k+ ) = f (x i+k+ ) q(x j ) = (x j x i )f (x j ) (x j x i+k+ )f (x j ) x i+k+ x i = f (x j ), i + j i + k This mens tht q(x) lso interpoltes f t x i,, x i+k+, nd by the uniqueness of the interpoltion polynomil, must equl p i,k+ (x) Let s now compre the coefficients of x k+ in both polynomils The coefficient of x k+ in p i,k+ is k+, s cn be seen from (6) By the induction hypothesis, the polynomils p i+,k (x) nd p i,k (x) hve the form p i+,k (x) = f [x i+,, x k+i+ ]x k + lower order terms, p i,k (x) = f [x i,, x k+i ]x k + lower order terms

17 5 NEWTON S DIVIDED DIFFERENCES 3 By plugging into (7), we see tht the coefficient of x k+ in q(x) is f [x i+,, x i+k+ ] f [x i,, x i+k ] x i+k+ x i = f [x i,, x i+k+ ] This coefficient hs to equl k+, nd the clim follows EXAMPLE 7 Let s find the divided difference form of cubic interpoltion polynomil for the points The divided difference tble would look like (8) (,),(,),(3,8),(, 39) j x j f j f [x j, x j + ] f [x j, x j +, x j + ] f [x, x, x, x 3 ] 6 3 ( ) = = ( ) = = = 8 The coefficients j = f [x,, x j ] re given by the upper digonl, the interpoltion polynomil is thus p 3 (x) = + (x x ) + (x x )(x x ) + 3 (x x )(x x )(x x ) = + 5x(x + ) + 7x(x + )(x 3) Now suppose we dd nother dt point (4,8) This mounts to dding only one new term to the polynomil The new coefficient 4 = f [x,, x 4 ] is clculted by dding new line t the bottom of Tble (8) s follows: f 4 = 8, f [x 3, x 4 ] = 4, f [x, x 3, x 4 ] = 96, f [x,, x 4 ] =, f [x,, x 4 ] = 4 = 3 The updted polynomil is therefore p 4 (x) = + 5x(x + ) + 7x(x + )(x 3) + 3x(x + )(x 3)(x + ) Evluting this polynomil cn be done conveniently using Horner s method, using only four multiplictions + x( + (x + )(5 + (x 3)(7 + 3(x + )))), Another thing to notice is tht the order of the x i plys role in ssembling the Newton interpoltion polynomil, while the order did not ply role in Lgrnge interpoltion Recll the chrcteristion of the interpoltion polynomil in terms of the Vndermonde mtrix from Week The coefficients i of the Newton divided difference form cn lso be derived s the solution of system of liner equtions, this time in convenient tringulr form: f f x x f f n = x x (x x )(x x ) x n x (x n x )(x n x ) j <n(x n x j ) n,

18

19 CHAPTER 3 Integrtion nd Qudrture We re interested in the problem of computing n integrl f (x) d x If possible, one cn compute the ntiderivtive F (x) (the function such tht F (x) = f (x)) nd obtin the integrl s F (b) F () However, it is not lwys possible to compute the ntiderivtive, s in the cses e x d x, π cos(x ) d x More prominently, the stndrd norml (or Gussin) probbility distribution mounts to evluting the integrl z e x d x, π which is not possible in closed form Even if it is possible in principle, evluting the ntiderivtive my not be numericlly the best thing to do The problem is then is to pproximte such integrls numericlly s good s possible 3 The Trpezium Rule The Trpezium Rule seeks to pproximte the integrl, interpreted s re under the curve, by the re of trpezium defined by the grph of the function One step Trpezium Rule

20 6 3 INTEGRATION AND QUADRATURE Assume we wnt to pproximte the integrl between nd b, nd tht b = h Then the trpezium pproximtion is given by f (x) d x I (f ) = h (f () + f (b)), s cn be esily verified The Trpezium Rule cn be interpreted s integrting the liner interpolnt of f t the points x = nd x = b The liner interpolnt is given s p (x) = x b b f () + x b f (b) Integrting this function gives rise to the representtion s re of trpezium: p (x) d x = h (f () + f (b)) Using the interpoltion error, we cn derive the integrtion error for the Trpezium Rule We clim tht f (x) d x = p (x) d x h3 f (ξ) for some ξ (,b) To derive this, recll tht the interpoltion error is given by (x )(x b) f (x) = p (x) + f (ξ(x))! for some ξ (,b) We cn therefore write the integrl s f (x) d x = p (x) d x + By the Integrl Men Vlue Theorem, there exists ξ (,b) such tht Using integrtion by prts, we get (x )(x b)f (ξ(x)) d x = f (ξ) (x )(x b) d x = For the whole expression we therefore get (x )(x b)f (ξ(x)) d x [ (x ) ] b (x b) = 6 ( b)3 = 6 h3 (x )(x b) d x (x b) d x s climed f (x) d x = p (x) d x h3 f (ξ), EXAMPLE 3 Let s compute the integrl + x d x The ntiderivtive is ln(+ x), so we get the exct expression ln(5) 453 for this integrl Using the Trpezium Rule we get I = (f () + f ()) = ( + ) = 5 3 = 467 to four significnt figures The trpezium rule is n exmple of qudrture rule A qudrture rule seeks to pproximte n integrl s weighted sum of function vlues n f (x) d x w k f (x k ), where the x k re the qudrture nodes nd the w k re clled the qudrture weights k=

21 3 SIMPSON S RULE 7 3 Simpson s Rule Simpson s rule uses three points x =, x = b, nd x = ( + b)/ to pproximte the integrl If h = (b )/, then it is defined by f (x) d x I (f ) = h 3 (f (x ) + 4f (x ) + f (x )) EXAMPLE 3 For the function f (x) = /( + x) from to, Simpson s rule gives the pproximtion I (f ) = ( ) This is much closer to the true vlue 455 thn wht the trpezium rule provides EXAMPLE 33 For the function f (x) = 3x x + nd intervl, (tht is, h = 5), Simpson s rule gives the pproximtion I (f ) = ( + 4(3 4 6 ) + ) + 3 = 3/ The ntiderivtive of this polynomil is x 3 x / + x, so the true integrl is / + = 3/ In this cse, Simpson s rule gives the exct vlue of the integrl! As we will see, this is the cse for ny qudrtic polynomil Simpson s rule is specil cse of Newton-Cotes qudrture rule A Newton-Cotes scheme of order n uses the Lgrnge bsis functions to construct the interpoltion weights Given nodes x k = + kh, k n, where h = (b )/n, the integrl is pproximted by the integrl of the Lgrnge interpolnt of degree n t these points If n p n = L k f (x k ), then f (x) d x I n (f ) := k= p n (x) d x = n w k f (x k ), where w k = L k(x) d x We now show tht Simpson s rule is indeed Newton-Cotes rule of order Let x =, x = b nd x = ( + b)/ Define h := x x = (b )/ The qudrtic interpoltion polynomil is given by We clim tht p = (x x )(x x ) (x x )(x x ) f (x ) + (x x )(x x ) (x x )(x x ) f (x ) + (x x )(x x ) (x x )(x x ) f (x ) I (f ) = k= p (x) d x = h 3 (f (x ) + 4f (x ) + f (x )) To show this, we mke use of the identities x = x +h, x = x +h, to get the representtion (we use f i := f (x i )) p = f h (x x )(x x ) + f h (x x )(x x ) + f h (x x )(x x ) Using integrtion by prts or otherwise, we cn evlute the integrl x Using this, we get x (x x )(x x ) d x = 3 h3, x x (x x )(x x ) d x = 4 3 h3, x x p d x = h 3 (f + 4f + f ) x x (x x )(x x ) d x = 3 h3 This shows the clim As with the Trpezium rule, we cn bound the error for Simpson s rule THEOREM 3 Let f C 4 ([,b],r), h = (b )/ nd x =, x = x + h, x = b Then there exists ξ (,b) such tht E(f ) := f (x) d x h ( f (x ) + 4f (x ) + f (x ) ) = h5 3 9 f (4) (ξ)

22 8 3 INTEGRATION AND QUADRATURE Note tht is some plces in the literture, the bound is written in terms of (b ) s This is equivlent, noting tht h = (b )/ (b )5 E(f ) = 88 f (4) (ξ) PROOF The proof is bsed on Chpter 7 of Süli nd Myers, An Introduction to Numericl Anlysis Consider the chnge of vrible Define F (t) = f (x(t)) Then d x = hd t, nd x x(t) = x + ht, t [,] x f (x) d x = h F (τ) dτ In terms of this function, the integrtion error is written s f (x) d x h ( 3 (f + 4f + f ) = h F (τ) dτ ) 3 (F ( ) + 4F () + F ()) Define t G(t) = F (τ) dτ t (F ( t) + 4F () + F (t)) t 3 for t [,] In prticulr, hg() is the integrtion error we re trying to estimte Consider the function H(t) = G(t) t 5 G() Since H() = H() =, by Rolle s Theorem there exists ξ (,) such tht H (ξ) = Since lso H () =, there exists ξ (,) such tht H () (ξ ) = Since lso H () () = H (3) =, we cn pplying Rolle s Theorem repetedly to find tht there exists µ (,) such tht H (3) (µ) = Note tht the third derivtive of G is given by G (3) (t) = t 3 (F (3) (t) F (3) ( t)), from which it follows tht We cn rewrite this eqution s H (3) (µ) = µ 3 (F (3) (µ) F (3) ( µ)) 6µ G() = 3 µ F (3) (µ) F (3) ( µ) (µ ( µ) = 3 µ 9G() Using tht µ, we cn divide both sides by µ /3 By the Men Vlue Theorem there exists ξ ( µ,µ) such tht 9G() = F (4) (ξ), from which we get for the error (fter multiplying with h), hg() = h 9 F (4) (ξ) Now note tht, using the substitution x = x + th we did t the beginning, F ( 4)(t) = d 4 This finishes the (slightly tricky) proof d t 4 f (x) = d 4 d t 4 f (x + ht) = h 4 f (4) (x) From this derive the error bound E (f ) = f (x) d x I (f ) 9 h5 M 4, where M 4 is n upper bound on the bsolute vlue of the fourth derivtive on the intervl [,b]

23 33 COMPOSITE INTEGRATION RULES 9 One step Trpezium Rule One step Simpson Rule Composite integrtion rules The trpezium rule uses only two points to pproximte n integrl, certinly not enough for most pplictions There re different wys to mke use of more points nd function vlues in order to increse precision One wy, s we hve just seen with the Newton-Cotes scheme nd Simpson s rule, is to use higher-order interpolnts A different direction is to subdivide the intervl into smller intervls nd use lower-order schemes, like the trpezium rule, on these smller intervls For this, we subdivide the integrl f (x) d x = n j = x j + x j f (x) d x, where x =, x j = + j h for j n The composite trpezium rule pproximtes ech of these integrls using the trpezium rule: f (x) d x h (f (x ) + f (x )) + h (f (x ) + f (x )) + + h (f (x n ) + f (x n )) ( = h f (x ) + f (x ) + f (x n ) + ) f (x n) EXAMPLE 34 Let s look gin t the function f (x) = /(+x) nd pply the composite trpezium rule with h = on the intervle [,] (tht is, with n = ) Then ( + x d x ) = Recll tht the exct integrl ws 455, nd tht Simpson s rule lso gven n pproximtion of 456 THEOREM 3 If f C (,b) nd = x < < x n = b, then there exists µ (,b) such tht ( f (x) d x = h f (x ) + f (x ) + + f (x n ) + ) f (x n) h (b )f (µ) In prticulr, if M = mx x b f (x), then the bsolue error is bounded by h (b )M PROOF Recll from the error nlysis of the trpezium rule tht, for every j nd some ξ j (x j, x j + ), n ( h f (x) d x = (f (x j ) + f (x j + )) ) h3 f (ξ j ) j = = h ( f (x ) + f (x ) + + f (x n ) + f (x n) ) n h3 f (ξ j ) j =

24 3 INTEGRATION AND QUADRATURE Clerly the vlues f (ξ j ) lie between the minimum nd mximum of f on the intervl (,b), nd so their verge is lso bounded by min f (x) n f (ξ j ) mx x [,b] n f (x) x [,b] j = Since the function f is continuous on [,b], it ssumes every vlue between the minimum nd the mximum, nd in prticulr lso the vlue given by the verge bove (this is the sttement of the Intermedite Vlue Theorem) In other words, there exists µ (,b) such tht the verge bove is ttined: Therefore we cn write the error term s n n j = f (ξ j ) = f (µ) n h3 f (ξ j ) = h (nh)f (µ) = h (b )f (µ), j = where we used tht h = (b )/n This is the climed expression for the error EXAMPLE 35 Consider the function f (x) = e x /x nd the integrl e x x d x Wht choice of prmeter h will ensure tht the pproximtion error of the composite trpezium rule will by below 5? Let M denote n upper bound on the second derivtive of f (x) The pproximtion error for the composite trpezium rule with steplength h is bounded by We cn find out M by clculting the derivtives of f : E(f ) (b )h M f (x) = e x ( x + x ) f (x) = e x ( x + x + x 3 ) The second derivtive f (x) hs mximum t x =, nd the vlue is M 84 In the intervl [,] we therefore hve the bound E(f ) 84h ( ) = 533h If we choose, for exmple, h = 5 (this corresponds to tking n = steps), then the error is bounded by To derive the composite version of Simpson s rule, we subdivide the intervl [, b] into m intervls nd set h = (b )/m, x j = + j h, j m Then, for j m, f (x) d x = m j = xj x j f (x) d x Applying Simpson s rule to ech of the integrls, we rrive t the expression h ( f (x ) + 4f (x ) + f (x ) + 4f (x 3 ) + + 4f (x m 3 ) + f (x m ) + 4f (x m ) + f (x m ) ), 3 where the coefficients of the f (x i ) lternte between 4 nd for i m Using n error nlysis similr to the cse of the composite trpezium rule, one obtins n error bound b 8 h4 M 4, where M 4 is n upper bound on the bsolute vlue of the fourth derivtive of f on [,b]

25 33 COMPOSITE INTEGRATION RULES EXAMPLE 36 Hving n error of order h mens tht, every time we hlve the stepsize (or, equivlently, double the number of points), the error decreses by fctor of 4 More precisely, for n = k we get n error E(f ) k Looking t the exmple function f (x) = /( + x) nd pplying the composite Trpezium rule, we get the following reltionship between the logrithm of the number of points logn nd the logrithm of the error The fitted line hs slope of 9935, s expected from the theory 6 Error for composite trpezium rule 8 log of error log of number of steps Summrising, we hve seen the following integrtion schemes with their corresponding error bounds: Trpezium: (b )3 M Composite Trpezium: h (b )M Simpson: 88 (b )5 M 4 Composite Simpson: 8 h4 (b )M 4 Note tht we expressed the error bound for Simpson s rule in terms of (b ) rther thn h = (b )/ The h in the bounds for the composite rules corresponds to the distnce between ny two nodes, x j + x j We conclude the section on chpter with definition of the order of precision of qudrture rule DEFINITION 3 A qudrture rule I (f ) hs degree of precision k, if it evlutes polynomils of degree t most k exctly Tht is, I (x j ) = x j d x = j + (b j + j + ), j k For exmple, it is esy to show tht the Trpezium rule hs degree of precision (it evlutes nd x exctly), while Simpson s rule hs degree of precision 3 (rther thn s expected!) In generl, Newton-Cotes qudrture of degree n hs degree of precision n if n is odd, nd n + if n is even

26

27 CHAPTER 4 Numericl Liner Algebr Problems in numericl nlysis cn often be formulted in terms of liner lgebr For exmple, the discretiztion of prtil differentil equtions leds to problems involving lrge systems of liner equtions The bsic problem in liner lgebr is to solve system of liner equtions (4) Ax = b, where n A = m mn is n m n mtrix with rel numbers s entries, nd x x =, b = x n b b m re vectors We will often del with the cse m = n There re two min clsses of methods for solving such systems () Direct methods ttempt to solve (47) using finite number of opertions An exmple is the wellknown Gussin elimintion lgorithm () Itertive methods generte sequence x, x, of vectors in the hope tht x k converges to solution x of (47) s k Direct methods generlly work well for dense mtrices nd modertely lrge n Itertive methods work well with sprse mtrices, tht is, mtrices with very few non-zero entries i j, nd lrge n EXAMPLE 4 Consider the differentil eqution u xx = f (x) with boundry conditions u() = u() =, where u is twice differentible function on [,], nd u xx = u/ x denotes the second derivtive in x We cn discretize the intervl [,] by setting x = /(n + ), x j = j x, nd u j := u(x j ), f j := f (x j ) The second derivtive cn be pproximted by finite difference u xx u i u i + u i+ ( x) At ny one point x j, the differentil eqution thus trnsltes to u i u i + u i+ ( x) = f j 3

28 4 4 NUMERICAL LINEAR ALGEBRA Mking use of the initil conditions u() = u() =, we get the system of equtions u u f u f 3 ( x) u f 3 4 = u n f n f u n n The mtrix is very sprse, it hs only 3n non-zero entries, out of n possible! This form is typicl for mtrices rising from prtil differentil equtions, nd is well-suited for itertive methods tht exploit the specific structure of the mtrix 4 The Jcobi nd Guss Seidel methods In the following, we ssume our mtrices to be squre (m = n) We will lso use the following bit of nottion: upper indices re used to identify individul vectors in sequence of vectors, for exmple, x, x,, x i Individul entries of vector x re denoted by x i, so, for exmple, the i -th entry of k-th vectors would be written s x (k) or simply x k i i A templte for itertively solving liner system cn be derived s follows Write the mtrix A s difference A = A A, where A nd A re somewht simpler to hndle thn the originl mtrix Then the system of equtions Ax = b cn be written s A x = A x + b This motivtes the following pproch: strt with vector x nd successively compute x k+ from x k by solving the system (4) A x k+ = A x k + b Note tht t fter the k-th step, the right-hnd side is known, while the unknown to be found is the vector x k+ on the left-hnd side 4 Jcobi s Method Decompose the mtrix A s A = L + D +U, where 3 (n ) n 3 (n ) n 3 3 3(n ) 3n L =, U = (n ) (n ) (n )3 (n )n n n n3 n(n ) re the lower nd upper tringulr prts, nd D = dig(,, nn ) := nn is the digonl prt Jcobi s method chooses A = D nd A = (L +U) The corresponding itertion (48) then looks like Dx k+ = (L +U )x k +b Once we know x k, this is prticulrly simple system of equtions: the mtrix on the left is digonl! Solving for x k+, (43) x k+ = D (b (L +U )x k ) Note tht since D is digonl, it is esy to invert: just invert the individul entries

29 4 THE JACOBI AND GAUSS SEIDEL METHODS 5 EXAMPLE 4 For concrete exmple, tke the following mtrix with its decomposition in digonl nd off-digonl prt: ( ) ( ) ( ) A = = + Since we get the itertion scheme ( ) ( ) / = = / x k+ = ( ), ( ) / x k + / b We cn lso write the itertion (43) in terms of individul entries If we denote ie, we write x (k) i x (k) x k := x (k) n for the i -th entry of the k-th iterte, then the itertion (43) becomes ( ) (44) x (k+) i =, b i i j x (k) j j i for i n Let s try this out with b = (,), to see if we get solution Let x = to strt with Then ( ) ( x / ) ( ) = + = / ( ) ( x / ) ( )( = x + / ) ( ) ( 3 ) = + = 4 3 / / 4 ( ) ( x 3 / ) ( )( = x + / 3 ) ( ) ( 7 ) = = 8 7 / / 4 8 We see pttern emerging: in fct, one cn show (do this!) tht in this exmple, ( x k k) = k In prticulr, s k the vectors x k pproch (,), which is esily verified to be solution of Ax = b 4 Guss-Seidel In the Guss-Seidel method, we use different decomposition, nmely A = D + L nd A = U This leds to the following system (45) (D + L)x k+ = Ux k + b Though the right-hnd side is not digonl, s in the Jcobi method, the system is still esily solved for x k+ when x k is given To derive the entry-wise formul for this method, we tke closer look t (45) x (k+) b n x (k) x (k+) = b n x (k) n n nn b n x (k+) n Writing out the equtions, we get ( ) x (k+) = b x (k) + + n x n (k) ( ) x (k+) + x (k+) = b 3 x (k) n x n (k) ( ) i x (k+) + + i i x (k+) = b i i i i+ x (k) i+ + + i n x n (k) x (k) n

30 6 4 NUMERICAL LINEAR ALGEBRA Rerrnging this, we get the formul x (k+) i = i i ( b i i j x (k+) j i j x (k) j j <i j >i for the k + -th iterte of x i Note tht in order to compute the k + -th iterte of x i, we lredy use vlues of the k + -th iterte of x j for j < i Note how this differs from (44), where we only resort to the k-th iterte Both methods hve their dvntges nd disdvntges While Guss-Seidel my require less storge (we cn overwrite ech x (k) i by x (k+) i s we don t need the old vlue subsequently), Jcobi s method cn be used esier in prllel (tht is, ll the x (k+) i cn be computed by different processors for ech i ) EXAMPLE 43 Consider simple system of the form x = Note tht this is the kind of system tht rises in the discretistion of prtil differentil eqution Although for mtrices of this size we cn esily solve the system directly, we will illustrte the use of the Guss-Seidel method The Guss-Seidel itertion hs the form x k+ = ) x k + If we choose the strting point x = (,,), then x = x + = The system is esily solved to find x = (/,3/4,7/8) Continuing this process we get x, x 3, until we re stisfied with the ccurcy of the solution Alterntively, we cn lso solve the system using the coordintewise interprettion 4 Vector nd Mtrix Norms We hve seen in Exmple 4 tht the sequence of vectors x k generted by the Jcobi method pproches the solution of the system of equtions A s we keep going In order to mke this type of convergence precise, we need to be ble to mesure distnces between vectors nd mtrices DEFINITION 4 A vector norm on R n is rel-vlued function tht stisfies the following conditions: () For ll x R n : x nd x = if nd only if x = () For ll α R: αx = α x (3) For x, y R n : x + y x + y (Tringle Inequlity) EXAMPLE 44 The typicl exmples re the following: () The -norm n x = x i = ( x x ) / i= This is just the usul notion of Eucliden length () The -norm n x = x i (3) The -norm x = mx x i i n A convenient wy to visulise these norms is vi their unit circles If we look t the sets for p =,,, then we get the following shpes: i= {x R : x p = }

31 4 VECTOR AND MATRIX NORMS 7 Now tht we hve defined wy of mesuring distnces between vectors, we cn tlk bout convergence DEFINITION 4 A sequence of vectors x k R n, k =,,,, converges to x R n with respect to norm, if for ll ε > there exists n N > such tht for ll k N : x k x < ε In words: we cn get rbitrry close to x by choosing k sufficiently lrge We sometimes write lim k xk = x or x k x to indicte tht sequence x, x, converges to vector x If we wnt to indicte the norm with respect to which convergence is mesured, we sometimes write x k x, x k x, x k x to indicte convergence with respect to the -, -, nd -norms, respectively The following lemm implies tht for the purpose of convergence, it doesn t mtter whether we tke the - or the -norm LEMMA 4 For x R n, x x n x PROOF Let M := x = mx i n x i Note tht ( n x ) i x = M M n = x n, i= M becuse x i /M for ll i This shows the second inequlity For the first one, note tht there is n i such tht M = x i It follows tht ( n x ) i x = M M = x This completes the proof i= M A similr reltionship cn be shown between the -norm nd the -norm, nd lso between the -norm nd the -norm COROLLARY 4 Convergence in the -norm is equivlent to convergence in the -norm: x k x x k x In words: if x k x with respect to the -norm, then x k x with respect to the -norm, nd vice vers PROOF Assume tht x k x with respect to the -norm nd let ε > Since x k converges with respect to the -norm, there exists N > such tht for ll k > N, x k x < ε Since x k x x k x, we lso get convergence with respect to the -norm Now ssume conversely tht x k converges with respect to the -norm Then given ε >, for ε = ε/ n there exists N > such tht x k x < ε for k > N But since x k x n x k x < nε = ε, it follows tht x k lso converges with respect to the -norm

32 8 4 NUMERICAL LINEAR ALGEBRA The benefit of this type of result is tht some norms re esier to compute thn others Even if we re interested in mesuring convergence with respect to the -norm, it my be esier to show tht sequence converges with respect to the -norm, nd once this is shown, convergence in the -norm follows utomticlly by the bove corollry EXAMPLE 45 Let s look t the vector The different norms re x = 3, x = 3, x = x = The following lemm llows to compre different norm LEMMA 4 For x R n, x x n x PROOF Let M := x = mx i n x i Note tht ( n x ) i x = M M n = x n, i= M becuse x i /M for ll i This shows the second inequlity For the first one, note tht there is n i such tht M = x i It follows tht ( n x ) i x = M M = x This completes the proof i= M A similr reltionship cn be shown between the -norm nd the -norm, nd lso between the -norm nd the -norm COROLLARY 4 Convergence in the -norm is equivlent to convergence in the -norm: x k x x k x In words: if x k x with respect to the -norm, then x k x with respect to the -norm, nd vice vers PROOF Assume tht x k x with respect to the -norm nd let ε > Since x k converges with respect to the -norm, there exists N > such tht for ll k > N, x k x < ε Since x k x x k x, we lso get convergence with respect to the -norm Now ssume conversely tht x k converges with respect to the -norm Then given ε >, for ε = ε/ n there exists N > such tht x k x < ε for k > N But since x k x n x k x < nε = ε, it follows tht x k lso converges with respect to the -norm The benefit of this type of result is tht some norms re esier to compute thn others Even if we re interested in mesuring convergence with respect to the -norm, it my be esier to show tht sequence converges with respect to the -norm, nd once this is shown, convergence in the -norm follows utomticlly by the bove corollry EXAMPLE 46 Let s look t the vector The different norms re x = 3, x = 3, x = x = To study the convergence of itertive methods we lso need norms for mtrices Recll tht the Jcobi nd Guss-Seidel methods generte sequence x k of vectors by the rule x k+ = T x k + c for some mtrix T The hope is tht this sequence will converge to vector x such tht x = T x + c Given such n x, we cn subtrct x from both sides of the itertion to obtin x k+ x = T x k + c x = T (x k x)

33 4 VECTOR AND MATRIX NORMS 9 Tht is, the difference x k+ x rises from the previous difference x k x by multipliction with T For convergence we wnt x k x to become smller s k increses, or in other words, we wnt multipliction with T to reduce the norm of vector In order to quntify the effect of liner trnsformtion T on the norm of vector, we introduce the concept of mtrix norm DEFINITION 43 A mtrix norm is non-negtive function on the set of rel n n mtrices such tht, for every n n mtrix A, () A nd A = if nd only if A = () For ll α R, αa = α A (3) For ll n n mtrices A,B: A + B A + B (4) For ll n n mtrices A,B: AB A B Note tht prts -3 just stte tht mtrix norm lso is vector norm, if we think of the mtrix s vector Prt 4 of the definition hs to do with the mtrix-ness of mtrix The most useful clss of mtrix norms re the opertor norms induced by vector norm s DEFINITION 44 Given vector norm, the corresponding opertor norm of n n n mtrix A is defined Ax A = mx x x = mx Ax x R n x = REMARK 4 To see the second inequlity, note tht for x we cn write Ax x = A x x = Ay, with y = x/ x, where we used Property () of the definition of vector norm The vector y = x/ x is vector with norm y = x/ x =, so tht for every x there exists vector y with y = such tht Ax x = Ay In prticulr, minimizing the left-hnd side over x give the sme result s minimizing the right-hnd side over y with y = First, we hve to verify tht the opertor norm is indeed mtrix norm THEOREM 4 The opertor norm corresponding to vector norm is mtrix norm PROOF Properties -3 re esy to verify from the corresponding properties of the vector norms For exmple, A becuse by the definition, there is no wy it could be negtive To show the property 4, nmely, AB A B for n n mtrices A nd B, we first note tht for ny y R n, Ay Ax mx y x x = A, nd therefore Now let y = B x for some x with x = Then Ay A y AB x A B x A B As this inequlity holds for ll unit-norm x, it lso holds for the vector tht mximises AB x, nd therefore we get AB = mx AB x A B x = This completes the proof The interprettion is tht the opertor norm mesures how much x is being stretched by mtrix A Even though the opertor norms with respect to the vrious vector norms re of immense importnce in the nlysis of numericl methods, they re hrd to compute or even estimte from their definition lone It is therefore useful to hve lterntive chrcteristions of these norms The first of these chrcteristions is concerned with the norms nd, nd provides n esy criterion to compute these

34 3 4 NUMERICAL LINEAR ALGEBRA by LEMMA 43 For n n n mtrix A, the opertor norms with respect to the -norm nd the -norm re given A = mx j n i= A = mx i n j = n i j (mximum bsolute column sum), n i j (mximum bsolute row sum) PROOF We will proof this for the -norm We first show the inequlity A mx n i n j = i j Let x be vector such tht x = Tht mens tht ll the entries hve bsolute vlue x i It follows tht n Ax = mx i j x j i n mx j = i n j = mx i n j = n i j x j n i j, where the inequlity follows from writing out the mtrix vector product, interpreting the -norm, using the tringle inequlity for the bsolute vlue, nd the fct tht x j for j n Since this holds for rbitrry x with x =, it lso holds for the vector tht mximises mx x = Ax = A, which concludes the proof of one directions In order to show A mx n i n j = i j, let i be the index t which the mximum of the sum is ttined: n n mx i j = i j i n j = Choose y to be the vector with entries y j = if i j > nd y j = if i j < This vector stisfies y = nd, moreover, n n y j i j = i j, by the choice of the y j We therefore hve This finishes the proof EXAMPLE 47 Consider the mtrix j = j = j = A = mx Ax n Ay = i j = mx x = j = 7 3 A = The opertor norms with respect to the - nd -norm re i n j = n i j A = mx{3,3,6} = 3, A = mx{,,} = How do we chrcterise the mtrix -norm A of mtrix? The nswer is in terms of the eigenvlues of A Recll tht (possibly complex) number λ is n eigenvlue of A, with ssocited eigenvector u, if Au = λu DEFINITION 45 The spectrl rdius of A is defined s THEOREM 4 For n n n mtrix A we hve ρ(a) = mx{ λ : λ eigenvlue of A} A = ρ(a A)

35 4 VECTOR AND MATRIX NORMS 3 PROOF Note tht for vector x, Ax = x A Ax We cn therefore express the squred -norm of A s A = mx x = Ax = mx x = x A Ax As continuous function over compct set, f (x) = x A Ax ttins unique mximum u on the unit sphere {x : x = } By the Lgrnge multiplier theorem (see the ppendix), there exist prmeter λ such tht (46) f (u) = λu To compute the grdient f (x), set B = A A, so tht n n f (x) = x B x = b i j x i x j = i i x i=b i + b i j x i x j, i<j i,j = where the lst inequlity follows from the symmetry of B (tht is, b i j = b j i ) Then f = b kk x k + n b ki x i = b ki x i x k But this expression is just the two times the k-th row of B x, so tht Using this in Eqution (5), we find i k f (x) = B x = A Ax A Au = λu, so tht λ is n eigenvlue of A A Using tht u u = u =, we lso hve u A Au = λu u = λ, nd since u ws mximiser of the left-hnd function, it follows tht λ is the mximl eigenvlue of A A, λ = ρ(a A) Summrising, we hve A = ρ(a A), wht ws to be shown EXAMPLE 48 Let A = The eigenvlues re the roots of the chrcteristic polynomil λ p(λ) = det(a λ) = det λ λ Evluting this determinnt, we get the eqution ( λ)(λ λ + 4) = The solutions re given by λ = nd λ,3 = ± 3i The spectrl rdius of A is therefore Then ρ(a) = mx{,,} = For symmetric mtrices, tht is, mtrices such tht A = A, the -norm is just the spectrl rdius LEMMA 44 If A is symmetric, then A = ρ(a) PROOF Let λ be n eigenvlue of A with corresponding eigenvector u, so tht Au = λu i= A Au = A λu = λa u = λau = λ u It follows tht λ is eigenvlue of A A with corresponding eigenvector u In prticulr, Tking squre roots on both sides, the clim follows A = ρ(a A) = mx{λ : λ eigenvlue of A} = ρ(a)

36 3 4 NUMERICAL LINEAR ALGEBRA EXAMPLE 49 We compute the eigenvlues, nd thus the spectrl rdius nd the -norm, of the finite difference mtrix A = Let h = /(n + ) We first clim tht the vectors u k, k n, defined by sin(kπh) u k = sin(nkπh) re the eigenvectors of A, with corresponding eigenvlues This cn be verified by checking tht λ k = ( cos(kπh)) Au k = λ k u k In fct, for k n, the left-hnd side of the j -th entry of the bove product is given by sin(j kπh) sin((j )kπh) sin((j + )kπh) Using the trigonometric identity sin(x + y) = sin(x)cos(y) + cos(x)sin(y), we cn write this s sin(j kπh) (cos(kπh)sin(j kπh) cos(j kπh)sin(kπh)) (cos(kπh)sin(j kπh) + cos(j kπh)sin(kπh)) = ( cos(kπh)) sin(j kπh) Now sin(j kπh) is just the j -th entry of u k s defined bove, so the coefficient in front must equl the corresponding eigenvlue The rgument for k = nd k = n is similr The spectrl rdius is the mximum modulus of such n eigenvlue, ρ(a) = mx k n λ k = ( cos ( nπ n + As the mtrix A is symmetric, this is lso equl to the mtrix -norm A: ( ( nπ )) A = cos n + )) 43 Convergence of Itertive Algorithms In this section we focus on lgorithms tht ttempt to solve system of equtions (47) Ax = b by strting with some vector x nd then successively computing sequence x k, k, by mens of rule (48) x k+ = T x k + c for some mtrix T nd vector c The hope is tht the resulting sequence will converge to solution x of Ax = b EXAMPLE 4 The Jcobi nd Guss-Seidel methods fll into this frmework Recll the decomposition A = L + D +U, where L is the lower tringulr, D the digonl nd U the upper tringulr prt Then the Jcobi method corresponds to the choice T = T J = D (L +U ), c = D b, while the Guss-Seidel method corresponds to T = T GS = (L + D) U, c = (L + D) b,

37 43 CONVERGENCE OF ITERATIVE ALGORITHMS 33 LEMMA 45 Let T nd c be the mtrices in the itertion scheme (48) corresponding to either the Jcobi method or the Guss-Seidel method, nd ssume tht D nd L + D re invertible Then x is solution of the system of equtions (47) if nd only if x is fixed point of the itertion (48), tht is, x = T x + c PROOF We write down the proof for the cse of Jcobi s method, the Guss-Seidel cse being similr We hve This shows the clim Ax = b (L + D +U )x = b Dx = (L +U )x + b x = D (L +U )x + D b x = T x + c The problem of solving Ax = b is thus reduced to the problem of finding fixed point to n itertion scheme The following importnt result shows how to bound the distnce of n iterte x k from the solution x in terms of the opertor norm of T nd n initil distnce of x THEOREM 43 Let x be solution of Ax = b, nd x k, k, be sequence of vectors such tht x k+ = T x k + c for n n n mtrix T nd vector c R n Then, for ny vector norm nd ssocited mtrix norm, we hve x k+ x T k+ x x for ll k PROOF We proof this by induction on k Recll tht for every vector x, we hve T x T x Subtrcting the identity x = T x + c from x k+ = T x k + c nd tking norms we get (49) x k+ x = T (x k x) T x k x Setting k = gives the clim of the theorem for this cse If we ssume tht the clim holds for k, k, then x k x T k x k x by this ssumption, nd plugging this into (49) finishes the proof COROLLARY 43 Assume tht in ddition to the ssumptions of Theorem 43, we hve T < Then the sequence x k, k, converges to fixed point x with x = T x + c with respect to the chosen norm PROOF Assume x x (otherwise there is nothing to prove) nd let ε > Since T <, T k s k In prticulr, there exists n integer N > such tht for ll k > N, T k ε < x x It follows tht for k > N we hve x k x < ε, which completes the convergence proof Recll tht for the Guss-Seidel nd Jcobi methods, fixed point of T x + c ws the sme s solution of Ax = b It follows tht the Guss-Seidel nd Jcobi methods converge to solution (with respect to some norm) provided tht T < Note lso tht either one of T < or T < will imply convergence with respect to both the -norm nd the -norm The reson is the equivlence of norms x x n x, which implies tht if the sequence x k, k, converges to x with respect to one of these norms, it lso converges with respect to the other one Such n equivlence cn lso be shown between the - nd the -norm So fr we hve seen tht the condition T < ensures tht n itertive scheme of the form (48) converges to vector x such tht x = T x + c s k The converse is not true, there re exmples for which T but the itertion (48) converges nevertheless

38 34 4 NUMERICAL LINEAR ALGEBRA EXAMPLE 4 Recll the finite difference mtrix A = nd pply the Jcobi method to compute solution of Ax = b The Jcobi method computes the sequence x k+ = T x k + c, where c = b nd T = T J = D (L +U ) = We hve T =, so the convergence criterion doesn t pply for this norm However, one cn show tht ll the eigenvlues stisfy λ < Since the mtrix T is symmetric, we hve T = ρ(t ) <, where ρ(t ) denotes the spectrl rdius It follows tht the itertion (48) does converge with respect to the -norm, nd therefore lso with respect to the -norm, despite hving T = It turns out tht the spectrl rdius gives rise to necessry nd sufficient condition for convergence THEOREM 44 The itertes x k of (48) converge to solution x of x = T x + c for ll strting points x if nd only if ρ(t ) < PROOF Let x be ny strting point, nd define, for ll k, z k = x k x Then z k+ = T z k, s is esily verified The convergence of the sequence x k to x is then equivlent to the convergence of z k to Assume T hs n eigenvlues λ k (possibly ), k n, nd ssume for the moment tht the eigenvectors u k form bsis of R n (equivlently, tht T is digonlisble) We cn write (4) z = for some coefficients α j For the iterte we get z k+ = T z k n α j u j j = = T k+ z ( ) n = T k+ α j u j j = = = n α j T k+ u j j = n j = α j λ k+ j u j Now ssume ρ(t ) < Then λ j < for ll eigenvlues λ j, nd therefore λ k+ s k Therefore, j z k+ s k nd x k+ x If, on the other hnd, ρ(t ), then there exists n index j such tht λ j If we choose strting point x such tht the coefficient α j in (4) is not zero, then α j λ k+ α j j for ll k nd we deduce tht z k+ does not converge to zero

39 43 CONVERGENCE OF ITERATIVE ALGORITHMS 35 If T is not digonlisble, then we still hve the Jordn norml form J = P T P, where P is n invertible mtrix nd J consists of Jordn blocks λ i λ i λ i on the digonl for ech eigenvlue λ i Rther thn considering bsis of eigenvectors, we tke one consisting of generlized eigenvectors, tht is, solutions to the eqution (A λ i ) k =, where k <= m nd m is the geometric multiplicity of λ i So fr we hve seen tht n itertive method x k+ = T x k + c converges to fixed point x = T x + c if nd only if the spectrl rdius ρ(t ) < Since the eigenvlues re in generl not esy to compute, the question is whether there is convenient wy to estimte ρ(t ) One wy to bound the size of the eigenvlues is by mens of Gershgorin s Theorem Recll tht eigenvlues of mtrix A cn be complex numbers THEOREM 45 Every eigenvlue of n n n mtrix A lives in one of the circles C,,C n, where C i hs centre t the digonl i i nd rdius r i = i j j i EXAMPLE 4 Consider the mtrix A = 4 8 The centres re given by,4,8, nd the rdii by r =, r =, r 3 = FIGURE Gershgorin s circles PROOF Let λ be n eigenvlue of A, with ssocited eigenvector u, so tht The i -th row of this eqution looks like Bringing i i to the left, this implies the inequlity λu i = Au = λu n i j u j j = λ i i i j u j j i u i If the index i is such tht u i is the component of u with lrgest bsolute vlue, then the right-hnd side is bounded by r i, nd we get λ i i r i,

40 36 4 NUMERICAL LINEAR ALGEBRA which implies tht λ lies in circle of rdius r i round i i Gershgorin s Theorem hs implictions on the convergence of Jcobi s method To stte these implictions, we need definition DEFINITION 46 A mtrix A is clled digonlly dominnt, if for ll indices i we hve i i > r i COROLLARY 44 Let A be digonlly dominnt Then the Jcobi method converges to solution of the system Ax = b for ny strting point x PROOF We need to show tht if A is digonlly dominnt, then ρ(t J ) <, where T J = D (L +U ) is the itertion mtrix of Jcobi s method The i -th row of T J is i i ( i i i i i+ i n ) By Gershgorin s Theorem, ll the eigenvlues of T J lie in circle round of rdius r i = i j i i It follows tht if A is digonlly dominnt, then r i <, nd therefore λ < for ll eigenvlues λ of T J In prticulr, ρ(t J ) < nd Jcobi s method converges for ny x j i 44 The Condition Number In this section we discuss the sensitivity of system of equtions Ax = b to perturbtions in the dt This sensitivity is quntified by the notion of condition number We begin by illustrting the problem with smll exmple EXAMPLE 43 Let s look t the system of equtions with ( ) ( ) ε + δ A =, b =, where < ε, δ << (tht is, much smller thn ) The solution of Ax = b is ( δ ) x = ε We cn think of δ s cused by rounding error Thus δ = would give us n exct solution, while if δ is smll nd ε << δ, then the chnge of x due to δ cn be lrge! The following definition is delibertely vgue, nd will be mde more precise in light of the condition number DEFINITION 47 A system of equtions Ax = b is clled ill-conditioned, if smll chnges in the system cuse big chnges in the solution To mesure the sensitivity of solution with respect to perturbtions in the system, we introduce the condition number of mtrix s DEFINITION 48 Let be mtrix norm nd A n invertible mtrix The condition number of A is defined cond(a) = A A We write cond (A), cond (A), cond (A) for the condition number with respect to the, nd norms Let x be the true solution of system of equtions Ax = b, nd let x c = x + x be the solution of perturbed system (4) A(x + x) = b + b, where b is perturbtion of b We re interested in bounding the reltive error x x

41 44 THE CONDITION NUMBER 37 in terms of b / b We hve b = A(x + x) b = A x, from which we get x = A b nd x = A b A b On the other hnd, b = Ax A x, nd combining these estimtes, we get we get (4) x x A A b b = cond(a) b b The condition number therefore bounds the reltive error in the solution in terms of the reltive error in b We cn lso derive similr bound for perturbtions A in the mtrix A Note tht smll condition number is good thing, s it implies smll error The bove nlysis cn lso be rephrsed in terms of the residul of computed solution Suppose we hve A nd b exctly, but solving the system Ax = b by computtionl method gives computed solution x c = x + x tht hs n error We don t know the error, but we hve ccess to the residul r = Ax c b We cn rewrite this equtions s in ( 5), with r insted of b, so tht we cn interpret the residul s perturbtion of b The condition number bound (4) therefore implies x x cond(a) r b We now turn to some exmples of condition numbers EXAMPLE 44 Let A = ( ) ε The inverse is given by A = ( ) ε ε The condition numbers with respect to the, nd norms re esily seen to be ( + ε) cond (A) =, cond (A) = ε ε, cond ( + ε) (A) = ε If ε is smll, the condition numbers re lrge nd therefore cn t gurntee smll errors EXAMPLE 45 A well-known exmple is the Hilbert mtrix Let H n by the n n mtrix with entries h i j = i + j for i, j n This mtrix is symmetric nd positive definite (tht is, Hn = H n nd x H n x > for ll x ) For exmple, for n = 3 the mtrix looks s follows 3 H 3 = 3 Exmples such s the Hilbert mtrix re not common in pplictions, but they serve s reminder tht one should keep n eye on the conditioning of mtrix n 5 5 cond (H n ) It cn be shown tht the condition number of the Hilbert mtrix is symptoticlly cond(h n ) ( + ) 4n+4 5/4 πn for n To see the effect tht this conditioning hs to solving systems of equtions, let s look t system H n x = b,

42 38 4 NUMERICAL LINEAR ALGEBRA 5 Conditioning of Hilbert mtrix log (cond (H)) n FIGURE Condition number of Hilbert s mtrix with entries b i = n j j = i+j The system is constructed such tht the solution is x = (,,) For n = we get, solving the system using MATLAB, solution x + x with differs considerbly from x The reltive error x x Wht tht mens is tht the computed solution is useless EXAMPLE 46 An importnt exmple is the condition number of the omnipresent finite difference mtrix A = It cn be shown tht the condition number of this mtrix is given by cond (A) = 4 π h, where h = /(n + ) If follows tht the condition number increses with the number of discretistion steps n EXAMPLE 47 Wht is the condition number of rndom mtrix? If we generte rndom mtrices with normlly distributed entries nd look t the frequency of the logrithm of the condition number, then we get the following: It should be noted tht rndom mtrix is not the sme s ny old mtrix, nd eqully not the sme s typicl mtrix rising in pplictions, so one should be creful in interpreting sttements bout rndom mtrices! Computing the condition number cn be difficult, s it involves computing the inverse of mtrix In mny cses one cn find good bounds on the condition number, which cn, for exmple, be used to tell whether problem is ill-conditioned

43 44 THE CONDITION NUMBER 39 4 Distribution of log of condition number Frequency log(cond (A)) FIGURE 3 Condition number of rndom mtrix EXAMPLE 48 Consider the mtrix ( ) ( ) A =, A = 4 The condition number with respect to the -norm is given by cond (A) = 4 4 We would like to find n estimte for this condition number without hving to invert the mtrix A To do this, note tht for ny x nd b = Ax we hve Ax = b x = A b x A b, nd we hve the lower bound A x b Choosing x = (,) in our cse, we get b = (,) nd the estimte cond (A) = A A x b A 4 This estimte is of the right order of mgnitude (in prticulr, it shows tht the condition number is lrge), nd no inversion ws necessry To summrise: A smll condition number is good thing, s smll chnges in the dt led to smll chnges in the solution Condition numbers my depend on the problem the mtrix rises from nd cn be very lrge A lrge condition number is sign tht the mtrix t hnd is close to being non-invertible Condition numbers lso ply role in the convergence nlysis of itertive mtrix lgorithms We will not discuss this spect here nd refer to more dvnced lectures on numericl liner lgebr nd mtrix nlysis

44

45 CHAPTER 5 Nonliner Equtions Given function f : R R, we would like to find solution to the eqution (5) f (x) = f(x) x For exmple, if f is polynomil of degree, we cn write down the solutions in closed form (though, s seen in Lecture, this by no mens solves the problem from numericl point of view!) In generl, we will encounter functions for which closed form does not exist, or is not convenient to write down or evlute The best wy to del with (5) is then to find n pproximte solution using n itertive method Here we will discuss two methods: The bisection method Newton s method The bisection method only requires tht f be continuous, while Newton s method lso requires differentibility but is fster 5 The bisection method Let f : R R be continuous function on n intervl [,b], < b Assume tht f ()f (b) <, tht is, the function vlues t the end points hve different signs By the intermedite vlue theorem (or common sense) there exists n x with < x < b such tht f (x) = The most direct method of finding such zero x is by divide nd conquer: determine the hlf of the intervl [,b] tht contins x nd shrink the intervl to tht hlf, then repet until the boundry points re resonbly close to x This pproch is clled the bisection method 4

46 4 5 NONLINEAR EQUATIONS 5 5 Itertion:, x = 375, f(x) = Itertion:, x = 5, f(x) = Itertion:, x = 5, f(x) = Itertion: 3, x = 375, f(x) = 98 5 FIGURE The bisection method To be more precise, strting with [,b] such tht f ()f (b) <, we construct series of decresing intervls [ n,b n ], n, ech contining x At ech step, we clculte the midpoint p n = ( n + b n )/ nd evlute f (p n ) If f (p n )f ( n ) < we set [ n+,b n+ ] = [ n, p n ], else [ n+,b n+ ] = [p n,b n ] We stop whenever b n n < TOL for some predefined tolernce TOL, for exmple, 4, nd return the vlue p n In MATLAB this would look like: while (b >= TOL) p = (+b)/; if f()*f(p)< b = p; else = p; end end x = p; % Clculte midpoint % Chnge of sign detected % Set right boundry to p % Set left boundry to p % Computed solution EXAMPLE 5 Let s look t the polynomil x 6 x on the intervl [,] with tolernce TOL = (tht is, we stop when we hve locted n intervl of length contining the root x) Note tht no closed form solution exists for polynomils of degree 5 The bisection method is best crried out in form of tble At ech step the midpoint p n is obtined, nd serves s next left or right boundry, depending on whether f ( n )f (p n ) < or not n n f ( n ) b n f (b n ) p n f (p n ) We see tht b 4 4 = 5 < TOL, so we stop there nd declre the solution p 4 = 875

47 5 NEWTON S METHOD 43 The following result shows tht the bisection method indeed pproximtes zero of f to rbitrry precision LEMMA 5 Let f : R R be continuous function on n intervl [,b] nd let p n, n, be the sequence of midpoints generted by the bisection method on f Let x be such tht f (x) = Then In prticulr, p n x s n A convergence of this form is clled liner p n x b n PROOF Let x [,b] be such tht f (x) = Since p n is the midpoint of [ n,b n ] nd x [ n,b n ], we hve p n x b n n By bisection, ech intervl hs hlf the length of the preceding one: b n n = b n n Therefore, p n x b n n = b n n This completes the proof = = n b = b n 5 Newton s method If the function f : R R is differentible, nd we know how to compute f (x), then we cn (under certin conditions) find the root of f much quicker by Newton s method The ide behind Newton s method is to pproximte f t point x n by its tngent line, nd clculte the next iterte x n+ s the root of this tngent line Given point x n with function vlue f (x n ), we need to find the zero-crossing of the tngent line t (x n, f (x n )): y = f (x n )(x x n ) + f (x n ) = Solving this for x, we get x = x n f (x n) f (x n ), which is defined provided f (x n ) Formlly, Newton s method is s follows: Strt with x [,b] such tht f (x ) At ech step, compute new iterte x n+ from x n s follows: x n+ = x n f (x n) f (x n ) Stop if x n+ x n < TOL for some predefined tolernce

48 44 5 NONLINEAR EQUATIONS Itertion:, x = 7, f(x) = 8997 Itertion:, x = 437, f(x) = Itertion:, x = 489, f(x) = 45 5 Itertion: 3, x = 3344, f(x) = FIGURE Newton s method EXAMPLE 5 Consider gin the function f (x) = x 6 x The derivtive is f (x) = 6x 5 We pply Newton s method using tolernce TOL = We get the sequence: x = x = x f (x ) f (x ) = x 3 = x f (x ) f (x ) = 436 x 4 = 349 x 5 = 347 The difference x 5 x 4 is below the given tolernce, so we stop nd declre x 5 to be our solution We cn lredy see tht in just four itertions we get better pproximtion tht using the bisection method We will see tht the error of Newton s method is bounded s x n+ x k x n x for constnt k, provided we strt sufficiently close to x This will be shown using the theory of fixed point itertions, discussed in the next lecture Newton s method is not without difficulties One cn esily come up with strting points where the method does not converge One exmple is when f (x ), in which cse the tngent line t (x, f (x )) is lmost horizontl nd tkes us fr wy from the solution Another one would be where the itertion oscilltes between two vlues, s in the following exmple In the following lectures we will derive precise conditions under which Newton s method converges to solution

49 53 FIXED-POINT ITERATIONS 45 6 y=x 3 x f(x) x FIGURE 3 Newton s method fils 53 Fixed-point itertions A root of function f : R R is number x R such tht f (x) = A fixed-point is root of function of the form f (x) = g (x) x DEFINITION 5 A fixed-point of function g : R R is number x such tht g (x) = x In Newton s method we hve g (x) = x f (x) f (x), where x is fixed-point of g if nd only if x is root of f A feture of the fixed-point formultion is tht we my generte sequence x n, n, by mens of x n+ = g (x n ) nd hope tht it converges to fixed-point of g We will study conditions under which this hppens EXAMPLE 53 Let f (x) = x 3 + 4x There re severl wys to rephrse the problem f (x) = s fixedpoint problem g (x) = x () Let g (x) = x x 3 4x + Then g (x) = x if nd only if f (x) =, s is esily seen () Let g (x) = ( x 3 ) / Then g (x) = x x = 4 ( x3 ) f (x) = (3) Let g 3 (x) = ( ) / 4+x Then it is lso not difficult to verify tht g3 (x) = x is equivlent to f (x) = EXAMPLE 54 We briefly discuss more intriguing exmple, the logistic mp g (x) = r x( x) with prmeter r [,4] Whether the itertion x n+ = g (x n ) converges to fixed-point, nd how it converges, depends on the vlue of r Three exmples re shown below If we record the movement of x n for r rnging between nd 4, the following picture emerges: It turns out tht for smll vlues of r we hve convergence (which, incidentlly, does not depend on the strting vlue), for vlues slightly bove 3 oscilltion between two, then four vlues, while for lrger r we hve chotic behviour In tht region, the trjectory of x n is lso highly sensitive to perturbtions of the initil vlue x The precise behviour of such itertions is studied in dynmicl systems

50 46 5 NONLINEAR EQUATIONS 7 r = 8 r = 35 r = Logistic mp x = r x( x) x r Given fixed-point problem, the ll importnt question is when the itertion x n+ = g (x n ) converges The following theorem gives n nswer to this question THEOREM 5 (fixed-point theorem) Let g be smooth function on [,b] Assume () g (x) [,b] for x [,b], nd () g (x) < for x [,b] Then there exists unique fixed-point x = g (x) in [,b] nd the sequence {x n } defined by x n+ = g (x n ) converges to x Moreover, x n+ x λ n x x for some λ < PROOF Let f (x) = g (x) x Then by (), f () = g (), nd f (b) = g (b) b By the intermedite vlue theorem, there exists n x [,b] such tht f (x) = Hence, there exists x [,b] such tht g (x) = x, showing the existence of fixed-point Next, consider x n+ = g (x n ) for n, nd let x = g (x) be fixed-point Then x n+ x = g (x n ) g (x) Assume without lck of generlity x n > x By the men vlue theorem there exists ξ (x, x n ) such tht g (ξ) = g (x n) g (x), x n x

51 53 FIXED-POINT ITERATIONS 47 nd hence x n+ x = g (ξ)(x n x) Since ξ [,b], ssumption () gives g (ξ) λ for some λ < Hence, x n+ x λ x n x λ n x x This proves the convergence To show uniqueness, ssume x, y re two distinct fixed-points of g with x < y By the men vlue theorem nd ssumption (), there exists ξ (x, y) such tht But since both x nd y re fixed-points, we hve so we hve contrdiction nd x = y Then g (x) g (y) = g (ξ) < x y g (x) g (y) =, x y EXAMPLE 55 Let s look t the functions from the exmple (53) to see for which one we hve convergence () g (x) = x x 3 4x + on [,b] = [,] Note tht g () = 6 [,], therefore ssumption () is violted () g (x) = ( x3 ) / The derivtive is given by g (x) = 3x 4( x 3 ) /, nd therefore g () = Condition () fils (3) The third formultion is ( ) / g 3 (x) = 4 + x The derivtive is given by g 3 (x) = 5 (4 + x) 3/, nd therefore the function is strictly decresing on [,] Since g 3 () = 5/3 nd g 3 () = re both in [,], condition () is stisfied Furthermore, g 3 (x) / < for x [,], so condition () is lso stisfied It follows tht the itertion x n+ = g 3 (x n ) converges to fixed-point of g 3 We cn try this out: x = 5, x = 3484, x 3 = 3674, x 4 = 3665, We cn pply the fixed-point theorem to Newton s method Let g (x) = x f (x) f (x) g (x) = f (x) f (x) + f (x)f (x) f (x) = f (x)f (x) f (x) Let α be root of f Then f (α) = nd f (α), so tht g (α) = Hence, g (α) < t the fixed-point Now let ε > be smll nd = α ε, b = α + ε Then, by continuity, for x [, b], nd () holds Furthermore, g (x) < g (x) α = g (ξ) x α x α < ε for x [,b] Hence, g (x) [,b] nd () holds It follows tht in smll enough neighbourhood of root of f (x), Newton s method converges to tht root (provided f (x) = t tht root)

52 48 5 NONLINEAR EQUATIONS Note tht the rgument with ε illustrtes key spect of Newton s method: it only converges when the initil guess x is close enough to root of f Wht close enough mens is often not so cler Recll tht the fixed-point theorem gurntees convergence of n itertion x n+ = g (x n ) to unique fixedpoint of g in n intervl [,b] if g (x) [,b] for x [,b], nd g (x) < on [,b] We cn pply this to show tht Newton s method converges if we strt with n x tht is sufficiently close to root of f (x) Newton s method for finding root of continuously differentible function f (x) is fixed-point itertion with The derivtive is g (x) = x f (x) f (x) g (x) = f (x) f (x) + f (x)f (x) f (x) = f (x)f (x) f (x) If α is root of f, tht is, f (α) =, nd if f (α), then g (α) = Since g (x) is continuous, for every δ > there exists n ε > such tht g (x) < δ for x α < ε In prticulr, g (x) < / in smll intervl [,b] with = α ε, b = α + ε By the men vlue theorem there exists ξ (,b) such tht g (x) α = g (ξ) x α x α < ε for x [,b] Hence, g (x) [,b] nd both conditions of the fixed-point theorem re stisfied Newton s method, interpreted s fixed-point itertion, converges In the next section we derive stronger result for Newton s method, nmely tht it converges qudrticlly if the strting point is close enough 54 Rtes of convergence The speed of itertive numericl methods is chrcterised by the rte of convergence DEFINITION 5 The sequence x n, n, converges to α with order one, or linerly, if x n+ α k x n α for some < k < The sequence converges with order r, r, if x n+ α k x n α r with k > If the sequence converges with order r =, it is sid to converge qudrticlly EXAMPLE 56 Consider the sequence x n = / r n for r > Then x n s n Note tht x n+ = = ( ) r r n+ r n r = r = x r n n, nd therefore x n+ x n r We hve convergence of order r For exmple, if sequence converges qudrticlly, then if x n α, in the next step we hve x n+ α k We would like to show tht Newton s method converges qudrticlly to root of function f if we strt the itertion sufficiently close to tht root THEOREM 5 Let g be continuously differentible in the neighbourhood of fixed point α The fixed-point itertion x n+ = g (x n ) converges qudrticlly to α, if g (α) = nd the strting point x is sufficiently close to α Agin, sufficiently close mens tht there exists n intervl [,b] for which this holds PROOF Consider the Tylor expnsion round α: g (x) = g (α) + g (α)(x α) + g (α)(x α) + R, where R is reminder term of order O((x α) 3 ) Assume g (α) = Then g (x) g (α) = g (α)(x α) + R Sufficiently close mens tht there exists n intervl round the root such tht the method converges for ny strting point in tht intervl

53 55 NEWTON S METHOD IN TWO DIMENSIONS 49 Assume tht x α < ε Since R is proportionl to (x α) 3, we cn lso write g (x) g (α) = g (α)(x α) ( + g (α) R ), where R = R/(x α) = O((x α)) Cε for constnt C Tking bsolute vlues, we get g (x) g (α) k x α for constnt k Set x = x n, x n+ = g (x n ),α = g (α) Then g (x) g (α) = x n+ α nd This shows qudrtic convergence x n+ α k x n α COROLLARY 5 Newton s method converges qudrticlly if we strt sufficiently close to root In summry, we hve the following points worth noting bout the bisection method nd Newton s method The bisection method requires tht f is continuous on [,b], nd tht f ()f (b) < Newton s method requires tht f is continuous nd differentible, nd moreover requires good strting point x The bisection method converges linerly, while Newton s method converges qudrticlly There is no obvious generlistion of the bisection method to higher dimension, while Newton s method generlises esily Newton s method in the complex plne Before discussing Newton s method in higher dimensions, we present yet nother exmple illustrting the intricte behviour of fixed-point itertions, this time over the complex numbers EXAMPLE 57 Consider the function This function hs exctly three roots, the roots of unity for k =,, As in the rel cse, Newton s method f (z) = z 3 z k = e πi k 3 z n+ = z n f (z n) f (z n ) converges to one of these roots of unity if we strt close enough But wht hppens t the boundries? The following picture illustrtes the behviour of Newton s method for this function in the complex plne, where ech col or indictes to which root strting vlue converges: If we look t the speed of convergence, we get the following picture: 55 Newton s method in two dimensions We look t the problem of finding vector x R such tht ( ) ( ) f (x) f (x) = = = f (x) To derive Newton s method, we use the Tylor series in higher dimensions where f (y) = f (x) + J(x)(y x) + higher order terms, J(x) = ( f x f x f x f x is the Jcobin mtrix of prtil derivtives As in the one dimensionl cse, we use the liner pproximtion t x f (x) + J(x)(y x) )

54 5 5 NONLINEAR EQUATIONS nd look for the zero crossing of this expression If the Jcobin mtrix is invertible, then y = x J(x) f (x) Newton s method in two dimension cn then be stted s follows: () Strt with x, () At ech step n, solve J(x n )ε = f (x n ) for ε nd set x n+ = x n + ε