Notes o iteratio ad Newto s method Iteratio Iteratio meas doig somethig over ad over. I our cotet, a iteratio is a sequece of umbers, vectors, fuctios, etc. geerated by a iteratio rule of the type 1 f where f is some fied fuctio or operatio. We eed to provide a startig value for the sequece to get the process goig, say 0 or 1 (whichever is more coveiet to call this startig value) ad the the iteratio rule geerates the terms that follow i the sequece. We usually call 0 the "iitial coditio" of the iteratio. I may applicatios, the iteratio 1 f represets, or ca be iterpreted as, the evolutio of a "dyamical system", i which the ide plays the role of time. E1. 1 a, with a a fied umber ad a real umber. Thus we are usig the iteratio rule 1 f with the choice of f a. I this case it is easy to calculate, give 0, that a 0. This is a case where we ca eplicitly calculate i terms of. Ofte, that is ot possible. This eample is a eample of a "liear iteratio" because the fuctio f is liear. E2. 1 1. I this case oe caot obtai a eplicit formula for. Oe might ask, however, (ad oe ca aswer) importat qualitative questios such as, does?, ad oe ca eve obtai approimatios for the value of i terms of. This is a "oliear" iteratio without a eplict formula for, but evertheless it is oe that ca be aalyzed fairly well. E3: 1 A where A is a m mmatri ad 1,... are m 1 vectors. This very simple, but very importat liear iteratio, has the "eplicit" solutio A 0 but actually eplicitly givig a formula for A is a complicated problem requirig the theory of eigevalues ad eigevectors. E4.: 1 1. Here is a parameter (much like a is i the iteratio 1 a ). The "allowable" values of are 0 4 because for these values of, if we start with 0 0 1 the 0 1 for all. (this should t be too hard to figure out yourself). This is the famous "chaotic" iteratio. For some values of the iteratio behaves very icely ad predictably, ad eve has a limit, but for other values of, the seem to wader upredictably over the iterval 0, 1. E5: Here we costruct a sequece of fuctios g 0, g 1,.., usig the iteratio g 1 1 0 g t dt. If we start with g 0 0, we obtai g 1 1, g 2 1,... What happes? If you fi a value of, what is g approachig for large? This is a very simple-lookig iteratio but iteratios of this type, resultig i a sequece of fuctios, are very importat i mathematical theory. E6: Cellular automata. We discussed these before. I this case represets a ifiite strog of zeros ad 1 s ad the fuctio f is the rule by which we geerate the et state of the system, 1. 1
Fied poits ad equilibria i iteratios: Give a iteratio 1 f, if a value satisfies f the is called a "fied poit" of the iteratio 1 f, or simply a fied poit of f. I the case of the dyamical systems iterpretatio of 1 f, we ote that if, the 1 ad 2,... i.e. the values of stay fied. We refer the to as a "equilibrium poit", or simply, a equilibrium, of the system 1 f. We are particularly iterested i iteratios that coverge to a fied poit, i.e. i iteratios 1 f for which, where is a fied poit. (This may, of course, deped o the right choice of the startig value 0 ). Ideed, if a ad f is cotiuous, the it automatically follows that a f a must be true. We will use such iteratios to umerically compute solutios to certai equatios that we have put ito the form f. To determie where a fied poit eists, you ca simply graph y f ad y ad look for itersectio poits, or graph f ad look for zeroes. More precisely, you are lookig (either lookig literally o the graph, or figuratively by computig umbers) for values a, b of for which a f a ad b f b have opposite sig (i.e. oe is positive ad oe is egative). The, assumig that f is cotiuous (it almost always is), there must be a fied poit betwee a ad b. E: Suppose we wish to umerically solve 2 si (there is o "eact" aswer). Graphically, we observe a solutio below i the viciity of 2. y 3 2 1 0 0 1 2 3 Now we eecute the iteratio 1 2 si with 0 2, ad obtai 1.81859485365136 1.93890945306856 1.86601601636051 1.91347649344167 1.88371495915469 1.90287832191698 1.89073127521091 1.89851175825404 1.89356034515336 By ow it appears that the iterates (the sequece of values) is covergig, boucig back ad forth while the distace betwee them shriks. Note that we do t kow what the "eact" aswer is, but we ca prove (we ll see how later) that the iterates will coverge to 2
the eact aswer. Graphical view of iteratio: As i the graph above, we plot the fuctio y f ad the lie y. The we draw a lie from our curret poit, f o the graph y f to the poit, f 1, 1 that lies at the same height o the lie y. This places us at the correct coordiate for 1, We et draw a lie to the poit 1, f 1 o the graph of y f, ad the process cotiues. The process starts by drawig a lie from the iitial poit 0, 0 to 0, f 0. Here s what it looks like for f 2 si with 0 2. 5. 3.5 3 2.5 2 1.5 1 0.5 0 0 0.5 1 1.5 2 2.5 3 3.5 Aalysis of iteratios ear a fied poit: Take the equatio 1 f ad subtract the equatio f which is satisfies by a fied poit. The quatity we deote by e ad iterpret this quatity as the "error" of i approimatig. We obtai 1 f f, e 1 f f Now, by the mea value theorem from calculus, f f f c where c is betwee ad. Substitutig, we obtai e 1 f f f c f c e If is close to, we have the liear approimatio f f f ad so we ca also write, somewhat less precisely, e 1 f e From these results, we ca obtai the followig formulatios: 1) If is a fied poit of f with f 1 the the iteratio 1 f will coverge to provided 0 is sufficietly close to. 2) If is a fied poit of f with f r 1 for all i the iterval a, a the the iteratio 1 f will coverge to for ay 0 cotaied i a, a, ad e r e 0 will be true. Proof: From above, we have e 1 f c e ad if 0 is cotaied i a, a the f c r 1, so that e 1 r e 0. But the 1 is the also i a, a ad so e 2 r e 1 r 2 e 0 ad cotiuig, we have e r e 0, so that e 0 ad so. 3
3) If is a fied poit of f with a, b, ad f 1 for all i the iterval a, b where b a, the the iteratio 1 f will coverge to for ay 0 i a, b. (Sice we typically do t kow the value of this result is more practical tha 3). 4) If is a fied poit of f with f 1 the the iteratio 1 f caot coverge to uless eactly for some. (This is because if we are close to the error will icrease i size.) 5) For a a iteratio that coverges to a fied poit of f, we have for small values of the "error" e, e 1 f e. A coverget sequece with limit L that satisfies e 1 ae for some a satisfyig a 1, where e L, is said to be a liearly coverget sequece, or to coverge liearly. The result e 1 f e shows that the error approimately is reduced by a costat factor whe the begi to get close to. The practical effect of this is that sigificat digits i as a appromiatio to are accumulatig as a early costat rate. So if, for istace, 5 iteratios take you from 2 sigificat digits to 4 sigificat digits the aother 5 iteratios will result i about 6 sigificat digits. E: It is usually ot hard to come up with a iteratio that will work. For istace if we wat to solve 2 a, we might cosider rewritig it as a/ ad usig this as the basis of a iteratio 1 a/. However, this fails i a iterestig way (why?). However, cosider the followig sequece of steps: 2 a 2 a 1 a a 1 f Now f a 1 1 ad whe 2 a (a fied poit of f ) we have f a 1 1 so we ca obtai a coverget iteratio if we begi sufficietly 1 a 2 close to a. E. Cosider the equatio 4 si. Will the iteratio 1 4 si coverge to the fied poit i 0,? Ca you fid a iteratio with the same fied poit that will coverge? (Hit: Try addig a to both sides ad the dividig by a 1 ) I a dyamical system, 1 f, a fied poit is called a equilibrium. If we start out ear a equilibrium ad the system seds us back toward the equilibrium poit, the that poit is called a asymptotically stable equilibrium. We ca say the that if the equilibrium of 1 f satisfies f 1, the the equilibrium is asymptotically stable. O the other had, if a slight deviatio from eqiulibrium will sed us away from the equilibrium, the such a poit is called a ustable equilibrium. We ca the say that if the equilibrium 4
of 1 f satisfies f 1, the the equilibrium is ustable. Newto s method Newto s method, ad its variatios, represets probably the most popular root-fidig techique. It is simple to implemet ad it coverges very rapidly whe i the viciity of a root. However, as with iteratio covergece ca oly be guarateed whe startig sufficietly close to the root, ad the aalysis of covergece is a bit more difficult. Newto s method is easily geeralizable to ay umber of equatios/variables. It possesses quadratic covergece; the error e satisfies e 1 Ce 2. The idea of Newto s method is very simple: At a give guess, fid the Taylor polyomial P 1 of degree 1 about (the same as the liear approimatio) ad calculate the roots of P 1 as the et guess, 1. Thus we form P 1 f f ad we set 0 P 1 f f. Solvig for, we obtai f as out et guess. Newto s method therefore is to perform the f iteratio 1 f f Newto s method has a stadard graphical iterpretatio (at least i oe dimesio) which you should kow: We costruct a lie taget to the graph of f at ad follow that taget lie to where it hits the ais. This gives us our ew guess 1. Here s the picture for the followig problem: Solvig f 2 2 0 is the problem ad we have 3. Below, we costruct the taget lie at 3 ad where it hits the ais is 1 3 7/6. Oe cotiues this process ad quickly arrives at the solutio. Oe might ote that Newto s method for 2 a 0 is the iteratio 1 2 a 2 a ; ad the 2 2 iteratio will coverge to a if 0 0 ad will coverge to a if 0 0 (ca you see why graphically?). Of course this will oly work for a 0; however the iteratio is well defied if a 0. Fu project: What happes if a 0? y 15 10 5 0-5 1 2 3 4 Error aalysis for Newto s method: Newto s method, whe it coverges, ehibits quadratic covergece: e 1 Ce 2 for some costat C idepedet of. Oe might epect this from the fact that Newto s method is eact for liear fuctios. Ideed, if we eamie the iteratio 1 f f the this is of the form 1 g where g f. Now at the fied poit we seek, f we have f 0. Covergece will deped o the value of g. A easy calculatio 5
shows that g f ad so g 0. So if we aalyze covergece of 2 1 g, we have, usig a Taylor polyomial of first degree about to epad g, ad the Taylor remaider, we obtai e 1 g 1 g g g 1 2 g c 2 g e 1 1 2 g c 2 1 2 g c e 2, demostratig the quadratic covergece. Note that as usual, c is betwee ad. We ca i fact ow easily see that because f 0 we obtai g f f ad so 1 2 g c 1 f 2 f whe is ear ad so e 1 1 f 2 f e 2. Here is a more precise "mea-value type" error aalysis which you ca go through if you like: The guess 1 satisfies 0 f f 1. O the other had, epadig f about i a liear polyomial plus remaider, we have f f f 1 2! f c 2 ad so for we have 0 f f f 1 2! f c 2 with c betwee ad. Subtractig our two equatios: 0 f f 1 2! f c 2 0 f f 1 we obtai 0 f 1 2! 1 f c 2. Recallig that e, we obtai e 1 1 f c 2 f e 2. We ca observe that if e is small the c ad e 1 1 f 2 f e 2, as we previously obtaied. I Newto s method, ad ideed ay quadratically coverget method, the umber of sigificat digits approimately doubles at each iteratio: Suppose the relative error i is about 10 k 2, so that there are about k sigificat digits i. The e 1 Ce so e 1 C e 2 ad the umber of sigificat digits satisfies: k 1 log e 1 10 log 10 C 2 log e 10 log 10 C 2k The first term o the right is typically o bigger tha 1 or 2 i absolute value so k 1 2k is approimately true ad grows more accurate with larger k. We ca also say, somewhat more accurately, that if we gai k sigificat digits at iteratio, the we will gai 2k sigificat digits at the et iteratio. Roughly speakig, i other words, at each iteratio i Newto s method, we gai twice as may sigificat digits as we gaied o the previous iteratio. Net, we cosider how to formulate Newto s method i more tha oe variable. Here we are dealig with N equatios i N ukows. The priciple is the same: Approimate each equatio with a liear approimatio, ad the solve the resultig liear system of N equatios i N ukows. Recall that if we have a fuctio of more tha oe variable, say 6
for the fuctio f, y, z of three variables, the liear approimatio about a poit 0, y 0, z 0 is epressed i terms of the partial derivatives of f at the give poit: f, y, z f 0, y 0, z 0 0 y 0 0 0,y 0,z 0 0,y 0,z 0 0,y 0,z 0 I the above equatio, each of the partial derivatives is beig evaluated at the poit 0, y 0, z 0 ; below, we will abbreviate by, ad similarly for the other 0,y 0,z 0 0 partial derivatives. (There is a geeral Taylor epasio with a remaider formula for fuctios of several variables, but we will ot develop that further here.) Now suppose we are tryig to solve the three equatios f, y, z 0, g, y, z 0, h, y, z 0. I Newto s method each fuctio is replaced by its liear approimatio about the curret guess, which we deote, y, z. So, we must solve 0 f, y, z 0 g, y, z 0 h, y, z h h I matri form, this system ca be writte 0 f, y, z g, y, z h, y, z I matri form, let s deote h y y y y h h by X ad the h f, y, z g, y, z y y z by F X. The matri of z h, y, z, g, h partial derivatives above is called the Jacobia, ad usually deoted by J, y, z, y, z. We ll simply deote this matri by J X F X ad at X X, we ll use the otatio F. X Our liear system the has the matri form 0 F X F X X or, i the form X A b, ca be writte F X X F X. We ca solve for X symbolically usig X the matri iverse F 1 1 : X X F F X ad this defies the et guess X 1 : X X 1 X 1 X F F X X This is the vector form, or multivariable form, of Newto s method. The geeralizatio to ay umber of variables is simple to costruct. I this geeral form, the multivariable form of Newto s method looks much like Newto s method i oe variable: 1 1 f. The MATLAB laguage allows us, through its use of array ad matri operatios, to implemet the multivariable Newto s method i virtually the same way as the equatio above. However, as i oe dimesio, ad perhaps eve more so tha oe dimesio, the iitial guess will eed to be sufficietly close to the desired root i order for 7
the method to coverge. Here is a MATLAB implemetatio of Newto s method i ay umber of dimesios, with the fuctio Newtd. fuctio Newtd(f,Jacobia,0,ma,tol) %for solvig N equatios fu() 0 i N variables % is matri cotaiig sequece of guesses i its colums %0 always deotes curret guess, 0(:,) startig with 1 %fu is N-dimesioal vector fuctio, probably defied with M-file %Jacobia is NN matri fuctio, probably defied with M-file %get (:, 1) by solvig 0((:,)) Jacobia((:,))*(-(:,)) %so (:, 1)(:,)-Jacobia((:,))\f((:,)), usig MATLAB backslash operatio %for solvig liear systems of equatios %ma is maimum umber of iteratios %stop whe (:, 1)-(:,) tol as measured by maimum compoet differece 0; for 1:ma y0eval(f,0); 00-feval(Jacobia,0)\y0; [ 0]; if ma(abs((:, 1)-(:,))) tol,break,ed ed if ma,disp( Method failed );ed; The fuctio y f ad the the Jacobia matri fuctio A J are defied i separate M-files. The curret guess is always represeted by 0 ad a record of the guesses is cotaied i the matri. Note the use of the feval futio, which allows us to use fuctio ames as variables. Also ote the use of the backslash operatio i -feval(jacobia,0)\y0 for computig the solutio of the liear system F X X F X. X 8