Notes on iteration and Newton s method. Iteration

Transcription

1 Notes o iteratio ad Newto s method Iteratio Iteratio meas doig somethig over ad over. I our cotet, a iteratio is a sequece of umbers, vectors, fuctios, etc. geerated by a iteratio rule of the type 1 f where f is some fied fuctio or operatio. We eed to provide a startig value for the sequece to get the process goig, say 0 or 1 (whichever is more coveiet to call this startig value) ad the the iteratio rule geerates the terms that follow i the sequece. We usually call 0 the "iitial coditio" of the iteratio. I may applicatios, the iteratio 1 f represets, or ca be iterpreted as, the evolutio of a "dyamical system", i which the ide plays the role of time. E1. 1 a, with a a fied umber ad a real umber. Thus we are usig the iteratio rule 1 f with the choice of f a. I this case it is easy to calculate, give 0, that a 0. This is a case where we ca eplicitly calculate i terms of. Ofte, that is ot possible. This eample is a eample of a "liear iteratio" because the fuctio f is liear. E I this case oe caot obtai a eplicit formula for. Oe might ask, however, (ad oe ca aswer) importat qualitative questios such as, does?, ad oe ca eve obtai approimatios for the value of i terms of. This is a "oliear" iteratio without a eplict formula for, but evertheless it is oe that ca be aalyzed fairly well. E3: 1 A where A is a m mmatri ad 1,... are m 1 vectors. This very simple, but very importat liear iteratio, has the "eplicit" solutio A 0 but actually eplicitly givig a formula for A is a complicated problem requirig the theory of eigevalues ad eigevectors. E4.: 1 1. Here is a parameter (much like a is i the iteratio 1 a ). The "allowable" values of are 0 4 because for these values of, if we start with the 0 1 for all. (this should t be too hard to figure out yourself). This is the famous "chaotic" iteratio. For some values of the iteratio behaves very icely ad predictably, ad eve has a limit, but for other values of, the seem to wader upredictably over the iterval 0, 1. E5: Here we costruct a sequece of fuctios g 0, g 1,.., usig the iteratio g g t dt. If we start with g 0 0, we obtai g 1 1, g 2 1,... What happes? If you fi a value of, what is g approachig for large? This is a very simple-lookig iteratio but iteratios of this type, resultig i a sequece of fuctios, are very importat i mathematical theory. E6: Cellular automata. We discussed these before. I this case represets a ifiite strog of zeros ad 1 s ad the fuctio f is the rule by which we geerate the et state of the system, 1. 1

2 Fied poits ad equilibria i iteratios: Give a iteratio 1 f, if a value satisfies f the is called a "fied poit" of the iteratio 1 f, or simply a fied poit of f. I the case of the dyamical systems iterpretatio of 1 f, we ote that if, the 1 ad 2,... i.e. the values of stay fied. We refer the to as a "equilibrium poit", or simply, a equilibrium, of the system 1 f. We are particularly iterested i iteratios that coverge to a fied poit, i.e. i iteratios 1 f for which, where is a fied poit. (This may, of course, deped o the right choice of the startig value 0 ). Ideed, if a ad f is cotiuous, the it automatically follows that a f a must be true. We will use such iteratios to umerically compute solutios to certai equatios that we have put ito the form f. To determie where a fied poit eists, you ca simply graph y f ad y ad look for itersectio poits, or graph f ad look for zeroes. More precisely, you are lookig (either lookig literally o the graph, or figuratively by computig umbers) for values a, b of for which a f a ad b f b have opposite sig (i.e. oe is positive ad oe is egative). The, assumig that f is cotiuous (it almost always is), there must be a fied poit betwee a ad b. E: Suppose we wish to umerically solve 2 si (there is o "eact" aswer). Graphically, we observe a solutio below i the viciity of 2. y Now we eecute the iteratio 1 2 si with 0 2, ad obtai By ow it appears that the iterates (the sequece of values) is covergig, boucig back ad forth while the distace betwee them shriks. Note that we do t kow what the "eact" aswer is, but we ca prove (we ll see how later) that the iterates will coverge to 2

3 the eact aswer. Graphical view of iteratio: As i the graph above, we plot the fuctio y f ad the lie y. The we draw a lie from our curret poit, f o the graph y f to the poit, f 1, 1 that lies at the same height o the lie y. This places us at the correct coordiate for 1, We et draw a lie to the poit 1, f 1 o the graph of y f, ad the process cotiues. The process starts by drawig a lie from the iitial poit 0, 0 to 0, f 0. Here s what it looks like for f 2 si with Aalysis of iteratios ear a fied poit: Take the equatio 1 f ad subtract the equatio f which is satisfies by a fied poit. The quatity we deote by e ad iterpret this quatity as the "error" of i approimatig. We obtai 1 f f, e 1 f f Now, by the mea value theorem from calculus, f f f c where c is betwee ad. Substitutig, we obtai e 1 f f f c f c e If is close to, we have the liear approimatio f f f ad so we ca also write, somewhat less precisely, e 1 f e From these results, we ca obtai the followig formulatios: 1) If is a fied poit of f with f 1 the the iteratio 1 f will coverge to provided 0 is sufficietly close to. 2) If is a fied poit of f with f r 1 for all i the iterval a, a the the iteratio 1 f will coverge to for ay 0 cotaied i a, a, ad e r e 0 will be true. Proof: From above, we have e 1 f c e ad if 0 is cotaied i a, a the f c r 1, so that e 1 r e 0. But the 1 is the also i a, a ad so e 2 r e 1 r 2 e 0 ad cotiuig, we have e r e 0, so that e 0 ad so. 3

4 3) If is a fied poit of f with a, b, ad f 1 for all i the iterval a, b where b a, the the iteratio 1 f will coverge to for ay 0 i a, b. (Sice we typically do t kow the value of this result is more practical tha 3). 4) If is a fied poit of f with f 1 the the iteratio 1 f caot coverge to uless eactly for some. (This is because if we are close to the error will icrease i size.) 5) For a a iteratio that coverges to a fied poit of f, we have for small values of the "error" e, e 1 f e. A coverget sequece with limit L that satisfies e 1 ae for some a satisfyig a 1, where e L, is said to be a liearly coverget sequece, or to coverge liearly. The result e 1 f e shows that the error approimately is reduced by a costat factor whe the begi to get close to. The practical effect of this is that sigificat digits i as a appromiatio to are accumulatig as a early costat rate. So if, for istace, 5 iteratios take you from 2 sigificat digits to 4 sigificat digits the aother 5 iteratios will result i about 6 sigificat digits. E: It is usually ot hard to come up with a iteratio that will work. For istace if we wat to solve 2 a, we might cosider rewritig it as a/ ad usig this as the basis of a iteratio 1 a/. However, this fails i a iterestig way (why?). However, cosider the followig sequece of steps: 2 a 2 a 1 a a 1 f Now f a 1 1 ad whe 2 a (a fied poit of f ) we have f a 1 1 so we ca obtai a coverget iteratio if we begi sufficietly 1 a 2 close to a. E. Cosider the equatio 4 si. Will the iteratio 1 4 si coverge to the fied poit i 0,? Ca you fid a iteratio with the same fied poit that will coverge? (Hit: Try addig a to both sides ad the dividig by a 1 ) I a dyamical system, 1 f, a fied poit is called a equilibrium. If we start out ear a equilibrium ad the system seds us back toward the equilibrium poit, the that poit is called a asymptotically stable equilibrium. We ca say the that if the equilibrium of 1 f satisfies f 1, the the equilibrium is asymptotically stable. O the other had, if a slight deviatio from eqiulibrium will sed us away from the equilibrium, the such a poit is called a ustable equilibrium. We ca the say that if the equilibrium 4

5 of 1 f satisfies f 1, the the equilibrium is ustable. Newto s method Newto s method, ad its variatios, represets probably the most popular root-fidig techique. It is simple to implemet ad it coverges very rapidly whe i the viciity of a root. However, as with iteratio covergece ca oly be guarateed whe startig sufficietly close to the root, ad the aalysis of covergece is a bit more difficult. Newto s method is easily geeralizable to ay umber of equatios/variables. It possesses quadratic covergece; the error e satisfies e 1 Ce 2. The idea of Newto s method is very simple: At a give guess, fid the Taylor polyomial P 1 of degree 1 about (the same as the liear approimatio) ad calculate the roots of P 1 as the et guess, 1. Thus we form P 1 f f ad we set 0 P 1 f f. Solvig for, we obtai f as out et guess. Newto s method therefore is to perform the f iteratio 1 f f Newto s method has a stadard graphical iterpretatio (at least i oe dimesio) which you should kow: We costruct a lie taget to the graph of f at ad follow that taget lie to where it hits the ais. This gives us our ew guess 1. Here s the picture for the followig problem: Solvig f is the problem ad we have 3. Below, we costruct the taget lie at 3 ad where it hits the ais is 1 3 7/6. Oe cotiues this process ad quickly arrives at the solutio. Oe might ote that Newto s method for 2 a 0 is the iteratio 1 2 a 2 a ; ad the 2 2 iteratio will coverge to a if 0 0 ad will coverge to a if 0 0 (ca you see why graphically?). Of course this will oly work for a 0; however the iteratio is well defied if a 0. Fu project: What happes if a 0? y Error aalysis for Newto s method: Newto s method, whe it coverges, ehibits quadratic covergece: e 1 Ce 2 for some costat C idepedet of. Oe might epect this from the fact that Newto s method is eact for liear fuctios. Ideed, if we eamie the iteratio 1 f f the this is of the form 1 g where g f. Now at the fied poit we seek, f we have f 0. Covergece will deped o the value of g. A easy calculatio 5

6 shows that g f ad so g 0. So if we aalyze covergece of 2 1 g, we have, usig a Taylor polyomial of first degree about to epad g, ad the Taylor remaider, we obtai e 1 g 1 g g g 1 2 g c 2 g e g c g c e 2, demostratig the quadratic covergece. Note that as usual, c is betwee ad. We ca i fact ow easily see that because f 0 we obtai g f f ad so 1 2 g c 1 f 2 f whe is ear ad so e 1 1 f 2 f e 2. Here is a more precise "mea-value type" error aalysis which you ca go through if you like: The guess 1 satisfies 0 f f 1. O the other had, epadig f about i a liear polyomial plus remaider, we have f f f 1 2! f c 2 ad so for we have 0 f f f 1 2! f c 2 with c betwee ad. Subtractig our two equatios: 0 f f 1 2! f c 2 0 f f 1 we obtai 0 f 1 2! 1 f c 2. Recallig that e, we obtai e 1 1 f c 2 f e 2. We ca observe that if e is small the c ad e 1 1 f 2 f e 2, as we previously obtaied. I Newto s method, ad ideed ay quadratically coverget method, the umber of sigificat digits approimately doubles at each iteratio: Suppose the relative error i is about 10 k 2, so that there are about k sigificat digits i. The e 1 Ce so e 1 C e 2 ad the umber of sigificat digits satisfies: k 1 log e 1 10 log 10 C 2 log e 10 log 10 C 2k The first term o the right is typically o bigger tha 1 or 2 i absolute value so k 1 2k is approimately true ad grows more accurate with larger k. We ca also say, somewhat more accurately, that if we gai k sigificat digits at iteratio, the we will gai 2k sigificat digits at the et iteratio. Roughly speakig, i other words, at each iteratio i Newto s method, we gai twice as may sigificat digits as we gaied o the previous iteratio. Net, we cosider how to formulate Newto s method i more tha oe variable. Here we are dealig with N equatios i N ukows. The priciple is the same: Approimate each equatio with a liear approimatio, ad the solve the resultig liear system of N equatios i N ukows. Recall that if we have a fuctio of more tha oe variable, say 6

7 for the fuctio f, y, z of three variables, the liear approimatio about a poit 0, y 0, z 0 is epressed i terms of the partial derivatives of f at the give poit: f, y, z f 0, y 0, z 0 0 y 0 0 0,y 0,z 0 0,y 0,z 0 0,y 0,z 0 I the above equatio, each of the partial derivatives is beig evaluated at the poit 0, y 0, z 0 ; below, we will abbreviate by, ad similarly for the other 0,y 0,z 0 0 partial derivatives. (There is a geeral Taylor epasio with a remaider formula for fuctios of several variables, but we will ot develop that further here.) Now suppose we are tryig to solve the three equatios f, y, z 0, g, y, z 0, h, y, z 0. I Newto s method each fuctio is replaced by its liear approimatio about the curret guess, which we deote, y, z. So, we must solve 0 f, y, z 0 g, y, z 0 h, y, z h h I matri form, this system ca be writte 0 f, y, z g, y, z h, y, z I matri form, let s deote h y y y y h h by X ad the h f, y, z g, y, z y y z by F X. The matri of z h, y, z, g, h partial derivatives above is called the Jacobia, ad usually deoted by J, y, z, y, z. We ll simply deote this matri by J X F X ad at X X, we ll use the otatio F. X Our liear system the has the matri form 0 F X F X X or, i the form X A b, ca be writte F X X F X. We ca solve for X symbolically usig X the matri iverse F 1 1 : X X F F X ad this defies the et guess X 1 : X X 1 X 1 X F F X X This is the vector form, or multivariable form, of Newto s method. The geeralizatio to ay umber of variables is simple to costruct. I this geeral form, the multivariable form of Newto s method looks much like Newto s method i oe variable: 1 1 f. The MATLAB laguage allows us, through its use of array ad matri operatios, to implemet the multivariable Newto s method i virtually the same way as the equatio above. However, as i oe dimesio, ad perhaps eve more so tha oe dimesio, the iitial guess will eed to be sufficietly close to the desired root i order for 7

8 the method to coverge. Here is a MATLAB implemetatio of Newto s method i ay umber of dimesios, with the fuctio Newtd. fuctio Newtd(f,Jacobia,0,ma,tol) %for solvig N equatios fu() 0 i N variables % is matri cotaiig sequece of guesses i its colums %0 always deotes curret guess, 0(:,) startig with 1 %fu is N-dimesioal vector fuctio, probably defied with M-file %Jacobia is NN matri fuctio, probably defied with M-file %get (:, 1) by solvig 0((:,)) Jacobia((:,))*(-(:,)) %so (:, 1)(:,)-Jacobia((:,))\f((:,)), usig MATLAB backslash operatio %for solvig liear systems of equatios %ma is maimum umber of iteratios %stop whe (:, 1)-(:,) tol as measured by maimum compoet differece 0; for 1:ma y0eval(f,0); 00-feval(Jacobia,0)\y0; [ 0]; if ma(abs((:, 1)-(:,))) tol,break,ed ed if ma,disp( Method failed );ed; The fuctio y f ad the the Jacobia matri fuctio A J are defied i separate M-files. The curret guess is always represeted by 0 ad a record of the guesses is cotaied i the matri. Note the use of the feval futio, which allows us to use fuctio ames as variables. Also ote the use of the backslash operatio i -feval(jacobia,0)\y0 for computig the solutio of the liear system F X X F X. X 8