A short presentation on Newton-Raphson method

A short presentation on Newton-Raphson method Doan Tran Nguyen Tung 5 Nguyen Quan Ba Hong 6 Students at Faculty of Math and Computer Science. Ho Chi Minh University of Science, Vietnam email. nguyenquanbahong@gmail.com blog. http://hongnguyenquanba.wordpress.com May 2, 2016 5 Student ID: 1411352 6 Student ID: 1411103 oan Tran Nguyen Tung 7 Nguyen Quan Ba Hong Newton-Raphson 8 (Studentsmethod at Faculty of Math and Computer May 2, 2016 Science. 1 Ho / 24 Chi

Outline 1 Introduction 2 The Newton-Raphson method Geometrical view point Analytical view point 3 Selected examples 4 The Newton-Raphson method has fallen! The Newton-Raphson method can go bad 5 Improvements 6 N-R method for several variables 7 Extension to systems of equations oan Tran Nguyen Tung 9 Nguyen Quan Ba Hong Newton-Raphson 10 (Studentsmethod at Faculty of Math and Computer May 2, 2016 Science. 2 Ho / 24Ch

Historical notes 1 This algorithm was discovered by Sir Issac Newton, who formulated the result in 1669. Later improved by Joseph Raphson in 1690, the algorithm is presently known as the Newton-Raphson method, or more commonly Newton s method. 2 A method for finding the roots of an arbitrary function that uses the derivative was first circulated by Isaac Newton in 1669. John Wallis published Newton s method in 1685, and in 1690 Joseph Raphson (1648 1715) published an improved version, essentially the form in which we use it today. 3 Newton s work was done in 1669 but published much later. Numerical methods related to the Newton Method were used by alkāshī, Viète, Briggs, and Oughtred, all many years before Newton. oan Tran Nguyen Tung 11 Nguyen Quan BaNewton-Raphson Hong 12 (Students method at Faculty of Math andmay Computer 2, 2016Science. 3 / Ho 24C

Later studies 1 Raphson, like Newton, seems unaware of the connection between his method and the derivative. The connection was made about 50 years later (Simpson, Euler), and the Newton Method finally moved beyond polynomial equations. The familiar geometric interpretation of the Newton Method may have been first used by Mourraille (1768). Analysis of the convergence of the Newton Method had to wait until Fourier and Cauchy in the 1820s. 2 The method was then studied and generalized by other mathematicians like Simpson (1710-1761), Mourraille (1720-1808), Cauchy (1789-1857), Kantorovich (1912-1986),... The important question of the choice of the starting point was first approached by Mourraille in 1768 and the difficulty to make this choice is the main drawback of the algorithm. oan Tran Nguyen Tung 13 Nguyen Quan BaNewton-Raphson Hong 14 (Students method at Faculty of Math andmay Computer 2, 2016Science. 4 / Ho 24C

Why does N-R method appear? Why does N-R method appear? Although the bisection method will always converge on the root, the rate of convergence is very slow. A faster method for converging on a single root of a function is the Newton-Raphson method. Perhaps it is the most widely used method of all locating formulas. Being one of iterative methods! 1 An initial estimate of the root. found by drawing a graph of the function in the neighborhood of the root. 2 The method is based upon a knowledge of the tangent to the curve near the root. 3 A powerful technique for solving equations numerically, based on the simple idea of linear approximation. oan Tran Nguyen Tung 15 Nguyen Quan BaNewton-Raphson Hong 16 (Students method at Faculty of Math andmay Computer 2, 2016Science. 5 / Ho 24C

Steps of N-R method The method consists of the following steps: 1 Pick a point x 0 close to a root. Find the corresponding point (x 0, f (x 0 )) on the curve. 2 Draw the tangent line to the curve at that point, and see where it crosses the x-axis. 3 The crossing point, x 1, is your next guess. Repeat the process starting from that point. oan Tran Nguyen Tung 17 Nguyen Quan BaNewton-Raphson Hong 18 (Students method at Faculty of Math andmay Computer 2, 2016Science. 6 / Ho 24C

Geometrical view point 1 Consider the following diagram showing a function f(x) with a simple root at x = x whose value is required. Initial analysis has indicated that the root is approximately located at x = x 0. The aim of any numerical procedure is to provide a better estimate to the location of the root. 2 The basic premise of the Newton-Raphson method is the assumption that the curve in the close neighbourhood of the simple root at x is approximately a straight line. Hence if we draw the tangent to the curve at x 0, this tangent will intersect the x axis at a point closer to x than is x 0 oan Tran Nguyen Tung 19 Nguyen Quan BaNewton-Raphson Hong 20 (Students method at Faculty of Math andmay Computer 2, 2016Science. 7 / Ho 24C

Figure 1 Figure 1: Diagram for the curve oan Tran Nguyen Tung 21 Nguyen Quan BaNewton-Raphson Hong 22 (Students method at Faculty of Math andmay Computer 2, 2016Science. 8 / Ho 24C

Figure 2 Figure 2: Figure 2. oan Tran Nguyen Tung 23 Nguyen Quan BaNewton-Raphson Hong 24 (Students method at Faculty of Math andmay Computer 2, 2016Science. 9 / Ho 24C

Establish the formula From the geometry of this diagram we see that x 1 = x 0 P Q But from the right-angled triangle PQR we have RQ P Q = tan θ = f (x 0 ) and so P Q = RQ f (x 0 ) = f (x 0) f (x 0 ) x 1 = x 0 f (x 0) f (x 0 ) If f(x) has a simple root near x 0 then a closer estimate to the root is x 1 where x 1 = x 0 f (x 0) f (x 0 ) This formula can be used time and time again giving rise to the following: oan Tran Nguyen Tung 25 Nguyen Quan BaNewton-Raphson Hong 26 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 10 / Ho 24C

The N-R iterative formula The Newton-Raphson iterative formula If f(x) has a simple root near x n then a closer estimate to the root is x n+1 where x n+1 = x n f (x n) f (x n ) This is the Newton-Raphson iterative formula. The iteration is begun with an initial estimate of the root, x 0, and continued to find x 1, x 2,... until a suitably accurate estimate of the position of the root is obtained. oan Tran Nguyen Tung 27 Nguyen Quan BaNewton-Raphson Hong 28 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 11 / Ho 24C

Analysis view point We suppose that f is a C 2 function on a given interval, then using Taylor s expansion near x f (x + h) = f (x) + hf (x) + O ( h 2) and if we stop at the first order (linearization of the equation), we are looking for a small h such as f (x + h) = 0 f (x) + hf (x) giving h = f(x) f (x) x + h = x f(x) f (x) oan Tran Nguyen Tung 29 Nguyen Quan BaNewton-Raphson Hong 30 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 12 / Ho 24C

Example for N-R method We just take one example, you should be try yourself! Example. f (x) = x 2 + ln x has a root near x = 1.5. Use the Newton-Raphson formula to obtain a better estimate. Solution. Here x 0 = 1.5, f (1.5) = 0.5 + ln 1.5 = 0.0945, f (x) = 1 + 1 x, f (1.5) = 1 + 1 1.5 = 5 3. Hence using the formula x 1 = 1.5 0.0945 1.6667 = 1.5567 The Newton-Raphson formula can be used again: this time beginning with 1.5567 as our initial estimate. This time use: x 2 = x 1 f(x 1) = 1.5567 = 1.5567 f(1.5567) f (x 1 ) 1.5567 2+ln 1.5567 1+ 1 1.5567 f (1.5567) = 1.5571 This is in fact the correct value of the root to 4 d.p. oan Tran Nguyen Tung 31 Nguyen Quan BaNewton-Raphson Hong 32 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 13 / Ho 24C

The N-R method can go bad The Newton-Raphson method is a GOD TOOL, isn t it? To answer that question, we will show some non-trivial examples. Example 1. Consider the function defined by f (x) = { x x, if x 0 0, if x = 0 Easy to prove that f is continuous. The derivative of this function is f (x) = 1 2 x, x 0 oan Tran Nguyen Tung 33 Nguyen Quan BaNewton-Raphson Hong 34 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 14 / Ho 24C

Example (continue) If we choose any starting point off the actual root, x 1 = a 0, then x 2 = a a a 1 2 a = a 2a = a If follows that x n = { a, if n odd a, if n even Example 2. Take f : R R, x x 2 x + 1 and x 0 = 1. As f (x) = 2x 1, x 1 = 1 f(1) f (1) = 1 1 1 = 0 and x 2 = 0 f(0) f (0) = 0 1 1 = 1. It follows that { 1, if n is even x n = 0, if n is odd Thus {x n } does not converge. oan Tran Nguyen Tung 35 Nguyen Quan BaNewton-Raphson Hong 36 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 15 / Ho 24C

Nothing be perfect! Once the Newton Method catches scent of the root, it usually hunts it down with amazing speed. But since the method is based on local information, namely f(x n ) and f (x n ), the Newton Method s sense of smell is deficient. If the initial estimate is not close enough to the root, the Newton Method may not converge, or may converge to the wrong root The successive estimates of the Newton Method may converge to the root too slowly, or may not converge at all. oan Tran Nguyen Tung 37 Nguyen Quan BaNewton-Raphson Hong 38 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 16 / Ho 24C

Drawbacks The Newton-Raphson methods has some drawbacks. 1 It cannot handle multiple roots. 2 It has slow convergence (compared with newer techniques). 3 The solution may diverge near a point of inflection. 4 The solution might oscillates new local minima or maxima. 5 With near-zero slope, the solution may diverge or reach a different root. oan Tran Nguyen Tung 39 Nguyen Quan BaNewton-Raphson Hong 40 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 17 / Ho 24C

Improvements We can improve the Newton-Raphson method to (some) following other iterations. 1 Cubic iteration. 2 Householder s iteration. 3 High order iteration: Householder s method. More details for these iterations, please read attachment file of my team. oan Tran Nguyen Tung 41 Nguyen Quan BaNewton-Raphson Hong 42 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 18 / Ho 24C

N-R method for several variables Target Newton s method may also be used to find a root of a system of two or more non linear equations { f (x, y) = 0 g (x, y) = 0 where f and g are C 2 functions on a given domain. Using Taylor s expansion of the two functions near (x, y) we find f (x + h, y + k) = f (x, y) + h f x + k f y + O ( h 2 + k 2) g (x + h, y + k) = g (x, y) + h g x + k g y + O ( h 2 + k 2) and if we keep only the first order terms, we are looking for a couple (h, k) such as oan Tran Nguyen Tung 43 Nguyen Quan BaNewton-Raphson Hong 44 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 19 / Ho 24C

f (x + h, y + k) = 0 f (x, y) + h f x + k f y g (x + h, y + k) = 0 g (x, y) + h g x + k g y hence it s equivalent to the linear system ( ) f f ( ) x y h = k equivalent to g x g y J (x, y) ( h k ) = ( f (x, y) g (x, y) ( f (x, y) g (x, y) This suggest to define the new process ( ) ( ) xn+1 xn = J 1 (x n, y n ) y n+1 y n ) ) ( f (xn, y n ) g (x n, y n ) ) under certain conditions (which are not so easy to check and this is again the main disadvantage of the method), it s possible to show that this process converges to a root of the system. The convergence remains quadratic. oan Tran Nguyen Tung 45 Nguyen Quan BaNewton-Raphson Hong 46 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 20 / Ho 24C

Example. N-R for system of equations Example We are looking for a root near (x 0 = 0.6, y 0 = 0.6) of the following system { f (x, y) = x 3 3xy 2 1 g (x, y) = 3x 2 y y 3 here the Jacobian and its inverse become ( x 2 J (x n, y n ) = 3 n yn 2 ) 2x n y n 2x n y n ( x 2 n yn 2 J 1 1 x 2 (x n, y n ) = n yn 2 2x n y n 3(x 2 n +y2 n )2 2x n y n x 2 n yn 2 ) oan Tran Nguyen Tung 47 Nguyen Quan BaNewton-Raphson Hong 48 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 21 / Ho 24C

Example (continue and the process gives x 1 = 0.40000000000000000000, y 1 = 0.86296296296296296296 x 2 = 0.50478978186242263605, y 2 = 0.85646430512069295697 x 3 = 0.49988539803643124722, y 3 = 0.86603764032215486664 x 4 = 0.50000000406150565266, y 4 = 0.86602539113638168322 x 5 = 0.49999999999999983928, y 5 = 0.86602540378443871965... Depending on your initial guess Newton s process may converge to one of the three roots of the system: ( 1 ) ( 3 2,, 1 ) 3 2 2,, (1, 0) 2 and for some values of (x 0, y 0 ) the convergence of the process may be tricky! oan Tran Nguyen Tung 49 Nguyen Quan BaNewton-Raphson Hong 50 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 22 / Ho 24C

Final generalization of this text For further studying and for references only. Extension to systems of equations f 1 (x 1,..., x n ) = 0... f m (x 1,..., x n ) = 0 or f(x) = 0 The Newton-Raphson method becomes x n+1 = x n J 1 (x n ) f (x n ), n = 0, 1,... oan Tran Nguyen Tung 51 Nguyen Quan BaNewton-Raphson Hong 52 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 23 / Ho 24C

References Adi Ben-Israel, A Newton-Raphson Method for the Solution of Systems of Equations, Technion-Israel Institute of Technology and Northwestern University, Journal of Mathematical Analysis and Applications 15, 243-252 (1966). Helm, Workbook Level 1, The Newton-Raphson Method, March 18, 2004. Aaron Burton, Newton s method and fractals. David M. Bressoud, Newton-Raphson Method, Appendix to A Radical Approach to Real analysis 2nd edition, June 20, 2006. Pascal Sebah, Xavier Gourdon, Newton s method and high order iterations, October 3, 2001. Dr. Ibrahim A. Assakkaf, Numerical methods for engineers, Spring 2001. oan Tran Nguyen Tung 53 Nguyen Quan BaNewton-Raphson Hong 54 (Students method at Faculty of Math andmay Computer 2, 2016 Science. 24 / Ho 24C