NUMERICAL ANALYSIS. K.D. Anderson J.S.C. Prentice C.M. Villet

Transcription

1 NUMERICAL ANALYSIS KD Anderson JSC Prentice CM Villet

2 This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4 International License To view a copy of this license, visit Last updated: 1 October 15 Disclaimer: Links are followed at own risk when viewing this document electronically

3 Contents 1 INTRODUCTION 4 11 Numerical methods and analysis Numerical analysis 4 11 Types of methods 5 1 Number representations and round-off errors 5 11 Decimal number representation 5 1 Error 6 SERIES EXPANSIONS 7 1 Introduction 7 Sequences 7 3 Series 1 4 Taylor series expansions Taylor s Theorem (1715) 1 4 Taylor series 1 43 Lagrange estimate of the error term Expansion of multivariable functions 15 3 NONLINEAR EQUATIONS Introduction 16 3 The bisection method Linear interpolation Newton s method Fixed-point iteration 36 Systems of nonlinear equations 4 SYSTEMS OF LINEAR EQUATIONS 4 41 Introduction 4 4 Solvability 4 43 Cramer s rule 4 44 The Jacobi method 5 5 APPROXIMATION METHODS 7 51 Introduction 7 5 Polynomial interpolation 7 53 Lagrange s method 8 54 Least-squares curve fitting 3 55 Least-squares polynomial fitting Approximation with Chebyshev polynomials Definition Minimal property Expansion of a function in terms of Chebyshev polynomials 36 3

4 4 CONTENTS 6 NUMERICAL DIFFERENTIATION 4 61 Introduction 4 6 First derivative 4 63 Second derivative Higher-order derivatives 4 7 NUMERICAL INTEGRATION Introduction 44 7 Newton-Cotes formulae Trapezium rule Simpson s rule An analytical complication 49 8 ORDINARY DIFFERENTIAL EQUATIONS Introduction 51 8 One-step methods 5 83 Euler s method 5 84 The modified Euler method 5 85 The Runge-Kutta methods The second-order Runge-Kutta method (RK) The fourth-order Runge-Kutta method (RK4) Approximation error in one-step methods 55

5 Chapter 1 INTRODUCTION 11 Numerical methods and analysis When solving a mathematical problem, such as determining a definite integral or solving a differential equation, we attempt to do so analytically we determine an expression for the indefinite integral and then substitute the limits, or we apply an appropriate technique to find an expression that relates the dependent and independent variables of the DE The precise definition of an analytic expression or solution is one that can be expressed in terms of a bounded number of certain elementary functions: constants (including complex numbers), one variable x, elementary operations of arithmetic (+ ), n-th roots, exponents (which includes trigonometric functions and inverse trigonometric functions) and logarithms However, we are often confronted with mathematically posed problems that simply cannot be solved analytically, such as the transcendental equation sinx 65x = (11) or the non-linear differential equation dn dt = an k(t)n17 (1) both of which we shall revisit in later chapters Problems which cannot be solved analytically are generally nonlinear in nature, which is clearly the case with these two equations The only hope we have of dealing with such problems is by finding a numerical solution In equation(11) this wouldbe anumericalvalue forxthat satisfiesthe equation; inequation(1) the numerical solution is a set of numbers that represents the true solution over the relevant interval of integration Numerical methods are the mathematical tools we use to find such numerical solutions 111 Numerical analysis As the term suggests, numerical analysis is the mathematical study of numerical methods and, in the broader sense, the analysis of the field of numerical methods as a whole Numerical analysis addresses the following: (a) The derivation of numerical methods from fundamental mathematical ideas (b) Investigating the properties of numerical methods, such as accuracy and stability Numerical methods tend to be approximation techniques, ie they yield approximate solutions rather than exact solutions Through appropriate analysis of the method, we may understand the nature of the approximation error and we will probably be able to successfully implement the method subject to a desired level of accuracy If, through appropriate analysis of Consequently, the analysis of numerical methods is an extremely important part of the field 5

6 6 1 INTRODUCTION 11 Types of methods In these notes, we study the following methods: (a) Nonlinear algebraic equations in one variable - chapter 3 - we investigate the bisection method, linear interpolation, Newton s method, and fixed-point iteration (b) Systems of linear equations - chapter 4 - we investigate the Jacobi method that can be used to find numerical solutions to large systems of linear equations (c) Approximation of functions and data sets - chapter 5 - we investigate polynomial and Lagrange interpolation, least-squares polynomial fitting, and Chebyshev series (d) Numerical differentiation - chapter 6 - Approximations to derivatives of various order using Taylor series (e) Numerical integration (quadrature) - chapter 7 - we study the Trapezium and Simpson methods, based on Lagrange interpolation, used to approximate definite integrals (f) Initial-value problems - chapter 8 - Euler s method, modified Euler method, Runge-Kutta methods of orders two and four, used to approximate initial-value problems arising from ordinary differential equations We will devote our efforts to the derivation and error analysis of most of these methods, and we will demonstrate the methods by means of numerical examples Consequently, to a large extent, these notes are theoretical in nature 1 Number representations and round-off errors Numbers play an important part of numerical analysis (and mathematics in general with a whole field dedicated to their study) We normally think of numbers as constant-valued entities and use the numerals,1,,3,4,5,6,7,8,9, and combinations thereof, to represent them, eg, 4, 5345, etcetera A k-digit number is represented by using k numerals It should be clear the k must be a natural number, and we refer to the number as a finite digit number It is also quite possible for a number to be a -digit number, we simply refer to such a number as an infinitely digit number The natural numbers, integers, and rational numbers may all be represented as finite digit numbers while the irrational and real numbers are mostly infinite digit numbers For example, 1 is a 1-digit number, 1 is a -digit number and is a 5-digit number 11 Decimal number representation In the study of numerical analysis, we might want to make use of a finite precision device, eg a calculator or computer, to perform computations Finite precision devices tend to have limited memory and might only be able to store and represent finite digit numbers quite accurately Due to this memory constraint, there is a very specific way of representing numbers and using these numbers in computations within a finite precision environment Consider the mathematical constant π, which we know to be an irrational number and may be represented as the fraction 7 A finite precision device cannot interpret the meaning of the symbol π and we rather make use of 7 in computations We know that π can also be written as We call this representation of the number the decimal representation or decimal form of the number As another example, we know that we can represent 1 as 5 Most real numbers are represented this way We define the normalised floating-point form of a number to be ±d 1 d d k 1 n, (13)

7 1 INTRODUCTION 7 where d i, called a decimal digit, is an integer-valued number with 1 d 1 9 and d i 9, for each i =,3,,k We call any number represented by (13) a k-digit decimal number Thus, π in normalised decimal floating-point form is and the normalised floating-point form of 1 8, ie 15, which is a 3-digit decimal number Consider a positive real number y with the normalised floating-point form y = d 1 d d k d k+1 d k+ 1 n The finite floating-point form of y, which we denote fl(y), is obtained by terminating y after k decimal digits This termination is done in one of two ways The first way, called truncation or chopping, is done by simply getting rid of the digits d k+1 d k+, which produces fl(y) = d 1 d d k 1 n The second way, called rounding, is done by adding 5 1 n (k+1) to y and then chopping the result to obtain the floating-point form fl(y) = δ 1 δ δ k 1 n When rounding, if d k+1 5, we add 1 to d k to obtain fl(y), this is called rounding up; if d k+1 < 5, we simply chop off all the digits after the first k digits, this is called rounding down It should be clear that rounding down yields δ i = d i for each i = 1,,3,,k, but this is not necessarily the case when rounding up 1 Error Round-off error refers to errors in representing numbers in a finite precision environment For example, we know that π = but in most desktop computers we find that π = In other words, due to memory constraints as mentioned earlier, the value of π is rounded off to 14 decimal places In a finite precision device this round-off process is applied to all numerical values, so that round-off error is present most of the time (an obvious exception is the integers, which do not have a fractional part) The digits lost in this rounding-off process constitute the round-off error By and large, we do not expect round-off error to significantly compromise calculations performed on a computer (round-off error is typically of the order of 1 15 ), although sometimes round-off errors may be amplified in the course of a computation We generally find that the approximation errors mentioned previously are far more significant than round-off error, and so we restrict our study of error to approximation errors rather than round-off errors in these notes Indeed, it is appropriate to regard approximation errors as a consequence of the mathematical nature of the numerical method itself, whereas round-off errors may be seen as a technological limitation the price we pay for using finite-precision devices

8 Chapter SERIES EXPANSIONS 1 Introduction Sequences and series play an important role in numerical analysis For example, Newton s method (see 34) generates a sequence of approximations to the root of a non-linear equation We review the theory of sequences and series and describe a useful power series that may be used to approximate functions about a given point Sequences A sequence of real numbers is a function from the natural numbers N = {1,,} onto the real numbers R, ie 1 x 1 x 3 x 3 4 x 4 n x n Thus f(n) = x n The numbers x 1,x,x 3,,x n,, in the range of the function, are called the elements or terms of the sequence A sequence is called infinite if it has an infinite amount of terms, otherwise it is called finite Example 1 Consider the following

9 SERIES EXPANSIONS 9 This is indeed a sequence, because the numbers on the left are in N and the numbers on the right are in some subset of R We note that the sequence is generated by the function f(n) = n A sequence is denoted by its elements x 1,x,x 3,, or using the shorter notation (x n ) n=1, where n N This latter notation is simplified to (x n ) if clear from context that we are dealing with an infinite or a finite sequence Parentheses are deliberately used to emphasize the importance of ordering in a sequence If we consider the set {1,,3,4} and rewrite it as {4,3,,1}, then it is still considered to be the same set However, when we consider the sequence (1,,3,4) and rewrite it as (4,3,,1), then the two sequences are considered to be two different Why are these two sequences considered to be different from each other? Most often a sequence is defined by giving a formula for its nth term x n For example, consider the sequence of reciprocals of the odd numbers (1, 13, 15, 17, ) This sequence may be written as ( 1 ) n+1 : n N or more simply x n = 1 n+1, where it is understood that n N Another way of defining a sequence is to specify the value for x 1 and giving a formula for x n+1 in terms of x n, the sequence is then said to be defined recursively or inductively Example Consider the Fibonacci sequence F = (f n ) given by 1, 1,, 3, 5, 8, 13, 1, If we specify f 1 = 1 and f = 1, then we may give the recursion formula for every other term in the sequence as f n+1 = f n +f n 1, where n The limit of a sequence (x n ) is the real number l such that lim x n = l (1) n A sequence (x n ) is called convergent if its limit l exists; if this limit does not exist, then the sequence is called divergent It should be noted that the limit of a sequence is unique Given two sequences (x n ) and (y n ) such that lim n x n = l 1 and lim n y n = l, then lim n (x n ±y n ) = l 1 ±l lim (cx n) = cl 1, c R n lim (x ny n ) = l 1 l n ) lim n ( xn y n = l 1 l, y n and l lim (x n) (yn) = x y, x > and x n > n There are numerous other useful limit theorems and the reader is referred to consult an analysis text

10 1 SERIES EXPANSIONS Example 3 Consider the sequence ( ) 1 n If n then 1 n and thus Hence is the limit of the sequence lim n ( ) 1 = n A sequence (x n ) is bounded if there exists numbers a,b R such that a < x n < b for all n N The number a is called the lower bound of the sequence and the number b is called the upper bound of the sequence Example 4 Consider the sequence of even, positive integers (,4,6,8,) It should not be difficult to see that the number 1 is a lower bound of this sequence Next we consider the sequence of reciprocals of the even, positive integers ( 1, 1 4, 1 6, 1 8,) In this case, the number 1 is the upper bound of the sequence A sequence (x n ) is called increasing if x 1 x x 3 x n x m for all n < m and n,m N Similarly, the sequence (x n ) is called decreasing if x 1 x x 3 x n x m for all n < m and n,m N A sequence (x n ) is called monotone if it is increasing or decreasing It can be shown that a monotonic sequence is convergent if and only if it is bounded Example 5 A ball, with diameter ǫ, is dropped from a height h Each time it drops h metres, it rebounds rh metres Determine how many times the ball bounces before it stops moving Solution: We note that if the initial height was h, then after the first bounce the ball will reach a height of h 1 = rh, the height after the second bounce would be h = rh 1, and after n bounces the height would be h n = rh n 1 But it follows that h n = rh n 1 = r ( rh n ) = r h n = and we conclude that h n = r n h Clearly, the bounces of the ball form a recursive sequence The ball stops bouncing when h n ǫ Since h n = r n h, it follows that r n h ǫ r n ǫ h ( ) ǫ nln(r) ln h n ln(ǫ) ln(h ) ln(r) If h = 8m, r = 7 and ǫ = 7m, then we find n and we conclude that it would take 14 bounces before the balls stops

11 SERIES EXPANSIONS 11 3 Series Informally, a series is the sum of the terms of a sequence If we let (x n ) be a sequence, then sum of the first k terms of this sequence, ie s k = x 1 +x + +x k = k x i, k n is called the kth partial sum of the sequence Note that the partial sums form a sequence by themselves, ie s 1 = x 1 s = s 1 +x (= x 1 +x ) s 3 = s +x 3 (= x 1 +x +x 3 ) i=1 s n = s n 1 +x n (= x 1 +x +x 3 + +x n 1 +x n ) Thispairofsequences ( (x n ),(s n ) ) iscalledtheseries generatedbythesequence(x n ) Thenumbers x i, i = 1,,,n, in the partial sums are called the terms of the sequence A seriesis called infinite if it has an infinite amount of terms, otherwise it is called finite if it has a finite amount of terms Instead of writing ( (x n ),(s n ) ) every time to denote a series generated by (x n ), it is convention to use the notation n x i = x 1 +x +x 3 + +x n i=1 if dealing with a finite series; for an infinite series one would similarly have x i = x 1 +x +x 3 + i=1 The symbol Σ is the Greek letter sigma, and one often refers to the notation above as sigmanotation When clear from context whether we are dealing with a finite or infinite series, the sigma-notation is shortened to i x i Let i=1 x i be a series If the sequence (s n ) of partial sums of this series converges to the limit s, then the series is called convergent and the limit s is called the sum of the series The sum is denoted s = If this limit does not exist, then the series is said to be divergent i=1 x i Example 6 Consider the series i=1c, where c R is a constant Clearly this infinite series does not have a sum and thus is divergent However, if we were to consider the finite series n i=1c, then it should be clear that this series does have a sum and is convergent The sum is none other than s = nc Two very important questions arise when studying series 1 Does the series converge or diverge? What is the sum of the series if it is convergent?

12 1 SERIES EXPANSIONS Example 7 Given the infinite series ar i = a+ar+ar +ar 3 + +ar i +, () i= where a R is a non-zero constant Consider the nth partial sum s n = a + ar + ar + + ar n If we multiply s n by r and subtract the result from s n, then we obtain and therefore s n rs n = (a+ar+ar + +ar n ) r(a+ar+ar + +ar n ) = a+ar+ar + +ar n ar ar ar n+1 = a ar n+1 s n (1 r) = a(1 r n+1 ) s n = a(1 rn+1 ) 1 r If < r < 1, then the term r n+1 as n and we obtain the sum of the geometric series () as s = a 1 r (3) The series () is called a geometric series A simpler way of determining whether a series converges or diverges is to make use of the following theorem: Theorem 1 (Ratio Test) The series n= a n converges if lim a n < 1 n a n 1 and diverges if lim n a n a n 1 > 1 If this limit is equal to 1, then we cannot conclude anything about the convergence or divergence of the series Example 8 Let us reconsider the geometric series () By the ratio test, we now have x n = ar n = r x n 1 ar n 1 and thus we require r < 1 for convergence This concurs with our earlier analysis in example 7 4 Taylor series expansions From analysis it is known that a continuous function may be approximated by finite or infinite series, and these approximations are normally done by power series expansions A power series is

13 SERIES EXPANSIONS 13 an infinite series of the form a i x i = a +a 1 x+a x + +a n x n + (4) i= Example 7 is a special case of a power series In the following sections we shall discuss a very useful power series expansion the Taylor series expansion of a function which is a powerful analytical and numerical tool in applied mathematics and is used extensively in later chapters 41 Taylor s Theorem (1715) We consider a function f that has continuous derivatives up to (n+1)th order on an interval [a,b] From the fundamental theorem of calculus we have Repeated integration by parts yields f(b) = f(a)+ b a f (x)dx f(b) = f(a)+(x b) [ f (x) ] b b + (x b)f (x) dx a a [ ] (b x) = f(a)+(b a)f b b (a)+ f (b x) (x) + f (x) dx a a = f(a)+(b a)f (a)+ (b a) f (a)! ] b b + [ (b x)3 (b x) 3 3 f (x) + a a 3 f(4) (x) dx = f(a)+(b a)f (a)+ (b a) f (a)! b + (b a)3 f (b x) 3 (a)+ f (4) (x) dx 3! 3! After n steps we have Taylor s Theorem: a f(b) = f(a)+(b a)f (a)+ + (b a)n f (n) (a)+r n n! b (b x) n R n = f (n+1) (x) dx n! a With the substitutions b x,a x,x t we obtain the more familiar form f(x) = f(x )+(x x )f (x )+ + (x x ) n f (n) (x )+R n n! x (x t) n R n = f (n+1) (t) dt x n! We refer to R n as the residual term 4 Taylor series Consider a function f for which derivatives of all orders exist at x In other words, we may write (5) for arbitrary n The power series in (x x ) on the right-hand side of (5) will converge to a finite value if lim R n = Hence, we may expand f(x) then as an infinite power series: n (x x ) n f(x) = f (n) (x ) (6) n! n= The convergence, or lack thereof, of the series (6) may be investigated by means of estimates of R n Cauchy and Lagrange have given estimates for the residual term (see 43), but we could also make use of theorem 1 (5)

14 14 SERIES EXPANSIONS Example 9 We want to write f(x) = e x as an infinite power series Since e x and all its derivatives exist at x =, we choose x in (6) Then and so e x = f(x) = e x f() = 1 f (x) = e x f () = 1 f (n) (x) = e x f (n) () = 1 n= x n n! = 1+x+ x! + x3 3! + We investigate the convergence of this series using theorem 1: e x = n= a n a n = xn n! Hence a n a n 1 so that a n ( )/( ) x n x n 1 = = x n! (n 1)! n a n 1 Thus, this series converges for all values of x = x n as n Example 1 We obtain the so-called binomial expansion by determining the Taylor series of f(x) = (1+x) p for all real values of p Let x = Then and so where f(x) = (1+x) p f() = 1 f (x) = p(1+x) p 1 f (x) = p(p 1)(1+x) p f (n) = p(p 1) (p n+1)(1+x) p n (1+x) p = n= f () = p f () = p(p 1) ( ) p = p(p 1) (p n+1) n n! are called the binomial coefficients We note that ( p ) = 1 A special case is p = 1, eg ( 1 1 )( ) (1+x) 1 = 1+ x+ 1 x +! = 1+ 1 x 1 8 x + f (n) () = p(p 1) (p n+1) ( ) p x n (7) n

15 SERIES EXPANSIONS 15 We use theorem 1 to test the convergence of this series From (7) the series has the form (1+x) p = n= a n with Hence a n = p(p 1) (p n+1) x n n! a n = p(p 1) (p n+)(p n+1)xn (1)() (n 1) a n 1 (1)() (n 1)n p(p 1) (p n+)x n 1 and so a n a n 1 = (p n+1)x n = p+1 n 1 x Clearly an a n 1 x as n, and so this series converges only if x < 1 Example 11 Find e x to an accuracy of ǫ We calculate here the series 1+x+ x xn + +! N!, where the Nth term is the first one for which xn < ǫ A numerical complication that arises is N! that 69! 1 1 and that n! cannot be evaluated using a pocket calculator for n 7 Instead we make use of the recursion relation (from example 9) ( x a = 1 a n = a n 1 n > n) We obtain e 1 = 7188 with 1 terms Note the required accuracy: ǫ = 1 6, therefore six decimal places are shown 43 Lagrange estimate of the error term Lagrange obtained an estimate of the error term R n in (5) which is very useful for analytical purposes The generalized mean-value theorem for integral calculus allows us to write the residual term as x R n = f (n+1) (x t) n (ξ x ) dt x n! where x < ξ x < x The integral in the residual term is now easily determined to be [ ] x 1 R n = f (n+1) (ξ x ) n! (x t)n+1 (x x)n+1 f (n+1) (ξ x ) (n+1) (n+1)! The series in (5) now becomes x = f(x) = f(x )+(x x )f (x )+ + (x x ) n n! It is often useful to define x = x +h Then (8) becomes f(x +h) = f(x )+hf (x )+ f (n) (x )+ (x x ) n+1 f (n+1) (ξ x ) (n+1)! + hn n! f(n) (x )+ hn+1 (n+1)! f(n+1) (ξ x ) (8) (9)

16 16 SERIES EXPANSIONS where x < ξ < x +h 44 Expansion of multivariable functions If a function is dependent on more than one variable, we use consecutive Taylor expansions with respect to each of the variables Since all other variables are held constant while expanding with respect to a particular variable, all derivatives in the expansion are partial Here, we expand a function of two variables We consider f(x +h,y +k) and use (9) to obtain an expansion with respect to x (holding y constant) and then we expand each term of this expansion with respect to y (holding x constant), ie f(x +h,y +k) = f(x,y +k)+ h f x + h f (x,y +k)! x + (x,y +k) = f(x,y )+ k f y + k f (x,y )! y + (x,y ) ( + h f ) y x + h f (x,y +k)! x + (x,y +k) = f(x,y )+ k f y + k f (x,y )! y + (x,y ) ( ) f + h x + k f (x,y ) y x + (x,y ) ( ) + h f! x + 3 f (x,y ) y x + (x,y ) ( = f(x,y )+ h f x + k f ) (x,y ) y (x,y ) ( ) h f +! x + hk f + k f y x! y + and in summary f(x +h,y +k) = (x,y ) n=m= h n k m n!m! (x,y ) n+m f x n y m The series for a function of more than two variables is analogous (x,y ) (x,y )

17 Chapter 3 NONLINEAR EQUATIONS 31 Introduction Generally speaking, nonlinear equations and, in particular, so-called transcendental equations, cannot be solved analytically, and so may only be solved numerically We consider the solution of nonlinear equations in one variable, although we briefly describe a method for a system of two nonlinear equations at the end of this chapter 3 The bisection method Consider the equation f(x) =, where f is assumed to be continuous By the intermediate value theorem, if an interval [x 1,x ] can be found on which f changes sign, ie f(x 1 )f(x ) <, then f has at least one real root on the interval Assume that f has only one root on [x 1,x ], denote this root as x The interval is now halved by determining x 3 = x 1 +x This value x 3 may be regarded as an approximation to x The approximation may be improved by iterating (repeating) the halving process At each iteration there are two possibilities: (a) f(x 1 )f(x 3 ) < The root thus lies on [x 1,x 3 ], and x is replaced by x 3 (b) f(x 3 )f(x ) < The root lies on [x 3,x ], and x 1 is replaced by x 3 The halving process is iterated until a specified accuracy ǫ in the function value is reached, that is f(x 3 ) < ǫ (31) Convergence Since any curve on a small enough interval may be approximated by a straight line, we have that f(x 3 ) will converge to f(x ), in the vicinity of x, at the same rate that converges to zero For the nth iteration we have η = x x 3 η n η n 1 The bisection method is thus said to be linearly convergent 17

18 18 3 NONLINEAR EQUATIONS Example 31 Solvef(x) = sinx 65x = toan accuracyof8decimalplaces in the function value We show the values for the first four iterations in the following table: iteration x 1 x 3 x f(x 1 ) f(x 3 ) f(x ) After 5 iterations we find x = rad ( ) 33 Linear interpolation Although the bisection method is reliable, it is also slow This is because very little information about f(x) is used indeed, only the sign of f is used In the linear interpolation method we also make use of the numerical values of f(x) Consider the equation f(x) = Let (x 1,y 1 ) and (x,y ) be two points on the curves y = f(x) inthevicinityoftherootx = x Weapproximatethecurveinthisregionbyastraight line through the two points The zero point x 3 of the straight line may be regarded as an approximation to the root x The value of x 3 is obtained from the equation for the straight line ( ) y y 1 y y 1 = (x x 1 ) x x 1 The point (x 3,) lies on this line so that ( ) y y 1 y 1 = (x 3 x 1 ) x x 1 and so x 3 = x 1y x y 1 y y 1 The interpolation process is now iterated by finding the straight line through (x,y ) and (x 3,y 3 ) and hence the approximation x 4, and so on In general, the (i 1)th iteration uses (x i 1,y i 1 ) and (x i,y i ) to find x i+1 : x i+1 = x i 1y i x i y i 1 (3) y i y i 1 For a given accuracy ǫ this process is repeated until the condition (31) is satisfied Convergence Let the errorin x after the (i 1)th iteration be denoted by η i In other words, x i = x +η i Then from equation (3) it follows that x +η i+1 = (x +η i 1 )y i (x +η i )y i 1 y i y i 1 and so η i+1 = η i 1f(x +η i ) η i f(x +η i 1 ) f(x +η i ) f(x +η i 1 ) Each function may be expanded in a Taylor series about x : f(x +η i ) = f(x )+η i f (x )+ 1 η i f (x )+ = +η i f (x )+ 1 η i f (x )+

19 3 NONLINEAR EQUATIONS 19 We assume that f (x ) and f (x ) Since η i is small (by assumption) we have, to the lowest order in η i η i+1 η [ i 1 ηi f (x )+ 1 η i f (x ) ] [ η i ηi 1 f (x )+ 1 η i 1 f (x ) ] η i f (x ) η i 1 f (x ) = 1 ( ηi η i 1 (η i η i 1 )f ) (x ) (η i η i 1 )f (x ) ( f ) (x ) = f η i η i 1 (x ) Aη i η i 1, where A is a constant (33) We attempt to satisfy this relationship by assuming η i = Kη a i 1 η i+1 = Kη a i where K is some constant, so that, from (33), we have η i+1 Aη i ( 1 K η i ) 1 a But η i+1 = Kη a i from our earlier assumption, and since the powers of η i must be the same on both sides of the equation, it follows that 1+ 1 a = a which gives a a 1 = which has the root a = which is the also called the golden mean number Hence η i+1 Kη 1618 i This rate of convergence is termed superlinear (The other root a 6 corresponds to divergence) Example 3 Solve f(x) = sinx 65x = accurate to 8 decimal places in the function value The values for the first 3 iterations are shown in the following table i x i x i+1 y i y i+1 x i+ y i After 6 iterations we obtain x = rad (compare with 5 iterations for the bisection method)

20 3 NONLINEAR EQUATIONS 34 Newton s method For this method we assume that f is at least twice differentiable We now use both the values f(x) and its first derivative to solve f(x) = Let x 1 be an initial estimate of the root x, and draw a tangent line at the point (x 1,f(x 1 )) to the curve of f The x-intercept of the tangent line, denote it x, is presumably a better estimate of x than x 1 was The slope of the tangent line is f (x 1 ) and is determined by f (x 1 ) = f(x ) f(x 1 ) x x 1 However, since x is the x-intercept of the tangent line, we have f(x ) = and thus Solving this last equation for x we obtain f (x 1 ) = f(x 1) x 1 x x = x 1 f(x 1) f (x 1 ) This process can be repeated and we find, in general, after the ith iteration we have x i+1 = x i f(x i) f (x i ) (34) Note: The above result could also have been obtained by calculating the error η for an initial estimate x 1 by solving f(x 1 +η) = We expand the LHS in a Taylor series about x = x 1 For small η, f(x 1 )+ηf (x 1 ) and so f(x 1 )+ηf (x 1 )+ η! f (x 1 )+ = η f(x 1) f (x 1 ) Newton s method is thus equivalent to a first-order Taylor expansion of the function f(x) Convergence Let the error in x after the ith iteration be denoted by η i In other words, x i = x +η i From equation (34) x +η i+1 = x +η i f(x +η i ) f (x +η i ) A Taylor expansion about x = x for both f(x +η i ) and f (x +η i ) gives Expanding the denominator using we find η i+1 = η i f(x )+η i f (x )+ 1 η i f (x )+ f (x )+η i f (x )+ 1 1+x = 1 x+x x f (x )+η i f (x )+O(ηi ) = 1 ( ) f f (x ) 1+η (x ) i f (x ) +O(η i ) = 1 f (x ) = 1 f (x ) 1 f 1+η (x ) i f (x ) +O(η i ( ) f (x ) 1 η i f (x ) +O(η i ) )

21 3 NONLINEAR EQUATIONS 1 To lowest order in η we have η i+1 η i 1 f (x ) ( f (x ) f (x ) ( f )( ) (x ) 1 η i f +η i f (x )+ η i (x ) f (x ) ) η i (35) The convergence is said to be quadratic It is important to note that (33) and (35) give estimates for the convergence rate only if convergence actually occurs It is possible that both the linear interpolation method and Newton s method may diverge Example 33 Find the root of f(x) = sinx 65x = correct to 8 decimal places in the function value Equation (34) is iterated using and f(x) = sinx 65x f (x) = cosx 65 with an initial estimate x = 15 The results of the process are shown in the following table: i x i f(x i ) f (x i ) x i = f(xi) f (x i) After 4 iterations the root x = rad is found Calculation hints (a) Sometimes f is a cumbersome analytical expression In such cases f may be estimated numerically by f (x i ) f(x i +h) f(x i ) h where h is small One should be aware of complications that may arise from numerical differentiation (see chapter 6 for better approximations Linear interpolation is indeed Newton s method with simple numerical differentiation) (b) When f(x) = has more than one root, it is sometimes found that both linear interpolation and Newton s method converge to the same root, for any initial estimate Say x is such a root Then we may write f(x) = (x x )g(x) Once the root x is found, it may be eliminated from the problem by subsequently solving 35 Fixed-point iteration If an equation can be written in the form g(x) = f(x) x x x = g(x) (36)

22 3 NONLINEAR EQUATIONS then a root of the equation may be regarded as a fixed point that is mapped to itself under the map x = g(x) The iteration process x i+1 = g(x i ) (37) may then be used, possibly, to find the root Example 34 Find the positive root of x = sinx correct to 8 decimal places The following table shows 6 iterations of (37) with g(x) = sinx, using initial value x = : i x i g(x i ) The required accuracy is actually achieved after 39 iterations and we find x = A transcendental equation of the form (36) can often be written in this form in several ways For example, e x = 3x may be written as x = ln3x or e x x = ± 3 Convergence We now obtain an analytical condition for convergence Consider the error in x after the ith iteration, x i+1 x = g(x i ) g(x ) where x is the root of (36) From the mean-value theorem of differential calculus we have g(x i ) g(x ) x i x = g (ξ i ) where x < ξ i < x i We can thus rewrite the ith error as Iterating this recursion relation i times gives Now let Then we have x i+1 x = g (ξ i )(x i x ) x i+1 x = g (ξ i )g(ξ i 1 ) g (ξ 1 )(x 1 x ) m = max( g (ξ i ) ) x i+1 x m i x 1 x For convergence we must have that x i+1 x as i, which will only be true if m < 1

23 3 NONLINEAR EQUATIONS 3 Condition for convergence: For the iteration scheme (37) to converge, we require in the neighbourhood of the root g (x) < 1 (38) Example 35 We investigate the convergence for the case in example 34 For x = g(x) = sinx we have g (x) = cosx Hence, for convergence we require cosx < 1 and so we obtain an interval of convergence 147 < x < 94 The root x = 1895 indeed lies on this interval Note: Condition (38) may be applied to any iterative method that can be written in the form (36) Newton s method may be written as x = g(x) with and so g (x) = 1 g(x) = x f(x) f (x) ( f (x) ) f(x)f (x) ( f (x) ) Then we must have in the neighbourhood of the root f(x)f (x) ( f (x) ) < 1 36 Systems of nonlinear equations We describe Newton s method for two simultaneous nonlinear equations The extension to systems of more than two equations can be found in most standard texts Consider two equations in unknowns x and y: f 1 (x,y) = f (x,y) = (39) As for the one-dimensional Newton s method, we try to find the corrections (η x,η y ) to an initial guess (x 1,y 1 ) such that (39) is satisfied: A first-order Taylor expansion yields f 1 (x 1,y 1 )+η x ( f (x 1,y 1 )+η x ( f 1 (x 1 +η x,y 1 +η y ) = f (x 1 +η x,y 1 +η y ) = f 1 x f x (x1,y 1) (x1,y 1) )+η y ( )+η y ( f 1 y f y (x1,y 1) (x1,y 1) ) )

24 4 3 NONLINEAR EQUATIONS which may be written as [ f1 x f x f 1 y f y ] (x 1,y 1) [ ηx ] η y [ ] f1 (x 1,y 1 ) f (x 1,y 1 ) so that ] η y [ ηx [ f1 x f x f 1 y f y ] (x 1,y 1) 1 [ ] f1 (x 1,y 1 ) f (x 1,y 1 ) The square matrix here is known as a Jacobian matrix, and is evaluated at the point (x 1,y 1 )

25 Chapter 4 SYSTEMS OF LINEAR EQUATIONS 41 Introduction In this chapter we consider the numerical solution of a system of linear equations The numerical method we describe is an iterative one, known as the Jacobi method 4 Solvability Consider m equations in n unknowns a 11 x 1 + a 1 x + + a 1n x n = b 1 a 1 x 1 + a x + + a n x n = b a m1 x 1 +a m x + +a mn x n = b m which have the matrix representation Ax = b (41) From linear algebra the following is known regarding the solution of (41): (a) Generally, there are no solutions if m > n (b) Generally, there are infinitely many solutions if m < n (c) If m = n then A is a square matrix and possibly solvable 43 Cramer s rule For m = n we define D = deta Cramer s rule differentiates between three cases: (a) If D = and b then, in general, there are no solutions (b) If b = and D then there is only the trivial solution x = Thus, a necessarycondition for nontrivial solutions, if b =, is that D = (c) The general case is when D and b From Cramer s rule it follows that if D j is the determinant obtained when the jth column of A is replaced by b, then the solution is given by x j = D j D j = 1,,,n (4) 5

26 6 4 SYSTEMS OF LINEAR EQUATIONS Determining (4) is equivalent to inverting the matrix equation (41), ie x = A 1 b In practice (4) is not used because of the large amount of computational effort required: it can be shown that (n 1)n! multiplications and n! 1 additions are needed, which is approximately nn! arithmetical operations in all 44 The Jacobi method If the diagonal elements a ii of A are all nonzero, then we may implement an iterative technique to solve (41)), known as the Jacobi method Let A = D+L+U where D is the matrix of diagonal entries of A, L is the lower triangular part of A, and U is the upper triangular part of A This gives so that Ax = (D+L+U)x = Dx+(L+U)x = b x = D 1( b (L+U)x ) (43) provided that a ii Note that the computation of D 1 is straightforward; since D is a diagonal matrix, its inverse is obtained simply by taking the reciprocal of its elements (this requires no more than n arithmetical operations) Equation (43) suggests the iteration scheme x (k) = D 1( b (L+U)x (k 1)), k =,1,, (44) where k denotes the iteration count Clearly, the implementation of this method requires an initial guess x () It is interesting to note that (44)) is a form of fixed-point iteration, as seen in the previous chapter, although here it is of a multivariable nature (those variables being the components x i of x) It can be shown that convergence to the exact solution x is guaranteed provided a ii > n a ij (45) j=1 i j for i = 1,,,n In other words, on each row, the magnitude of the diagonal element must exceed the sum of the magnitudes of all the other elements on that row, and this must hold for all rows in A An indication of the quality of the approximate solution x (k) may be determined by computing the residual r (k) = Ax (k) b andtheiterationprocessisstoppedwhenthemagnitudeofr (k) islessthansomeimposedtolerance Itcanbeshownthat thejacobimethodrequiresaboutn arithmeticaloperationsperiteration; in comparison with a computational implementation of Cramer s rule we have, as measure of relative efficiency, Mn n n! = Mn nn(n 1)! = M (n 1)! where M is the number of Jacobi iterations Clearly, if n is large we would expect the Jacobi method to be more efficient

27 4 SYSTEMS OF LINEAR EQUATIONS 7 Example 41 Consider A = = }{{} D }{{} U 6 1 }{{} L b = 3 7 Clearly, A satisfies (45) so that the Jacobi method may be used Starting with we find, after 4 iterations, x (4) = x () = r (4) = with r (4) = The true solution is x = and we see that the difference between each of these entries and those in x (4) is less than 1 11 We note that there are other types of iterative methods for solving linear systems, such as the Gauss-Seidel method and successive over-relaxtion (SOR), but they are similar in spirit to the Jacobi method, which is the simplest of the three Like the Jacobi method, these other methods also require that the diagonal entries of A must all be nonzero Furthermore, for large n, all these iterative methods become relatively more efficient (with respect to arithmetic computation) than direct inversion

28 Chapter 5 APPROXIMATION METHODS 51 Introduction In this chapter we investigate two related problems: the approximation of given functions by other, simpler functions, and the fitting of known functions to given data In the cases studied here, we approximate a continuous function by means of a polynomial 5 Polynomial interpolation Consider a function y(x) given in the form of the coordinates (x i,y i ), i =,1,,n We may approximate the function with a polynomial p n (x) of degree n that is exactly equal to y(x) at the n+1 given points y i = p n (x i ) = a +a 1 x i +a x i + +a nx n i (51) We note that there are n+1 coefficients a i as well as n+1 points (x i,y i ) The polynomial in (51) is known as an interpolating polynomial There are n+1 unknowns a i that must be determined from the n+1 linear equations a + a 1 x + +a n x n = p(x ) = y a + a 1 x 1 + +a n x n 1 = p(x 1) = y 1 a +a 1 x n + +a n x n n = p(x n ) = y n which can be written compactly in matrix form as 1 x x n a y 1 x 1 x n 1 a 1 = y 1 1 x x n a n y n Ifdet ( x i j),these equationshaveauniquesolution Inpracticeitisdifficult and/orinefficientto solve these equations directly, and so we investigate easier and/or faster methods Firstly, though, we state a few points of general importance We may rightly ask whether this interpolating polynomial is unique Assume that it is not, and thus there are two interpolating polynomials p n (x) = a +a 1 x+a x + +a n x n q n (x) = b +b 1 x+b x + +b n x n 8

29 5 APPROXIMATION METHODS 9 Then Q(x) = p n (x) q n (x) = (a b )+(a 1 b 1 )x+(a b )x + +(a n b n )x n is a polynomial of degree n But Q(x) = at the n+1 points {x,x 1,x,,x n } However, since Q(x) is of degree n, it may only have n roots This contradiction is resolved only if Q(x) =, which implies p n (x) = q n (x) We conclude that the interpolating polynomial is unique Error analysis We obtain an estimate of the error made when using the polynomial p n (x) instead of the function y(x) We know that y(x) = p n (x) at x i,i =,1,,,n Now consider F(x) = y(x) p n (x) C n (x x i ) (5) Clearly,F(x) = ateachofthenodes{x,x 1,,x n } Consideranypointx n+1 / {x,x 1,,x n } We choose C such that F(x n+1 ) =, which gives i= C = y(x n+1) p n (x n+1 ) n i= (x n+1 x i ) (53) Assume y(x), and hence F(x), is continuous From the constructions (5) and (53) it follows thatf(x) = atthen+nodes{x,x 1,,x n,x n+1 } Hence, bythegeneralizedrolle stheorem, there exists a point ξ such that F (n+1) (ξ) = If we differentiate (5) we obtain = F (n+1) (ξ) = y (n+1) (ξ) C(n+1)! and so which yields, using (53), C = y(n+1) (ξ) (n+1)!, y(x n+1 ) p n (x n+1 ) = y(n+1) (ξ) (n+1)! But x n+1 was chosen arbitrarily, so that this error expression y(x) p n (x) = y(n+1) (ξ(x)) (n+1)! n (x n+1 x i ) is true for all x It is easily verified that (54) is also true at {x,x 1,,x n } 53 Lagrange s method i= n (x x i ) (54) We now describe a method developed by Lagrange that is a shortcut for determining the coefficients a i in (51) For each of the points (x i,y i ), i =,1,,n, we construct an nth degree polynomial that is equal to zero at each of the other points x,x 1,,x i 1,x i+1,,x n i= L i (x) = A(x x )(x x 1 ) (x x i 1 )(x x i+1 ) (x x n ) (55) Furthermore, we demand that this polynomial has the value 1 when x = x i 1 = A(x i x )(x i x 1 ) (x i x i 1)(x i x i+1 ) (x i x n )

30 3 5 APPROXIMATION METHODS from which we obtain so that A = L i (x) = 1 (x i x )(x i x 1 ) (x i x i 1 )(x i x i+1 ) (x i x n ) (x x )(x x 1 ) (x x i 1 )(x x i+1 ) (x x n ) (x i x )(x i x 1 ) (x i x i 1)(x i x i+1 ) (x i x n ) The nth degree polynomial through the points (x i,y i ) is then given by n p n (x) = y i L i (x) (56) i= where the L i (x) are known as the Lagrange coefficient polynomials We may test if this is the required polynomial by substituting x = x k into equation (56) n p n (x k ) = y i L i (x k ) i= = y L (x k )+y 1 L 1 (x k )+ +y k L k (x k )+ +y n L n (x k ) = ++ +y k (1)+ + = y k Example 51 Approximate the sine function with a polynomial that is equal to sinx at x =, π 4, π Determine the true error at x = π 6 and compare it to an easily calculable upper bound of the error The coordinates of the three points that the interpolating polynomial must pass through are shown the following table: i x i y i = sinx i 1 π 4 1 π 1 We first determine the polynomials defined in (55) ( )( ) x π 4 x π L (x) = ( )( ) π 4 π = 8 ( π x 3π ) π x+ 4 8 L 1 (x) = (x )( ) x π ( π 4 )( π 4 ) π = 16 ( π x π ) x L (x) = (x )( ) x π ( 4 π )( π ) π = 8 ( π 4 x π ) 4 x The interpolating polynomial is given by p (x) = y L (x)+y 1 L 1 (x)+y L (x) = + 1 L 1 +(1)L = 8( 1 ) π x + ( ) 1 x π We determine the approximation error at x = π 6 ( π ( π sin p = = 17 6) 6)

31 5 APPROXIMATION METHODS 31 It is interesting to compare this error with the upper limit (54) ( π ( π ) max [,π/] cosx ( π ) sin p 6) 6 3! i= = 1 ( π )( π π )( π 4 6 π ) = 4 We see that using (54) does indeed give, in this case, an upper bound for the magnitude of the approximation error 54 Least-squares curve fitting A function y = y(x) may be approximated by an nth degree polynomial that passes through each data point, as in 5 and 53 However, if we believe that the relationship y(x) is in fact linear, then it would make more sense to find the best straight line that approximates the function We need a criterion that allows us to determine the fitting function such that its deviation from the given points is minimized The deviation i at point x i is the difference between the fit f(x) and the actual function y(x), ie i = f(x i ) y i We examine a few optimization criteria, given n data points Minimization of i i: Since errors may be both positive and negative, a positive error and negative error summed will give a sum that is less in magnitude than either error Furthermore, since 1+1 = + = 3+3 = = it is clear that the sum cannot give a unique minimum This means that such a fitting criterion cannot allow the fitting function to be uniquely determined Minimization of i i : When we have an error range (y,y + ), y < y + around the y coordinate of a data point (x,y), then the straight line fit f(x) = mx+c passing between (x,y ) and (x,y + ) such that y f(x) y + yields f(x) y + f(x) y + y + f(x)+f(x) y = y + y which is independent of m and c Again, the fitting function cannot be determined uniquely Minimization of i i: Let f(x) be a function fitting the data Suppose we have two data points with the same x coordinates but with different y coordinates, namely (x,y 1 ) and (x,y ) We have that S = i = (f(x) y ) +(f(x) y 1 ) = e +(e+d), i where e = f(x) y and d = y +y 1 Then it follows that ds = e+(e+d) = (e+d) de and d S de = 4 > Therefore S will be a minimum if e = d This means that the value of f(x) is the mean of the two y coordinates; a result which is both unique and intuitively acceptable and from mathematical statistics this norm is known to be the correct choice The function f(x) is therefore chosen so that n S = [f(x i ) y i ] (57) i= is a minimum We will see that, in the case of polynomial fitting, the norm (57) gives a unique result

32 3 5 APPROXIMATION METHODS 55 Least-squares polynomial fitting A wide variety of functions may be used in least-squares curve fitting We will leave the fitting of trigonometric, exponential and logarithmic functions etc to the student as self-study, and discuss only the use of polynomials as fitting functions We consider the case where f(x) in (57) is an mth degree polynomial p m (x), with m n Then p m (x) must be chosen such that n S = [p m (x i ) y i ] (58) is a minimum Since we must demand that From (58) we then have From (59) we have that and so Furthermore, S a k = i= p m (x) = a +a 1 x+ +a m x m (59) S a k = for k =,1,,m (51) n [p m (x i ) y i ] p m (x i ) a k i= S a k = S a k p m a k (x i ) = x k i for k =,1,,m n [p m (x i ) y i ]x k i for k =,1,,m (511) i= = n x k ix k i i= for k =,1,,m, so that the requirement in (51) does indeed give a minimum for S This value is obtained from (51) and (511): n n p m (x i )x k i = y i x k i k =,1,,m (51) i= i= The system in (51) consists of m + 1 equations in the m + 1 unknowns a k ; the a k are thus determined uniquely From (59) we obtain a more explicit form for (51): a (n+1)+a 1 x i + +a m x m i = y i (k = ) i i a x i +a 1 x i + +a m i a x m i +a 1 i i i x m+1 i + +a m i i i x m+1 i To fit a straight line, for example, we have m = 1 and and the coefficients a k are obtained from i x m i p 1 (x) = a +a 1 x i = i = i x i y i (k = 1) x m i y i (k = m) a (n+1)+a 1 x i = y i i i a x i +a 1 x i = (513) x i y i i

33 5 APPROXIMATION METHODS 33 Variance A question arises regarding the order of the fitting polynomial If we increase the degree of the polynomial to n (number of data points = n+1) then the fitting polynomial becomes an interpolating polynomial, which has zero deviation (since an interpolating polynomial passes through each point) How may we measure the quality of the fit for various degrees? Mathematical statistics tells us that we choose the degree for which the variation σ shows a minimum, where σ i = i n m Example 5 Fit a straight line to the data points in the following table: Using the data in the table, we find i x i y i n+1 = 6 x i = 1 i y i = 4 i x i = 91 i x i y i = Substitution into equation (513) gives the two simultaneous equations which are solved to give i 6a +1a 1 = 4 1a +91a 1 = a = 53 a 1 = 198 The fit is thus p 1 (x) = 198x+53 Example 53 In the following table the coordinates of six data points are given Fit polynomials of degrees 1 to 4 to this data, and decide which is the best fit x y The fitting polynomials, and variance for each, are given in the following table:

34 34 5 APPROXIMATION METHODS Degree Fitting polynomial σ 1 p 1 (x) = x 1 p (x) = 18 43x+116x 1 3 p 3 (x) = 15 3x+999x +3x p 4 (x) = x+114x 64x 3 +1x 4 6 It is clear that the variance for all those with degree of two or higher are of the same order of magnitude, and that the optimal fit is the one with degree two 56 Approximation with Chebyshev polynomials One disadvantage of polynomial approximation is related to the fact that polynomial maxima and minima are spread unevenly on any interval: On [ 1,1], x k has maxima at 1 and 1 and a minimum at We now seek related functions that have evenly spaced maxima and minima, and for which the maxima and minima over a given interval are as small as possible Possible candidates are, for example, the cosine functions: cosθ,cosθ,cos3θ, 561 Definition The Chebyshev polynomial of the nth degree is defined by T n (x) = cos(narccosx) (514) Here n is an integer and only the cases with n need to be studied, since it follows from the definition that T n = T n We note that T n (x) is defined on [ 1,1] only, due to the presence of the arccos function in (514) From the definition it follows T (x) = 1 and T 1 (x) = x Higher order Chebyshev polynomials are generated using a recursion formula, given by For the particular case n = 1 it follows that T m+n (x)+t m n (x) = T m (x)t n (x) T m+1 (x) = xt m (x) T m 1 (x) (515) and we find, for example, T (x) = xt 1 (x) T (x) = x 1 T 3 (x) = xt (x) T 1 (x) = 4x 3 3x (516) The relationships in (515) and (516) may be inverted to give powers of x in terms of T n : 1 = T x = T 1 x = 1 (T +1) = T +T (517) x 3 = 1 4 (T 3 +3x) = T 3 +3T 1 4