Numerical Analysis and Computing

Transcription

1 Numerical Analysis and Computing Lecture Notes #02 Calculus Review; Computer Artihmetic and Finite Precision; and Convergence; Joe Mahaffy, Department of Mathematics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA jmahaffy Spring 2010 Joe Mahaffy, Lecture Notes #02 (1/63)

2 Outline 1 Calculus Review Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem 2 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation 3, Pseudo-Code Fundamental Concepts 4 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** Joe Mahaffy, mahaffy@math.sdsu.edu Lecture Notes #02 (2/63)

3 Why Review Calculus??? Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem It s a good warm-up for our brains! When developing numerical schemes we will use theorems from calculus to guarantee that our algorithms make sense. If the theory is sound, when our programs fail we look for bugs in the code! Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (3/63)

4 Background Material A Crash Course in Calculus Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Key concepts from Calculus Limits Continuity Convergence Differentiability Rolle s Theorem Mean Value Theorem Extreme Value Theorem Intermediate Value Theorem Taylor s Theorem Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (4/63)

5 Limit / Continuity Calculus Review Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Definition (Limit) A function f defined on a set X of real numbers X R has the limit L at x 0, written lim x x 0 f (x) = L if given any real number ǫ > 0 ( ǫ > 0), there exists a real number δ > 0 ( δ > 0) such that f (x) L < ǫ, whenever x X and 0 < x x 0 < δ. Definition (Continuity (at a point)) Let f be a function defined on a set X of real numbers, and x 0 X. Then f is continuous at x 0 if lim f (x) = f (x 0 ). x x 0 Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (5/63)

6 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Example: Continuity at x Here we see how the limit x x 0 (where x 0 = 0.5) exists for the function f (x) = x sin(2πx). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (6/63)

7 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Example: Continuity at x Here we see how the limit x x 0 (where x 0 = 0.5) exists for the function f (x) = x sin(2πx). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (6/63)

8 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Example: Continuity at x Here we see how the limit x x 0 (where x 0 = 0.5) exists for the function f (x) = x sin(2πx). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (6/63)

9 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Example: Continuity at x Here we see how the limit x x 0 (where x 0 = 0.5) exists for the function f (x) = x sin(2πx). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (6/63)

10 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Example: Continuity at x Here we see how the limit x x 0 (where x 0 = 0.5) exists for the function f (x) = x sin(2πx). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (6/63)

11 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Example: Continuity at x Here we see how the limit x x 0 (where x 0 = 0.5) exists for the function f (x) = x sin(2πx). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (6/63)

12 Examples: Jump Discontinuity Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem The function { x + 1 f (x) = 2 sin(2πx) x < 0.5 x sin(2πx) + 1 x > 0.5 has a jump discontinuity at x 0 = 0.5. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (7/63)

13 Examples: Spike Discontinuity Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem The function { 1 x = 0.5 f (x) = 0 x 0.5 The limit exists, but lim f (x) = 0 1 x 0.5 has a discontinuity at x 0 = 0.5. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (8/63)

14 Continuity / Convergence Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Definition (Continuity (in an interval)) The function f is continuous on the set X (f C(X)) if it is continuous at each point x in X. Definition (Convergence of a sequence) Let x = {x n } n=1 be an infinite sequence of real (or complex numbers). The sequence x converges to x (has the limit x) if ǫ > 0, N(ǫ) Z + : x n x < ǫ n > N(ǫ). The notation lim x n = x n means that the sequence {x n } n=1 converges to x. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (9/63)

15 Illustration: Convergence of a Complex Sequence Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem N 2 k>=n N 1 A sequence in z = {z k } k=1 converges to z 0 C (the black dot) if for any ǫ (the radius of the circle), there is a value N (which depends on ǫ) so that the tail of the sequence z t = {z k } k=n is inside the circle. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (10/63)

16 Differentiability Calculus Review Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Theorem If f is a function defined on a set X of real numbers and x 0 X, the the following statements are equivalent: (a) f is continuous at x 0 (b) If {x n} n=1 is any sequence in X converging to x 0, then lim n f (x n) = f (x 0 ). Definition (Differentiability (at a point)) Let f be a function defined on an open interval containing x 0 (a < x 0 < b). f is differentiable at x 0 if f (x 0 ) = lim x x 0 f (x) f (x 0 ) x x 0 exists. If the limit exists, f (x 0 ) is the derivative at x 0. Definition (Differentiability (in an interval)) If f (x 0 ) exists x 0 X, then f is differentiable on X. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (11/63)

17 Illustration: Differentiability Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Here we see that the limit f (x) f (x 0 ) lim x x 0 x x 0 exists and approaches the slope / derivative at x 0, f (x 0 ). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (12/63)

18 Illustration: Differentiability Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Here we see that the limit f (x) f (x 0 ) lim x x 0 x x 0 exists and approaches the slope / derivative at x 0, f (x 0 ). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (12/63)

19 Illustration: Differentiability Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Here we see that the limit f (x) f (x 0 ) lim x x 0 x x 0 exists and approaches the slope / derivative at x 0, f (x 0 ). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (12/63)

20 Illustration: Differentiability Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Here we see that the limit f (x) f (x 0 ) lim x x 0 x x 0 exists and approaches the slope / derivative at x 0, f (x 0 ). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (12/63)

21 Continuity / Rolle s Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Theorem (Differentiability Continuity) If f is differentiable at x 0, then f is continuous at x 0. Theorem (Rolle s Theorem Wiki-Link ) Suppose f C[a, b] and that f is differentiable on (a, b). If f (a) = f (b), then c (a,b): f (c) = 0. f (c)=0 a c b Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (13/63)

22 Mean Value Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Theorem (Mean Value Theorem Wiki-Link ) If f C[a, b] and f is differentiable on (a, b), then c (a, b): f f (b) f (a) (c) =. b a f (c)=[f(b) f(a)] / [b a] a c b Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (14/63)

23 Extreme Value Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Theorem (Extreme Value Theorem Wiki-Link ) If f C[a, b] then c 1, c 2 [a, b]: f (c 1 ) f (x) f (c 2 ) x [a, b]. If f is differentiable on (a, b) then the numbers c 1, c 2 occur either at the endpoints of [a, b] or where f (x) = 0. c2 = b a < c2 < b c1 = a a b a a < c1 < b b Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (15/63)

24 Intermediate Value Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Theorem (Intermediate Value Theorem Wiki-Link ) if f C[a, b] and K is any number between f (a) and f (b), then there exists a number c in (a, b) for which f (c) = K. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (16/63)

25 Taylor s Theorem Calculus Review Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Theorem (Taylor s Theorem Wiki-Link ) Suppose f C n [a, b], f (n+1) on [a, b], and x 0 [a, b]. Then x (a, b), ξ(x) (x 0, x) with f (x) = P n (x) + R n (x) where P n (x) = n k=0 f (k) (x 0 ) (x x 0 ) k, k! R n (x) = f (n+1) (ξ(x)) (x x 0 ) (n+1). (n + 1)! P n (x) is called the Taylor polynomial of degree n, and R n (x) is the remainder term (truncation error). This theorem is extremely important for numerical analysis; Taylor expansion is a fundamental step in the derivation of many of the algorithms we see in this class (and in Math693ab). Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (17/63)

26 Illustration: Taylor s Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem f (x) = sin(x) sin(x) t1(x) P 1 (x) = x Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (18/63)

27 Illustration: Taylor s Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem f (x) = sin(x) sin(x) t5(x) P 5 (x) = x 1 3! x ! x5 Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (18/63)

28 Illustration: Taylor s Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem f (x) = sin(x) sin(x) t9(x) P 9 (x) = x 1 3! x ! x5 1 7! x ! x9 Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (18/63)

29 Illustration: Taylor s Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem f (x) = sin(x) sin(x) t13(x) P 13 (x) = x 1 3! x ! x5 1 7! x ! x9 1 11! x ! x13 Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (18/63)

30 Illustration: Taylor s Theorem Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem f (x) = sin(x) sin(x) t5(x) 30 sin(x) t9(x) 30 sin(x) t13(x) P 5 (x) P 9 (x) P 13 (x) P 13 (x) = x 1 3! x ! x5 1 }{{} 7! x ! x9 P 5 (x) } {{ } P 9 (x) 1 11! x ! x13 Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (18/63)

31 Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Taylor Expansions Maple A Taylor polynomial of degree n requires all derivatives up to order n and degree n + 1 for the Remainder. In general, derivatives may be complicated expressions. Maple computes derivatives accurately and efficiently differentiation uses the command diff(f(x), x); Maple has a routine for Taylor series expansions finding the Taylor s series uses the command taylor(f(x), x=x0, n);, meaning the Taylor series expansion about x = x 0 using n terms in the expansion. A Maple worksheet is available with many of these basic commands through my webpage for this class. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (19/63)

32 Taylor Expansions Matlab Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem A Taylor polynomial of degree n requires all derivatives up to order n, and order n + 1 for the remainder. Derivatives may be [more] complicated expression [than the original function]. Matlab can compute derivatives for you: Matlab: Symbolic Computations >> syms x >> diff(sin(2*x)) >> diff(sin(2*x),3) >> taylor(exp(x),5) >> taylor(exp(x),5,1) Try this!!! Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (20/63)

33 Taylor s Theorem: Computer Programming MatLab Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem Most versions of MatLab have a symbolic package that includes Maple, so this symbolic package can help with derivatives. Often easier to play to the strengths of each language and let Maple find the Taylor coefficients to employ in the MatLab code. MatLab provides relatively efficient numerical programs that are similar and based on C Programming. A MatLab code is provided to show the convergence of the Taylor series to the cosine function with increasing numbers of terms. This is shown on the Maple worksheet also, and the code is accessible through my webpage. Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (21/63)

34 Taylor s Approximation for Cosine Function Limits, Continuity, and Convergence Differentiability, Rolle s, and the Mean Value Theorem Extreme Value, Intermediate Value, and Taylor s Theorem 4 Taylor Series Approx to Cosine 2 0 y axis x axis Joe Mahaffy, mahaffy@math.sdsu.edu Calculus Review (22/63)

35 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Computer Arithmetic and Finite Precision Computer Arithmetic and Finite Precision Joe Mahaffy, mahaffy@math.sdsu.edu (23/63)

36 Finite Precision Calculus Review Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation A single char Computers use a finite number of bits (0 s and 1 s) to represent numbers. For instance, an 8-bit unsigned integer (a.k.a a char ) is stored: Here, = = 77, which represents the upper-case character M (US-ASCII). Joe Mahaffy, mahaffy@math.sdsu.edu (24/63)

37 Finite Precision Calculus Review Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation A 64-bit real number, double The Binary Floating Point Arithmetic Standard (IEEE The Institute for Electrical and Electronics Engineers) standard specified the following layout for a 64-bit real number: Where sc 10 c 9... c 1 c 0 m 51 m m 1 m 0 Symbol Bits Description s 1 The sign bit 0=positive, 1=negative c 11 The characteristic (exponent) m 52 The mantissa r = ( 1) s 2 c 1023 (1 + m), c = 10 k=0 c k 2 k, m = 51 k=0 m k 2 52 k Joe Mahaffy, mahaffy@math.sdsu.edu (25/63)

38 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Burden-Faires Description is not complete... As described in previous slide, we cannot represent zero! There are some special signals in IEEE : Type S (1 bit) C (11 bits) M (52 bits) signaling NaN u 2047 (max).0uuuuu u (with at least one 1 bit) quiet NaN u 2047 (max).1uuuuu u negative infinity (max) positive infinity (max) negative zero positive zero From: Joe Mahaffy, mahaffy@math.sdsu.edu (26/63)

39 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Examples: Finite Precision 10 r = ( 1) s 2 c 1023 (1 + f ), c = c k 2 k, m = Example #1: 3.0 k=0 51 k=0 m k 2 52 k ( r 1 = ( 1) ) = = 3.0 Example #2: The Smallest Positive Real Number r 2 = ( 1) ( ) = ( ) Joe Mahaffy, mahaffy@math.sdsu.edu (27/63)

40 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Examples: Finite Precision 10 r = ( 1) s 2 c 1023 (1 + f ), c = c k 2 k, m = k=0 Example #3: The Largest Positive Real Number 51 k=0 m k 2 52 k ( r 3 = ( 1) ) 2 52 ( = ) Joe Mahaffy, mahaffy@math.sdsu.edu (28/63)

41 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Something is Missing Gaps in the Representation 1 of 3 There are gaps in the floating-point representation! Given the representation for the value The next larger floating-point value is i.e. the value The difference between these two values is = ( ) Any number in the interval 2 52, is not representable! Joe Mahaffy, mahaffy@math.sdsu.edu (29/63)

42 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Something is Missing Gaps in the Representation 2 of 3 A gap of doesn t seem too bad... However, the size of the gap depend on the value itself... Consider r = and the next value The difference is Joe Mahaffy, mahaffy@math.sdsu.edu (30/63)

43 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Something is Missing Gaps in the Representation 3 of 3 At the other extreme, the difference between and the previous value is = That s a fairly significant gap!!! The number of atoms in the observable universe can be estimated to be no more than Joe Mahaffy, mahaffy@math.sdsu.edu (31/63)

44 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation The Relative Gap It makes more sense to factor the exponent out of the discussion and talk about the relative gap: Exponent Gap Relative Gap (Gap/Exponent) Any difference between numbers smaller than the local gap is not representable, e.g. any number in the interval [ 3.0, ) 2 51 is represented by the value 3.0. Joe Mahaffy, mahaffy@math.sdsu.edu (32/63)

45 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation The Floating Point Theorem Theorem Floating point numbers represent intervals! Since (most) humans find it hard to think in binary representation, from now on we will for simplicity and without loss of generality assume that floating point numbers are represented in the normalized floating point form as... k-digit decimal machine numbers ±0.d 1 d 2 d k 1 d k 10 n where 1 d 1 9, 0 d i 9, i 2, n Z Joe Mahaffy, mahaffy@math.sdsu.edu (33/63)

46 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation k-digit Decimal Machine Numbers Any real number can be written in the form ±0.d 1 d 2 d 10 n given infinite patience and storage space. We can obtain the floating-point representation fl(r) in two ways: (1) Truncating (chopping) just keep the first k digits. (2) Rounding if d k+1 5 then add 1 to d k. Truncate. Examples fl t,5 (π) = , fl r,5 (π) = In both cases, the error introduced is called the roundoff error. Joe Mahaffy, mahaffy@math.sdsu.edu (34/63)

47 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Quantifying the Error Let p be an approximation to p, then... Definition (The Absolute Error) p p Definition (The Relative Error) p p, p 0 p Definition (Significant Digits) The number of significant digits is the largest value of t for which p p p < 5 10 t Joe Mahaffy, mahaffy@math.sdsu.edu (35/63)

48 Sources of Numerical Error Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Important!!! 1) Representation Roundoff. 2) Cancellation Consider: = this value has (at most) 1 significant digit!!! If you assume a canceled value has more significant bits (the computer will happily give you some numbers) I don t want you programming the autopilot for any airlines!!! Joe Mahaffy, mahaffy@math.sdsu.edu (36/63)

49 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Examples: 5-digit Arithmetic Rounding 5-digit arithmetic Truncating 5-digit arithmetic ( ) = ( ) = = ( ) = ( ) = = Rearrangement changes the result: ( ) = = Numerically, order of computation matters! (This is a HARD problem) Joe Mahaffy, mahaffy@math.sdsu.edu (37/63)

50 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Example: Loss of Significant Digits due to Subtractive Cancellation Consider the recursive relation x n+1 = 1 (n + 1)x n with x 0 = 1 1 e This sequence can be shown to converge to 0 (in 2 slides). Subtractive cancellation produces an error which is approximately equal to the machine precision times n!. The MatLab code for this example is provided on the webpage. Maple has a routine rsolve that solves this recursive relation exactly, using the Gamma function. Joe Mahaffy, mahaffy@math.sdsu.edu (38/63)

51 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Subtractive Cancellation Example: Output n x n n! n x n n! e e e e e e e e e e e e e e+006 Joe Mahaffy, mahaffy@math.sdsu.edu (39/63)

52 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Example: Proof of Convergence to 0 The recursive relation is with From the recursive relation This shows that x n+1 = 1 (n + 1)x n x 0 = 1 1 e = 1 1 2! + 1 3! 1 4! +... x 1 = 1 x 0 = 1 2! 1 3! + 1 4!... x 2 = 1 2x 1 = ! + 2 5!... x 3 = 1 3x 2 = 3! 4! 3! 5! + 3! 6!.... x n = 1 nx n 1 = n! (n + 1)! n! (n + 2)! + n! (n + 3)!... x n = 1 n as n. (n + 1)(n + 2) Joe Mahaffy, mahaffy@math.sdsu.edu (40/63)

53 Binary Representation, IEEE Something s Missing... Roundoff and Truncation, Errors, Digits Cancellation Example: Loss of Significant Digits Matlab code Matlab code: Loss of Significant Digits clear x(1) = 1-1/exp(1); s(1) = 1; f(1) = 1; for i = 2:21 x(i) = 1-(i-1)*x(i-1); s(i) = 1/i; f(i) = (i-1)*f(i-1); end n = 0:20; z = [n; x; s; f]; fprintf(1, \n\n n x(n) 1/(n+1) n!\n\n ) fprintf(1, %2.0f %13.8f %10.8f %10.3g\n,z) Joe Mahaffy, mahaffy@math.sdsu.edu (41/63)

54 Calculus Review, Pseudo-Code Fundamental Concepts Definition (Algorithm) An algorithm is a procedure that describes, in an unambiguous manner, a finite sequence of steps to be performed in a specific order. In this class, the objective of an algorithm is to implement a procedure to solve a problem or approximate a solution to a problem. Most homes have a collection of algorithms in printed form we tend to call them recipes. There is a collection of algorithms out there called Numerical Recipes, google for it! Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (42/63)

55 Pseudo-code Calculus Review, Pseudo-Code Fundamental Concepts Definition (Pseudo-code) Pseudo-code is an algorithm description which specifies the input/output formats. Note that pseudo-code is not computer language specific, but should be easily translatable to any procedural computer language. Examples of Pseudo-code statements: for i = 1,2,...,n Set x i = a i + i h While i < N do Steps If... then... else Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (43/63)

56 Key Concepts for Numerical, Pseudo-Code Fundamental Concepts Stability Definition (Stability) An algorithm is said to be stable if small changes in the input, generates small changes in the output. At some point we need to quantify what small means! If an algorithm is stable for a certain range of initial data, then is it said to be conditionally stable. Stability issues are discussed in great detail in Math 543. Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (44/63)

57 , Pseudo-Code Fundamental Concepts Key Concepts for Numerical Error Growth Suppose E 0 > 0 denotes the initial error, and E n represents the error after n operations. If E n CE 0 n (for a constant C which is independent of n), then the growth is linear. If E n C n E 0, C > 1, then the growth is exponential in this case the error will dominate very fast (undesirable scenario). Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (45/63)

58 , Pseudo-Code Fundamental Concepts Key Concepts for Numerical Error Growth Suppose E 0 > 0 denotes the initial error, and E n represents the error after n operations. If E n CE 0 n (for a constant C which is independent of n), then the growth is linear. If E n C n E 0, C > 1, then the growth is exponential in this case the error will dominate very fast (undesirable scenario). Linear error growth is usually unavoidable, and in the case where C and E 0 are small the results are generally acceptable. Stable algorithm. Exponential error growth is unacceptable. Regardless of the size of E 0 the error grows rapidly. Unstable algorithm. Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (45/63)

59 , Pseudo-Code Fundamental Concepts Example BF of 2 The recursive equation p n = 10 3 p n 1 p n 2, n = 2, 3,..., has the exact solution p n = c 1 ( 1 3 ) n + c 2 3 n for any constants c 1 and c 2. (Determined by starting values.) In particular, if p 0 = 1 and p 1 = 1 3, we get c 1 = 1 and c 2 = 0, so p n = ( 1 3) n for all n. Now, consider what happens in 5-digit rounding arithmetic... Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (46/63)

60 , Pseudo-Code Fundamental Concepts Example BF of 2 Now, consider what happens in 5-digit rounding arithmetic... which modifies p 0 = , p 1 = c 1 = , c 2 = The generated sequence is p n = ( ) n (3.0000) n }{{} Exponential Growth p n quickly becomes a very poor approximation to p n due to the exponential growth of the initial roundoff error. Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (47/63)

61 Reducing the Effects of Roundoff Error, Pseudo-Code Fundamental Concepts The effects of roundoff error can be reduced by using higher-order-digit arithmetic such as the double or multiple-precision arithmetic available on most computers. Disadvantages in using double precision arithmetic are that it takes more computation time and the growth of the roundoff error is not eliminated but only postponed. Sometimes, but not always, it is possible to reduce the growth of the roundoff error by restructuring the calculations. Joe Mahaffy, and Convergence (48/63)

62 Key Concepts Calculus Review, Pseudo-Code Fundamental Concepts Rate of Convergence Definition (Rate of Convergence) Suppose the sequence β = {β n } n=1 converges to zero, and α = {α n} n=1 converges to a number α. If K > 0: α n α < Kβ n, for n large enough, then we say that {α n } n=1 converges to α with a Rate of Convergence O(β n ) ( Big Oh of β n ). We write α n = α + O(β n ) Note: The sequence β = {β n } n=1 is usually chosen to be for some positive value of p. β n = 1 n p Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (49/63)

63 Examples: Rate of Convergence, Pseudo-Code Fundamental Concepts Example #1: If α n = α + 1 n then for any ǫ > 0 α n α = 1 (1 + ǫ) n }{{} K 1 n }{{} β n hence ( ) 1 α n = α + O n Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (50/63)

64 Examples: Rate of Convergence, Pseudo-Code Fundamental Concepts Example #2: Consider the sequence (as n ) ( ) 1 α n = sin 1 n n We Taylor expand sin(x) about x 0 = 0: ( ) 1 sin 1 n n 1 ( ) 1 6n 3 + O n 5 Hence ( ) α n = 1 1 6n 3 + O n 5 It follows that ( ) 1 α n = 0 + O n 3 Note: ( ) ( ) ( ) O n 3 + O n 5 = O n 3, since n 5 1 n 3, as n Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (51/63)

65 Generalizing to Continuous Limits, Pseudo-Code Fundamental Concepts Definition (Rate of Convergence) Suppose If K > 0: h < H (for some H > 0), then lim G(h) = 0, and lim F(h) = L hց0 hց0 F(h) L K G(h) F(h) = L + O(G(h)) we say that F(h) converges to L with a Rate of Convergence O(G(h)). Usually G(h) = h p, p > 0. Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (52/63)

66 Examples: Rate of Convergence, Pseudo-Code Fundamental Concepts Example #2-b: Consider the function α(h) (as h 0) α(h) = sin(h) h We Taylor expand sin(x) about x 0 = 0: Hence It follows that Note: sin(h) h h3 6 + O ( h 5) α(h) = h O ( h 5) lim α(h) = 0 + O ( h 3) h 0 O ( h 3) + O ( h 5) = O ( h 3), since h 5 h 3, as h 0 Joe Mahaffy, mahaffy@math.sdsu.edu and Convergence (53/63)

67 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** Our new favorite problem: f (x) = 0 Joe Mahaffy, mahaffy@math.sdsu.edu (54/63)

68 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** Introduction We are going to solve the equation f (x) = 0 (i.e. finding root to the equation), for functions f that are complicated enough that there is no closed form solution (and/or we are too lazy to find it?) In a lot of cases we will solve problems to which we can find the closed form solutions we do this as a training ground and to evaluate how good our numerical methods are. Joe Mahaffy, mahaffy@math.sdsu.edu (55/63)

69 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method 1 of 4 Suppose f is continuous on the interval (a 0, b 0 ) and f (a 0 ) f (b 0 ) < 0 This means the function changes sign at least once in the interval. The intermediate value theorem guarantees the existence of c (a 0, b 0 ) such that f (c) = 0. Without loss of generality (just consider the function f (x)), we can assume (for now) that f (a 0 ) < 0. We will construct a sequence of intervals containing the root c: (a 0, b 0 ) (a 1, b 1 ) (a n 1, b n 1 ) (a n, b n ) c Joe Mahaffy, mahaffy@math.sdsu.edu (56/63)

70 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method 2 of 4 The sub-intervals are determined recursively: Given (a k 1, b k 1 ), let m k = a k 1 + b k 1 2 If f (m k ) = 0, we re done, otherwise be the mid-point. (a k, b k ) = { (mk, b k 1 ) if f (m k ) < 0 (a k 1, m k ) if f (m k ) > 0 This construction guarantees that f (a k ) f (b k ) < 0 and c (a k, b k ). Joe Mahaffy, mahaffy@math.sdsu.edu (57/63)

71 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method 3 of 4 After n steps, the interval (a n, b n ) has the length we can take b n a n = ( ) 1 n b 0 a 0 2 m n+1 = a n + b n 2 as the estimate for the root c and we have c = m n+1 ± d n, d n = ( ) 1 n+1 b 0 a 0 2 Joe Mahaffy, mahaffy@math.sdsu.edu (58/63)

72 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method 4 of 4 Convergence is slow: At each step we gain one binary digit in accuracy. Since , it takes on average 3.3 iterations to gain one decimal digit of accuracy. Note: The rate of convergence is completely independent of the function f. We are only using the sign of f at the endpoints of the interval(s) to make decisions on how to update. By making more effective use of the values of f we can attain significantly faster convergence. First an example... Joe Mahaffy, mahaffy@math.sdsu.edu (59/63)

73 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 1 of 2 The bisection method applied to ( x ) 2 f (x) = sin(x) = 0 2 with (a 0, b 0 ) = (1.5, 2.0), and (f (a 0 ), f (b 0 )) = ( , ) gives: k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (60/63)

74 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

75 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

76 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

77 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

78 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

79 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

80 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

81 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

82 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of k a k b k m k+1 f (m k+1 ) Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

83 Calculus Review f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** The Bisection Method Example, 2 of 2 Joe Mahaffy, mahaffy@math.sdsu.edu (61/63)

84 The Bisection Method f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** Matlab code Matlab code: The Bisection Method % WARNING: This example ASSUMES that f(a)<0<f(b)... x = 1.5:0.001:2; f = inline( (x/2).ˆ2-sin(x), x ); a = 1.5; b = 2.0; for k = 0:9 plot(x,f(x), k-, linewidth,2) axis([ ]) grid on hold on plot([a b],f([a b]), ko, linewidth,5) hold off m = (a+b)/2; if( f(m) < 0 ) a = m; else b = m; end pause print( -depsc,[ bisec int2str(k).eps ], -f1 ); end Joe Mahaffy, mahaffy@math.sdsu.edu (62/63)

85 f (x) = 0, Root Finding The Bisection Method When do we stop?! *** Homework #1 *** Stopping Criteria When do we stop? We can (1) keep going until successive iterates are close: or (2) close in relative terms m k+1 m k < ǫ m k+1 m k m k+1 or (3) the function value is small enough < ǫ f (m k+1 ) < ǫ No choice is perfect. In general, where no additional information about f is known, the second criterion is the preferred one (since it comes the closest to testing the relative error). Joe Mahaffy, mahaffy@math.sdsu.edu (63/63)