1 ERROR ANALYSIS IN COMPUTATION

Transcription

1 1 ERROR ANALYSIS IN COMPUTATION 1.2 Round-Off Errors & Computer Arithmetic (a) Computer Representation of Numbers Two types: integer mode (not used in MATLAB) floating-point mode x R ˆx F(β, t, l, u), machine numbers β =base or radix (positive integer greater than 1) t =precision (positive integer) x = σm β e σ = ± : sign 1 m = mantissa, β m < 1 e = exponent or characteristic, integer l e u m = (0.a 1 a 2... a t ) β = a 1 β + a 2 β a t (base-β representation) 2 βt 0 a i β 1, i = 1,..., t called digits or, for β = 2, bits a 1 1 for a normalized floating-point number, which gives a unique representation with no digits wasted on leading zeros. Furthermore, leading bit must = 1 in binary (β = 2) and need not be stored either. Common examples of number bases in computer representations binary or base-2: digits (bits) 0,1 decimal or base-10: digits 0,1,...,9 hexadecimal or base-16: digits 0,1,...,9,a,...,f Binary is now used almost exclusively by all modern computers for machine computations and storage, but decimal and hexadecimal are used to interface with human users! 1

2 IEEE Standard 754 Floating Point Numbers: See Hollasch article online, or NCM, section 1.7 β = 2 ˆx = σm 2 E, M = 2m, E = e 1, L = l 1, U = u 1 1 M < 2, L E U M = 1 + F, 0 F < 1, F = fraction M = (1.a 1 a p ) 2, t = p + 1 (Warning! Heath s p is our t...) MATLAB: [m,e]=log2(x) E=e-1, F=2m-1 IEEE SINGLE: p = 23, 126 E 127 (r = 8) IEEE DOUBLE: p = 52, 1022 E 1023 (r = 11) sign exponents fraction SINGLE 1 bit + 8 bits + 23 bits = 32 bits DOUBLE 1 bit + 11 bits + 52 bits = 64 bits Note E + B = 0, 1, 2,..., 2 r 1, r = range of exponent (r =8 or 11) = (b r 1 b r 2 b 0 ) 2 exponent field B = 2 r 1 1 = bias Stored as (b r b r 1 b 0 ) 2, with b r = 1 for σ = 1, b r = 0 for σ = 1 However, (2 r 1 1) E 2 r 1 E = (2 r 1 1), b r 1 = = b 0 = 0 reserved to represent zero (or denormalized numbers more generally), and E = 2 r 1, b r 1 = = b 0 = 1 reserved to represent INF (and NAN). 2

3 Machine Numbers Total of 2(β 1)β p (U L+1)+1 normalized floating-point (f.p.) numbers, including 0 as an honorary member! machine epsilon (eps) =β p = distance from 1 to next larger f.p. number underflow level (UFL) =β L = smallest positive f.p. number overflow level (OFL) =(β β p )β U = largest f.p. number MATLAB: eps, realmin, realmax See program floatgui.m from NCM Underflow & Overflow x >OFL = overflow error fl(x)=inf (F = 0, E = 2 r 1 = 1024 in MATLAB) e.g. 1/0=INF Note that indeterminant operations such as 0/0, 0*INF, INF-INF, INF/INF instead give fl(x)=nan (F 0, E = 2 r 1 ) x <UFL = underflow error Usually f l(x)=0 but may be represented by denormalized floating-point numbers of the form σ(0.a 1 a p ) 2 2 L which in IEEE Standard are assigned E = (2 r 1 1) or b r 1 = b 0 = 0. zero is a special case with all a 1 = = a p = 0! Rounding Rules Assuming that then x R ˆx = fl(x) F x = σ (0.a 1 a 2... a t a t+1 ) β β e 3

4 chopping fl(x) = σ (0.a 1 a 2... a t ) β β e σ (0.a 1 a 2... a t ) β β e 0 a t+1 < β 2 rounding fl(x) = σ (0.a 1 a 2... a t ) β β e +( ) β β e β a 2 t+1 < β In case of tie, i.e. a t+1 = 1 2 β, a j = 0 for j t + 2, then round up if a t is odd (round-to-even rule). MATLAB function single rounds double-precision to single-precision (and double adds zero digits to convert single to double) Accuracy of Floating-Point Representation Absolute Error: δ(x) Err(x) = fl(x) x fl(x) = x + δ(x) Relative error: ɛ(x) Rel(x) = Err(x) x fl(x) = [1 + ɛ(x)] x. chopped fl(x) : β (t 1) ɛ(x) 0 rounded fl(x) : 1 2 β (t 1) ɛ(x) 1 2 β (t 1) measures of accuracy: { β (t 1) chopping machine epsilon=max x ɛ(x) = 1 2 β (t 1) rounding (Not the same as MATLAB eps!) The unit round is the smallest machine number δ > 0 such that fl(1 + δ) > 1. { β (t 1) chopped definition of fl(x) δ = 1 2 β (t 1) rounded definition of fl(x) maximum f.p. integer M : fl(m) = m if 0 m M, fl(m + 1) M + 1 M = β t 4

5 x L = smallest (normalized) positive floating point number = ( ) β β l = β l 1 x U = largest positive floating point number = (0. β β β) β β u = (1 β t )β u for β = β 1 If machine operation produces x such that x > x U = overflow (unless using denormalized f.p.) 0 < x < x L = underflow significant figures: Approximation ˆx has m significant digits with respect to x if Rel = ˆx x /x has magnitude less than or equal to 5 in the m th digit of x, counting to the right from the first digit after the decimal point in Rel. example: ˆπ = π = Err = Rel = , 2 5 but 23 > 5 = ˆπ has 6 significant figures (b) Sources of Error Before computation mathematical modelling of physical phenomena empirical measurements (systematic or random) inputs from previous computations During computation machine error or rounding mathematical truncation or discretization error program bugs/mistakes 5

6 example: compute volume of Earth using V = 4 3 πr3 Earth modelled as sphere (really more oblate spheroid) Value for radius R based on measurements with uncertainty Value for irrational π requires truncation (e.g. π = ) Value for input data & arithmetic operations rounded on computer Computational Error=Truncation Error+Rounding Error (ignoring Bugs!!!) truncation error= rounding error= error made in approximate numerical algorithm using exact arithmetic error made in algorithm because of the use of limited precision arithmetic example: Error in finite-difference approximation to the derivative f (x) D h f(x) f(x + h) f(x) h By Taylor s theorem with remainder, f(x+h) = f(x)+f (x)h+ 1 2 f (ξ)h 2 for some ξ between x, x + h. Thus D h f(x) f (x) = 1 2 f (ξ)h = truncation error 1 2 Mh for M = max f (x) x Suppose that f(x) f(x) ε for all x. Then using D h f(x) = ( ) ( ) D f(x + h) f(x + h) f(x) f(x) h f(x) D h f(x) = h h f(x+h) f(x) h = rounding error 2ε h computational error = D h f(x) f (x) 1 2ε Mh + 2 h 6

7 Noise in function evaluation (see noisyfun-demo.m) > f=inline( 1-4*x+6*x 2-4*x 3 +x 4 ) > ezplot(f,[0.9996, ]) f is a polynomial = graph smooth! But use of finite precision arithmetic leads to a noisy function evaluation, e.g. > p=[ ] > roots(p) Use of finite precision arithmetic leads to a noisy function evaluation. This feeds into other algorithms that use f, e.g. finding roots f(x ) = 0 In fact, 1 4x + 6x 2 4x 3 + x 4 = (x 1) 4 = unique real root x = 1 > hold on > g=inline( (x-1) 4 ) > ezplot(g,[ ]) > q=[ ] > roots(q)+1 Accuracy at the same machine precision can depend upon the algorithm! (c) Propagation of Errors Typical numerical problem: compute value of function f(x) : x=true value of input f(x)=true value of output ˆx=approximate input ˆf =approximate function computed Err= ˆf(ˆx) f(x) = [ ˆf(ˆx) f(ˆx)] }{{} computational error + [f(ˆx) f(x)] }{{} propagated error Note propagated error does not depend upon the algorithm to compute ˆf! Usually ˆf(x) fl(f(x)) = comp. error = ˆf(ˆx) f(ˆx) f(ˆx) 1 2 β p. 7

8 example in MATLAB: > format long > a=4/3 > b=a-1 > c=3*b > d=1-c Note 4/3 not exactly represented in F! Instead try a=3/2, c=2*b. Error propagation in arithmetic operations Caricature of machine arithmetic: x ˆ+y = fl[fl(x) + fl(y)] x ˆ y = fl[fl(x) fl(y)] x ˆ y = fl[fl(x) fl(y)] x ˆ y = fl[fl(x) fl(y)] x 1 x 2 ˆx 1ˆ ˆx 2 = [x 1 x 2 ˆx 1 ˆx 2 ] + [ˆx 1 ˆx 2 ˆx 1ˆ ˆx 2 ] propagated error in exact arithmetic computational error computational error: ˆx 1 ˆx 2 ˆx 1ˆ ˆx ˆx 1 ˆx 2 β p for rounding propagated error: Rel(x 1 x 2 ) = Rel(x 1 ) + Rel(x 2 ) + Rel(x 1 ) Rel(x 2 ) Rel(x 1 ) + Rel(x 2 ) Rel(x 1 x 2 ) = Rel(x 1) Rel(x 2 ) Rel(x 1 ) Rel(x 2 ) 1 Rel(x 2 ) ( ) ( ) x1 x2 Rel(x 1 + x 2 ) = Rel(x 1 ) + Rel(x 2 ) x 1 + x 2 x 1 + x ( ) ( 2 ) x1 x2 Rel(x 1 x 2 ) = Rel(x 1 ) Rel(x 2 ) x 1 x 2 x 1 x 2 8

9 There is a possibility that Rel(x 1 + x 2 ), Rel(x 1 x 2 ) max{rel(x 1 ), Rel(x 2 )} if x 1 + x 2 or x 1 x 2 x 1, x 2 Addition and subtraction are dangerous!!! Example: e = ê = r = 163/60 = ˆr = d = e r = ˆd = ê ˆr = 10 3 Rel(ê) = ê e e Rel(ˆr) = ˆr r r Rel( ˆd) = ˆd d d !!! Inaccuracy caused by cancellation of significant digits is called loss of significance error Example: The two roots of the quadratic polynomial ax 2 + bx + c = 0 are r 1 (a, b, c) = b + b 2 4ac 2a r 2 (a, b, c) = b b 2 4ac 2a where assume 4ac b 2. To be specific consider with 5-digit arithmetic: x 2 26x + 1 = 0 = r 1 = , r 2 = ˆr 1 = = ˆr 2 = =

10 Then, ε 1. = but ε 2 = !!! loss-of-significance error A solution is rationalization of the numerator: r 2 (a, b, c) = b ( ) b 2 4ac b + b2 4ac 2a b + b 2 4ac 2c = b + b 2 4ac 1 r 2 = ˆr 2 = = ε 2 = For another example, see lossofsig demo.m. Error propagation for other functions f(ˆx) = f(x + δx) f(x) + f (x)δx = Err(f(ˆx)) f (x)δx Rel(f(ˆx)) f (x) f(x) δx = xf (x) f(x) Rel(ˆx) = d ln f d ln x Rel(ˆx) example: f(x) = tan x, f (x) = sec 2 x = xf (x) f(x) = x sin x cos x = 2x sin(2x). x = 0 x = ( π 2 ) xf (x) = 1 f(x) xf (x) = !!! f(x) 10

11 1.3 Algorithms and Convergence A mathematical problem is well-posed or stable if the solution exists, is unique, depends continuously on input data. Otherwise, the problem is called ill-posed or unstable. examples: 1) Evaluating a continuous function f(x) on its range is well-posed 2) Solving for the two roots (r +, r ) of ax 2 + bx + c = 0 r ± = b ± b 2 4ac 2a always exist (as complex numbers), are unique and continuous in a, b, c = well-posed! 3) Solve [ ] [ x y ] = [ a b Solution [ ] does not exist [ if a/b 10. x x + 1 If is a soln, c ] 19 y y 1c is a soln for any c = not unique 4 This problem is ill-posed/unstable ]. However, there are degrees of well-posedness!! A mathematical problem is well-conditioned if a relative change in input causes a similar relative change in solution, and ill-conditioned if the relative change is much larger. condition number = relative change in output relative change in input 11

12 Evaluation of a continuous function f at x. Condition number: for x = x + δx with δx x. [f(x ) f(x)]/f(x) K f (x) = sup x (x x)/x x df f(x) dx = d ln f d ln x example: f(x) = tan x, K f (x) = 2x/ sin(2x). > f=inline( tan(x) ) > K=inline( 2*x./sin(2*x) ) > hold on > ezplot(f,[-pi/2 pi/2]) > ezplot(k,[-pi/2 pi/2]) More generally for y j = f j (x 1,..., x n ), j = 1,..., m n f j δy j δx i x i=1 i n x i f j ε yj (x)ε xi f i=1 j (x) x i K yj x i = sup x i f j x f j (x) x i In the examples of addition y = x 1 + x 2 or subtraction y = x 1 x 2 we see that K yx1 = K yx2 = +. These are ill-conditioned. For a more complete discussion, see Stoer & Bulirsch, Section 1.3. Algorithms sequences of calculations in exact arithmetic to approximately solve a problem in N steps (where it is possible N = ) E.g. y = f(x) might be solved by } x y n = f n (x n ) with n x n N f n f f n a sequence of elementary arithmetic operations 12

13 pseudocode structural outline of program to carry out (in machine arithmetic) an algorithm. See examples in text. Stability of numerical algorithm Algorithm f n may be unstable even if the original problem f is stable! Let ˆK(y) = lim n N K n (y n ) where K n (y n ) is the condition number of y n = f n (x n ) A criterion of stability of the numerical algorithm is that ˆK(y) 2K(y) Example: The Bessel functions are defined by ( x ) m ( 1) k ( x ) 2k J m (x) =. 2 k!(m + k)! 2 k=0 Consider the problem to calculate J 20 (x) by iterating in m the standard recursion relation ( ) 2m J m+1 (x) = J m (x) J m 1 (x) x with inputs x, J 0 = J 0 (x), J 1 = J 1 (x). Our MATLAB demo bessdemo.m for x = 2 shows that Rel(Ĵ20(x)) whereas all the inputs x, J 0, J 1 are known to relative error eps Thus, cond. num. = Rel(Ĵ20(x)) eps = algorithm unstable!! Note however that the condition number for Bessel function evaluation is K Jm (x) = m xj m+1(x) J m (x) using xj m(x) = mj m (x) xj m+1 (x) = K J20 ( 2) , stable!! 13

14 It is also possible to have ˆK(y) < K(y), e.g. an ill-posed problem can have a stable (but generally ill-conditioned) algorithm. example: Consider the MATLAB algorithm mldivide(a,b) or A\b to solve the linear system Ax = b. Try: > A=[19 4; ] > b=[23 2.3] > x=a\b A unique answer is obtained, although the mathematical problem has infinitely many solutions! However, the MATLAB algorithm is ill-conditioned. Try instead: > b=[ ] The output changes by a factor of 10 11! A transformation of an ill-posed problem into a well-posed (but often illconditioned) problem is called a regularization. Rates of convergence Another important issue with computational algorithms is the rate at which f n (x n ) f(x) as n. An algorithm is a pth-order approximation if ( ) 1 f n (x n ) f(x) = O n p We use the Landau big-o notation: If lim h 0 G(h) = 0 and F (h) L C G(h) for some constant C for h h 0, we write F (h) = L + O(G(h)) Note that a pth-order algorithm has the property that when n is increased by a factor of 10, then the approximation adds at least p significant digits. 14

15 This notion should not be confused with the so-called convergence rate of order p which holds when f n+1 (x n+1 ) f(x) lim = µ n f n (x n ) f(x) p for some constant µ > 0. In particular, convergence with order p = 1 is called linear convergence (with µ < 1) p = 2 is called quadratic convergence p = 3 is called cubic convergence, etc. Thus, an algorithm with linear convergence in one iteration adds log 10 (1/µ) significant digits, whereas a pth-order method for p 2 in one iteration increases the number of significant digits by a multiplicative factor of p!!! 1.4 Numerical Software Special Purpose Algorithms Burden-Faires student site: Numerical Computing with Matlab (NCM): General Purpose Algorithms freeware ( ( commercial software