1.1 COMPUTER REPRESENTATION OF NUM- BERS, REPRESENTATION ERRORS

Transcription

1 Chapter 1 NUMBER REPRESENTATION, ERROR ANALYSIS 1.1 COMPUTER REPRESENTATION OF NUM- BERS, REPRESENTATION ERRORS Floating-point representation x t,r of a number x: x t,r = m t P cr, where: P - base (the case P =2will further only be considered), m t - mantissa, m t < 1, t -numberof(basep) digits in the mantissa, c r -exponent, r - number of digits in the exponent. Usually, a normalized representation is used, i.e. 1 2 m t < 1. An example base 2 number for t =4and r =2: x = , where signs of the exponent and of the mantissa have been printed in bold. Converting the example base 2 number to decimal one, we have x =2 +( ) [ ]=2.75 Format of 32 bits number according to IEEE standard 754: sign exponent 24 bits normalized mantissa bit (shifted) (first bit always equals 1 omitted) (1 bit) (8 bits) (23 bits) The set of computer floating-point numbers M R is finite, moreover: 1

2 2 CHAPTER 1. NUMBER REPRESENTATION, ERROR ANALYSIS -thelargert the more numbers is contained in the same interval (the set M is more dense ), -thelargerr the wider interval of numbers is covered by the set M. Assuming the representation of the exponent is exact, i.e. c r = c (the set M covers the whole range of numbers of interest), and denoting floating-point representation of a number x by rd(x), we have The representation is most exact if rd (x) =m t 2 c. rd (x) x min g x. g M The requirement of most exact representation is fulfilled by a standard rounding: tx m t = e i 2 i + e (t+1) 2 t, where e -1 =1, e -i =0or 1 for i =2, 3,..t+1. Hence, the roundoff error satisfies m m t 2 (t+1), where m denotes the exact mantissa (x = m 2 c ). We have rd (x) x = (m t m)2 c = m t m x m2 c m, thus the roundoff relative error satisfies because m 2 1. Using the relation we can write rd (x) x x The obtained relation implies = m t m m 2 (t+1) 2 1 =2 t, m m t 2 (t+1) =2 t t m m m t = m ε, where ε 2 t. rd (x) =m t 2 c = m2 c +(m t m)2 c = m2 c + mε2 c = m2 c (1 + ε), whichcanbewrittenintheform(most important form of the roundoff relative error, in the numerical error analysis) rd (x) =x (1 + ε), ε 2 t = eps,

3 P. Tatjewski: NUMERICAL METHODS 3 where eps denotes machine epsilon ( macheps ), eps =2 t for the considered case of the roundoff error. Base 2 t-digit floating-point representation resulting from a truncation ( chopping) of the exact number representation: tx m t = e i 2 i, hence the chopping absolute error bound is 2 t (rounding absolute error bound was 2 (t+1) for the same number), m m t 2 t. Proceeding analogously as in the case of the roundoff error an analogous representation of the chopping relative error can be obtained rd (x) =x (1 + ε), ε 2 t+1 = eps. It can be easily concluded that if a double precision arithmetic is used then ε dp =(ε) 2, where the subscript dp denotes double precision. 1.2 FLOATING-POINT ARITHMETIC Elementary arithmetic operations: +,,,/. The arithmetic operations on floating-point numbers result in floating point numbers. Denoting by fl aresultofafloating point calculation, results of elementary arithmetic operations (supplemented by the square root finding) can be written in the form fl(x ± y) =(x ± y) (1 + ε) fl(x /y) =(x /y)(1+ε), fl x = x (1 + ε), where ε eps. This means an assumption that errors in elementary arithmetic operations result only from rounding the exact results of these operations. It results in practice from the fact that computer arithmetic units use registers of longer size than the length of the machine word corresponding to the used floating-point number representation (as applied to store numbers in computer memory). Remark: Machine epsilon eps can also be defined as a minimal positive machine floating-point number g satisfying the relation fl(1 + g) > 1, i.e. eps def =min{g M : fl(1 + g) > 1, g>0}. Therefore, eps defined in the above way is also called the unit round [?].

4 4 CHAPTER 1. NUMBER REPRESENTATION, ERROR ANALYSIS 1.3 CONDITION NUMBER Describe a mathematical calculation problem as a mapping φ : R n 7 R m, w = φ (d), where d =(d 1,..., d n ) a vector of data, w =(w 1,..., w m ) a vector of results. In practice, floating-point representation (integer representation also possible) of data numbers is used for calculations, i.e. rd (d i )=d i (1 + ε i ),where ε i eps, i =1,..., n. A calculation problem is said to be ill-conditioned if small data perturbations result in large, significant changes in the result clearly, relative errors should be taken into consideration. A number characterising quantitatively an increase of relative errors in the result versus relative errors in the data is called the condition number of the given calculation task. Example. Consider the following problem: to calculate the scalar product of two vectors a P and b, φ(a, b) = n a i b i, where a) a = [1 2 3], b =[2 6 5], b) a =[ ], b =[2 6 5]. Inthecasea)theresultisw = φ(a, b) = 1. Butinthecaseb)w = 0.52 data perturbations corresponding to 2% relative errors result in 48% relative error in the result. It can be easily checked that replacing the vector b by [2 6 5] yields much smaller perturbation in the result, with the relative error similar to the relative errors in the data. Conclusion: conditioning of a calculation task depends not only on the task itself, but first of all on actual values of the task data. Denoting an absolute data error by d, d =[ d 1, d 2,..., d n ] T, a relative data error can be written as k dk kdk. If the data are of the form d i (1 + ε i ),where ε i c d, then d i = d i ε i, d =[d 1 ε 1,d 2 ε 2,..., d n ε n ] T and k dk k d k = p (d1 ε 1 ) 2 +(d 2 ε 2 ) 2 + +(d 2 ε 2 ) 2 k d k k d kc d k d k = c d, (1.1) where c d = eps for floating point data representation errors and k k denotes the Euclidean norm (the result is also valid for another standard vector norms first norm, maximum norm, see Chapter 2). Denoting an absolute solution error by w, a relative error in the solution can be written as k wk kφ (d+ d) φ (d)k =. kwk kφ (d)k

5 P. Tatjewski: NUMERICAL METHODS 5 We are interested in a relation between the relative solution error versus the relative data error. Define cond(d) = sup k dk k wk kwk, (1.2) k dk kdk where sup denotes supremum, i.e. upper limit and the supremum in (1.2) is taken over all small data perturbations d for which the perturbed problem φ (d+ d) still makes sense. The number cond(d) is called the condition number. The condition number as defined by (1.2) is a measure of the sensivity of the solution w to small data errors d. It indicates the worst possible increase of the relative solution error k wk kwk versustherelativedataerror k dk kdk. The definition (1.2) can be rewritten as k wk kwk k dk cond (d) k d k, (1.3) where cond(d) is the smallest number for which (1.3) is true. The condition number describes the maximal possible relative error in the problem solution caused by the data errors only, i.e. without errors in all elementary mathematical operations leading from the (perturbed) data to the solution. It was not possible to evaluate the condition number (it is in fact a mapping dependent on the data d) precisely as defined by (1.2) for many practical problems. Therefore, it is common to describe as the condition number cond (d) the best known estimate (i.e. closest to that defined by (1.2)) for which the inequality (1.3) holds. The relative data error can be defined in a different way than (1.1). Therefore the relation (1.3) can also be written in the form k wk kwk cond(d) RDE, (1.4) where RDE denotes Relative Data Error that can be defined as in (1.1) or in another way. Example. P Consider the calculation task φ (a, b) = n a i b i. Data are perturbed, represented as a i (1 + α i ) and b i (1 + β i ),where α i and β i

6 6 CHAPTER 1. NUMBER REPRESENTATION, ERROR ANALYSIS define relative data errors. P k wk = w a i (1 + α i ) b i (1 + β i ) n a i b i = kwk w where = (a i b i α i + a i b i β i + a i b i α i β i ) a i b i (α i + β i ) a i b i 1 a i b i (α i + β i ) (a i b i α i + a i b i β i ) max { α i + β i } = cond (a, b) RDE i a i b i cond (a, b) = and RDE =max{ α i + β i } 2eps, i if α i eps and β i eps. Let us observe that defining the problem relative data error as a sum of relative data errors of both individual data vectors a and b leads to the identical estimate RDE = k ak kak + k bk kbk 2eps. Generally, if a data vector consists of subvectors, d =(d 1, d 2,..., d k ), andifasumof subvector relative data erors is taken as a RDE,then RDE = k d1 k kd 1 k + k d2 k kd 2 k d k kd k k c d = k d eps. The estimate RDE c d = k d eps, (1.5)

7 P. Tatjewski: NUMERICAL METHODS 7 where k d is a small multiplicity of 1, is used as a certain generalisation of the relation k dk eps the relation (1.1) for the case of floating-point data representation errors kdk when c d = eps. In the estimations used in the last example a symbol 1 ofanapproximate inequality has been used. This symbol denotes (roughly speaking) that we leave in the sum only these components where data errors enter at most linearly (k α i and k β i in the example, where k is a constant). This means that we omit terms with errors entering in higher orders (k α i β i in the example, the terms k (α i ) 2,etc. would also be omitted, analogously). Being more precise, terms decreasing (with the increase of t) at least k 2 t times quicker than other terms are omitted, where k is aconstant. 1.4 ALGORITHM AND ITS NUMERICAL REA- LISATIONS Three basic and different definitions: Amathematicalproblem w =φ (d), An algorithm A(d): a recipe how to calculate the problem φ (d), i.e. a definition of a uniquely ordered sequence of elementary arithmetical operations leading from the data to the result, Numerical realisation fl(a (d)) of the algorithm A (d): a) all numbers (constants) present in the definition of the algorithm A (d) are replaced by its numerical representations, b) calculation of all elementary arithmetical operations (including standard mathematical functions) in a sequence defined by the algorithm A (d) in floating-point arithmetic fl, i.e. with numerical errors in all operations. Example: a mathematical problem: φ(a.b) =a 2 b 2, the algorithm A1(a, b): a a b b, the algorithm A2(a, b): (a + b) (a b).

8 8 CHAPTER 1. NUMBER REPRESENTATION, ERROR ANALYSIS Numerical realisation of the algorithm A1(a, b) : fl(a1(a, b) = fl(a a b b) = fl[fl(a a) fl(b b)] = [a 2 (1 + ε 1 ) b 2 (1 + ε 2 )](1 + ε 3 ) = [a 2 b 2 + a 2 ε 1 b 2 ε 2 ](1 + ε 3 ) = a 2 b 2 + a 2 ε 1 b 2 ε 2 +(a 2 b 2 )ε 3 + a 2 ε 1 ε 3 b 2 ε 2 ε 3 = 1 a 2 b 2 + a 2 ε 1 b 2 ε 2 +(a 2 b 2 )ε 3 = (a 2 b 2 )(1 + a2 ε 1 b 2 ε 2 a 2 b 2 + ε 3 ) = (a 2 b 2 )(1 + δ 1 ), where δ 1 = a2 ε 1 b 2 ε 2 + ε a 2 b 2 3 is the result relative error, δ 1 = a 2 ε 1 b 2 ε 2 + ε a 2 b 2 3 a 2 + b 2 a 2 b 2 eps + eps. Numerical realisation of the algorithm A2(a, b) : fl(a2(a, b) = fl[(a + b) (a b)] = fl[fl(a + b) fl(a b)] = [(a + b)(1 + ε 1 ) (a b)(1 + ε 2 )](1 + ε 3 ) = (a 2 b 2 )(1 + ε 1 )(1 + ε 2 )(1 + ε 3 ) = 1 (a 2 b 2 )(1 + ε 1 + ε 2 + ε 3 ) = (a 2 b 2 )(1 + δ 2 ), where δ 2 = ε 1 + ε 2 + ε 3 is the result relative error, δ 2 = ε 1 + ε 2 + ε 3 3eps. It is evident that the algorithm A1 is numerically inferior: for cases when a 2 = b 2 the relative error δ 1 can become even several orders of magnitude larger than the data relative error, whereas the relative error δ 2 of the algorithm A2 is insensitive to the values of the data vectors a and b.

9 1.5. NUMERICAL STABILITY NUMERICAL STABILITY Consider again a mathematical task φ, w = φ(d), leading from the data d to the solution w and recall that the relative data error (RDE, see section 1.3) can be estimated as follows RDE k d eps, where k d is a constant of the order of one. According to the definition of the condition number (1.3) we have k φ (d+ d) φ (d) k kφ (d)k cond (d) k d eps, where w =φ (d) is the precise mathematical result corresponding to accurate data values d and φ (d+ d) is the precise mathematical result corresponding to perturbed data values d+ d in the above definitions numerical errors in a numerical realisation fl(a(d)) of the algorithm A(d) solving the problem φ (d) were not taken into account. Now, considering these errors leads to the definition of numerical stability. An algorithm A(d) calculating a mathematical task φ (d) is called numerically stable if there exist a positive constant K s such that for every data values d from a given data set of interest D and for every sufficiently small eps (i.e., for all sufficiently strong arithmetics) the following inequality holds: kfl(a (d)) φ (d)k kφ (d)k K s cond (d) k d eps, (1.6) where the constant K s, called the stability index, is usually assumed to be not much greater than 1. It means that a numerically stable algorithm produces solutions with relative errors of about the same order of magnitude as relative errors caused by data perturbations only. In another words, a relative error in a numerical solution obtained using a stable algorithm should be of about the same order of magnitude as an error caused by data perturbations only. It follows directly from the definition (1.6) that for a numerically stable algorithm k fl(a (d)) φ (d) k lim eps 0 kφ (d)k =0. (1.7) The relation (1.7) can be sometimes found in the literature as a definition of the numerical stability. Optional. Thequantity P (d, φ) =cond (d) k d eps kwk + eps kwk (1.8) is called the inevitable error (or optimal error level), because it is not dependent on the algorithm used to calculate the task φ(d). It consists of two terms:

10 10 CHAPTER 1. NUMBER REPRESENTATION, ERROR ANALYSIS the first is an estimate of the absolute solution error caused by data perturbations only and the second is the estimate of the (exact) solution floating-point representation error. The inevitable error depends on the task condition number (cond (d)) and on the used floating-point arithmetic ( eps). It follows directly from the definition (1.8) that the following important equality holds lim P (d, φ) =0. eps 0 The numerical stability can be defined using instead of (1.6) the relation in fact an equivalent definition. kfl(a (d)) φ (d)k K s P (d, φ), Concluding, a complete relative error of a numerical solution to a mathematical problem, obtained using a numerically stable algorithm depends on: - the problem conditioning (cond(d)), -the floating-point arithmetic used (eps), - the algorithm numerical stability index (K s ). Numerical stability of algorithms can be often proved using the method of equivalent perturbations. The method relies on proving (if possible) that the numerical result fl(a(d)) is equivalent to a slightly perturbed exact solution to the task with slightly perturbed data, i.e. where fl(a (d)) = φ (d + d) (1 + η), (1.9) d i k i eps d i, ηj k j eps, whereas k i and k j are constants (not too large). Algorithms satisfying the equality (1.9) are called numerically correct. Every numerically correct algorithm is numerically stable. Effectiveness of an algorithm is evaluated as a number of elementary arithmetic operations needed to pass from the data to the result, in particular numbers of additions and subtractions AS and multiplications and divisions MD are usually stated.

11 P. Tatjewski: NUMERICAL METHODS 11 Problems. 1. Design an algorithm for calculating the task φ (x) =x 1 + x 2 + x 3, gdzie x 1 >x 2 >x 3 > 0, in a way leading to the best (smallest) estimate of numerical errors. 2. Compare numerical properties and effectiveness of the following two algorithms calculating values of the polynomial w(x), w (x) =x 3 + a 1 x 2 + a 2 x + a 3, A1: x x x + a 1 x x + a 2 x + a 3 A2: x [x (x + a 1 )+a 2 ]+a 3 (Horner scheme) 3. Evaluate an upper bound on the relative error of a numerical result obtained using the following algorithm (a +2 b) (a a b), assuming rd(a) =a, rd(b) =b. Try to modify the algorithm to improve properties of the upper bound estimate. 4. Evaluate upper bounds for E i,if à 3X! fl a i b i = 3X a i b i (1 + E i ) and rd(a i )=a i, rd(b i )=b i, i =1, 2,..., n.