Numerics and Error Analysis CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Numerics and Error Analysis 1 / 30
A Puzzle What is the value of the following expression? abs(3.*(4./3.-1.)-1.) CS 205A: Mathematical Methods Numerics and Error Analysis 2 / 30
A Puzzle What is the value of the following expression? abs(3.*(4./3.-1.)-1.) Machine precision ε m : smallest ε m with 1 + ε m = 1 CS 205A: Mathematical Methods Numerics and Error Analysis 2 / 30
Prototypical Example double x = 1.0; double y = x / 3.0; if (x == y*3.0) cout << "They are equal!"; else cout << "They are NOT equal."; CS 205A: Mathematical Methods Numerics and Error Analysis 3 / 30
Take-Away Mathematically correct Numerically sound CS 205A: Mathematical Methods Numerics and Error Analysis 4 / 30
Using Tolerances double x = 1.0; double y = x / 3.0; if (fabs(x-y*3.0) < numeric_limits<double>::epsilon) cout << "They are equal!"; else cout << "They are NOT equal."; CS 205A: Mathematical Methods Numerics and Error Analysis 5 / 30
Counting in Binary: Integer 463 = 256 + 128 + 64 + 8 + 4 + 2 + 1 = 2 8 + 2 7 + 2 6 + 2 3 + 2 2 + 2 1 + 2 0 1 1 1 0 0 1 1 1 1 2 8 2 7 2 6 2 5 2 4 2 3 2 2 2 1 2 0 CS 205A: Mathematical Methods Numerics and Error Analysis 6 / 30
Counting in Binary: Fractional 463.25 = 256 + 128 + 64 + 8 + 4 + 2 + 1 + 1 /4 = 2 8 + 2 7 + 2 6 + 2 3 + 2 2 + 2 1 + 2 0 + 2 2 1 1 1 0 0 1 1 1 1. 0 1 2 8 2 7 2 6 2 5 2 4 2 3 2 2 2 1 2 0 2 1 2 2 CS 205A: Mathematical Methods Numerics and Error Analysis 7 / 30
Familiar Problem 1 3 = 0.0101010101... 2 Finite number of bits CS 205A: Mathematical Methods Numerics and Error Analysis 8 / 30
Fixed-Point Arithmetic 1 1 0. 0 1 1 2 l 2 l 1 2 0 2 1 2 k+1 2 k Parameters: k, l Z k + l + 1 digits total Can reuse integer arithmetic (fast; GPU possibility): a + b = (a 2 k + b 2 k ) 2 k CS 205A: Mathematical Methods Numerics and Error Analysis 9 / 30
Two-Digit Example 0.1 2 0.1 2 = 0.01 2 = 0.02 Multiplication and division easily change order of magnitude! CS 205A: Mathematical Methods Numerics and Error Analysis 10 / 30
Demand of Scientific Applications 9.11 10 31 6.022 10 23 Desired: Graceful transition CS 205A: Mathematical Methods Numerics and Error Analysis 11 / 30
Observations Compactness matters: 6.022 10 23 = 602, 200, 000, 000, 000, 000, 000, 000 CS 205A: Mathematical Methods Numerics and Error Analysis 12 / 30
Observations Compactness matters: 6.022 10 23 = 602, 200, 000, 000, 000, 000, 000, 000 Some operations are unlikely: 6.022 10 23 + 9.11 10 31 CS 205A: Mathematical Methods Numerics and Error Analysis 12 / 30
Scientific Notation ± sign Store significant digits (d 0 + d 1 b 1 + d 2 b 2 + + d p 1 b 1 p ) significand Base: b N Precision: p N Range of exponents: e [L, U] b e exponent CS 205A: Mathematical Methods Numerics and Error Analysis 13 / 30
Properties of Floating Point 0 0.5 1 1.5 2 2.5 3 3.5 1.25 1.75 0.625 0.75 0.875 Unevenly spaced Machine precision ε m : smallest ε m with 1 + ε m = 1 CS 205A: Mathematical Methods Numerics and Error Analysis 14 / 30
Properties of Floating Point 0 0.5 1 1.5 2 2.5 3 3.5 0.625 0.75 0.875 1.25 Unevenly spaced 1.75 Machine precision ε m : smallest ε m with 1 + ε m = 1 Needs rounding rule (e.g. round to nearest, ties to even ) CS 205A: Mathematical Methods Numerics and Error Analysis 14 / 30
Properties of Floating Point 0 0.5 1 1.5 2 2.5 3 3.5 0.625 0.75 0.875 1.25 Unevenly spaced 1.75 Machine precision ε m : smallest ε m with 1 + ε m = 1 Needs rounding rule (e.g. round to nearest, ties to even ) Can remove leading 1 CS 205A: Mathematical Methods Numerics and Error Analysis 14 / 30
Infinite Precision Q = { a /b : a, b Z} Simple rules: a /b + c /d = (ad+cb) /bd Redundant: 1 /2 = 2 /4 Blowup: 1 100 + 1 101 + 1 102 + 1 103 + 1 104 + 1 105 = 188463347 3218688200 Restricted operations: 2 2 CS 205A: Mathematical Methods Numerics and Error Analysis 15 / 30
Bracketing Store range a ± ε Special case of Interval Arithmetic Keeps track of certainty and rounding decisions Easy bounds: (x±ε 1 )+(y ±ε 2 ) = (x+y)±(ε 1 +ε 2 +error(x+y)) Implementation via operator overloading CS 205A: Mathematical Methods Numerics and Error Analysis 16 / 30
Sources of Error Rounding Discretization Modeling Input CS 205A: Mathematical Methods Numerics and Error Analysis 17 / 30
Example What sources of error might affect a financial simulation? CS 205A: Mathematical Methods Numerics and Error Analysis 18 / 30
Example Catastrophic cancellation Consider the function near x = 0. f(x) = ex 1 x 1 CS 205A: Mathematical Methods Numerics and Error Analysis 19 / 30
Absolute vs. Relative Error Absolute Error The difference between the approximate value and the underlying true value CS 205A: Mathematical Methods Numerics and Error Analysis 20 / 30
Absolute vs. Relative Error Absolute Error The difference between the approximate value and the underlying true value Relative Error Absolute error divided by the true value CS 205A: Mathematical Methods Numerics and Error Analysis 20 / 30
Absolute vs. Relative Error Absolute Error The difference between the approximate value and the underlying true value Relative Error Absolute error divided by the true value 2 in ± 0.02 in 2 in ± 1% CS 205A: Mathematical Methods Numerics and Error Analysis 20 / 30
Relative Error: Difficulty Problem: Generally not computable CS 205A: Mathematical Methods Numerics and Error Analysis 21 / 30
Relative Error: Difficulty Problem: Generally not computable Common fix: Be conservative CS 205A: Mathematical Methods Numerics and Error Analysis 21 / 30
Computable Measures of Success Root-finding problem For f : R R, find x such that f(x ) = 0. Actual output: x est with f(x est ) 1 CS 205A: Mathematical Methods Numerics and Error Analysis 22 / 30
Backward Error Backward Error The amount the problem statement would have to change to make the approximate solution exact CS 205A: Mathematical Methods Numerics and Error Analysis 23 / 30
Backward Error Backward Error The amount the problem statement would have to change to make the approximate solution exact Example 1: x CS 205A: Mathematical Methods Numerics and Error Analysis 23 / 30
Backward Error Backward Error The amount the problem statement would have to change to make the approximate solution exact Example 1: x Example 2: A x = b CS 205A: Mathematical Methods Numerics and Error Analysis 23 / 30
Conditioning Well-conditioned: Small backward error = small forward error Poorly conditioned: Otherwise Example: Root-finding CS 205A: Mathematical Methods Numerics and Error Analysis 24 / 30
Condition Number Condition number Ratio of forward to backward error CS 205A: Mathematical Methods Numerics and Error Analysis 25 / 30
Condition Number Condition number Ratio of forward to backward error Root-finding example: 1 f (x ) CS 205A: Mathematical Methods Numerics and Error Analysis 25 / 30
Theme Extremely careful implementation can be necessary. CS 205A: Mathematical Methods Numerics and Error Analysis 26 / 30
Example: x 2 double normsquared = 0; for (int i = 0; i < n; i++) normsquared += x[i]*x[i]; return sqrt(normsquared); CS 205A: Mathematical Methods Numerics and Error Analysis 27 / 30
Improved x 2 double maxelement = epsilon; for (int i = 0; i < n; i++) maxelement = max(maxelement, fabs(x[i])); for (int i = 0; i < n; i++) { double scaled = x[i] / maxelement; normsquared += scaled*scaled; } return sqrt(normsquared) * maxelement; CS 205A: Mathematical Methods Numerics and Error Analysis 28 / 30
More Involved Example: i x i double sum = 0; for (int i = 0; i < n; i++) sum += x[i]; CS 205A: Mathematical Methods Numerics and Error Analysis 29 / 30
Motivation for Kahan Algorithm ((a + b) a) b =? 0 Store compensation value! Details in textbook Next CS 205A: Mathematical Methods Numerics and Error Analysis 30 / 30