AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 09: Accuracy and Stability Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 12

Outline 1 Condition Number of Matrices 2 Accuracy and Stability 3 Stability of Algorithms Xiangmin Jiao Numerical Analysis I 2 / 12

Condition Number of Matrix-Vector Product Consider f (x) = Ax, with A C m n κ = J f (x) / x = A x Ax If A is square and nonsingular, since x / Ax A 1 κ A A 1 Question: For what x is equality achieved if 2-norm is used? Answer: x is equal to right singular vector corresponding to smallest singular value of A Question: What is condition number of Ax if A is singular? Answer: (is if x null(a)). What is the condition number for f (b) = A 1 b? Xiangmin Jiao Numerical Analysis I 3 / 12

Condition Number of Matrix We define condition number of matrix A as κ(a) = A A 1 It is the upper bound of the condition number of f (x) = Ax for any x Another way to interpret at κ(a) is κ(a) = sup δx,x δf / δx f (x) / x = sup δx Aδx / δx infx Ax / x For 2-norm, κ(a) = σ 1 /σ n Note about the distinction between the condition number of a problem (the map f (x)) and the condition number of a problem instance (the evaluation of f (x) for specific x) Note: condition number of a problem is a property of a problem, and is independent of its algorithm Xiangmin Jiao Numerical Analysis I 4 / 12

Outline 1 Condition Number of Matrices 2 Accuracy and Stability 3 Stability of Algorithms Xiangmin Jiao Numerical Analysis I 5 / 12

Accuracy Roughly speaking, accuracy means that error is small in an asymptotic sense, say O(ɛ machine ) Notation ϕ(t) = O(ψ(t)) means C s.t. ϕ(t) C ψ(t) as t approaches 0 (or ) Example: sin 2 t = O(t 2 ) as t 0 If ϕ depends on s and t, then ϕ(s, t) = O(ψ(t)) means C s.t. ϕ(s, t) C ψ(t) for any s as t approaches 0 (or ) Example: sin 2 t sin 2 s = O(t 2 ) as t 0 When we say O(ɛ machine ), we are thinking of a series of idealized machines for which ɛ machine can be arbitrarily small Xiangmin Jiao Numerical Analysis I 6 / 12

More on Accuracy An algorithm f is accurate if relative error is in the order of machine precision, i.e., f (x) f (x) / f (x) = O(ɛ machine ), i.e., C 1 ɛ machine as ɛ machine 0, where constant C 1 may depend on the condition number and the algorithm itself In most cases, we expect f (x) f (x) / f (x) = O(κɛ machine ), i.e., Cκɛ machine as ɛ machine 0, where constant C should be independent of κ and value of x (although it may depends on the dimension of x) How do we determine whether an algorithm is accurate or not? It turns out to be an extremely subtle question A forward error analysis (operation by operation) is often too difficult and impractical, and cannot capture dependence on condition number An effective solution is backward error analysis Xiangmin Jiao Numerical Analysis I 7 / 12

Outline 1 Condition Number of Matrices 2 Accuracy and Stability 3 Stability of Algorithms Xiangmin Jiao Numerical Analysis I 8 / 12

Stability We say an algorithm is stable if it gives nearly the right answer to nearly the right question More formally, an algorithm f for problem f is stable if (for all x) f (x) f ( x) / f ( x) = O(ɛ machine ) for some x with x x / x = O(ɛ machine ) We say an algorithm is backward stable if it gives exactly the right answer to nearly the right question More formally, an algorithm f for problem f is stable if (for all x) f (x) = f ( x) for some x with x x / x = O(ɛ machine ) Xiangmin Jiao Numerical Analysis I 9 / 12

Stability of Floating Point Arithmetic Backward stability of floating point operations is implied by these two floating point axioms: 1 x R, ɛ, ɛ ɛ machine s.t. fl(x) = x(1 + ɛ) 2 For floating-point numbers x, y, ɛ, ɛ ɛ machine s.t. x y = (x y)(1 + ɛ) Example: Subtraction f (x 1, x 2 ) = x 1 x 2 with floating-point operation f (x 1, x 2 ) = fl(x 1 ) fl(x 2 ) Axiom 1 implies fl(x1 ) = x 1 (1 + ɛ 1 ), fl(x 2 ) = x 2 (1 + ɛ 2 ), for some ɛ 1, ɛ 2 ɛ machine Axiom 2 implies fl(x 1 ) fl(x 2 ) = (fl(x 1 ) fl(x 2 ))(1 + ɛ 3 ) for some ɛ 3 ɛ machine Therefore, fl(x 1 ) fl(x 2 ) = (x 1 (1 + ɛ 1 ) x 2 (1 + ɛ 2 ))(1 + ɛ 3 ) = x 1 (1 + ɛ 1 )(1 + ɛ 3 ) x 2 (1 + ɛ 2 )(1 + ɛ 3 ) = x 1 (1 + ɛ 4 ) x 2 (1 + ɛ 5 ) where ɛ 4, ɛ 5 2ɛ machine + O(ɛ 2 machine ) Xiangmin Jiao Numerical Analysis I 10 / 12

Stability of Floating Point Arithmetic Cont d Example: Inner product f (x, y) = x y using floating-point operations and is backward stable Example: Outer product f (x, y) = xy using and is not backward stable Example: f (x) = x + 1 computed as f (x) = fl(x) 1 is not backward stable Example: f (x, y) = x + y computed as f (x, y) = fl(x) fl(y) is backward stable Xiangmin Jiao Numerical Analysis I 11 / 12

Accuracy of Backward Stable Algorithm Theorem If a backward stable algorithm f is used to solve a problem f with condition number κ using floating-point numbers satisfying the two axioms, then f (x) f (x) / f (x) = O(κ(x)ɛ machine ) Proof: Backward stability means f (x) = f ( x) for x such that x x / x = O(ɛ machine ) Definition of condition number gives f ( x) f (x) / f (x) (κ(x) + o(1)) x x / x where o(1) 0 as ɛ machine 0. Combining the two gives desired result. Xiangmin Jiao Numerical Analysis I 12 / 12