Math 54 - Numerical Analysis Lecture Notes Linear Algebra: Part B Outline Joseph M. Mahaffy, jmahaffy@mail.sdsu.edu Department of Mathematics and Statistics Dynamical Systems Group Computational Sciences Research Center San Diego State University San Diego, CA 9282-7720 http://jmahaffy.sdsu.edu 2 Vector 3 and Gaussian Elimination Spring 208 (/2 (2/2 Errors Errors: Consider the system Consider the system Ax b Ax b with x A b The coefficients in A and values in b are rarely known exactly Experimental (observational and round-off errors enter almost every system How much effect is there from perturbations to the system? Problems arise when A is singular or nearly singular Singular matrices result in either no solution to the system or the solution is not unique If A is near the identity, then small changes in b result in small changes in x Almost always some computed error Denote this by x The error is given by e x x The residual is given by r b Ax If either error is zero, theory gives the other being zero What if one of the errors is small? (3/2 (4/2
Roundoff Error Example Example: Perform 3-digit arithmetic on the system: ( ( ( 0.780 0.563 x 0.27 0.93 0.659 x 2 0.254 Perform Gaussian Elimination with partial pivoting, beginning with R R 2 The multiplier is m 0.780 0.93 0.854 and the operation is R 2 R 2 m R, resulting in ( ( ( 0.93 0.659 x 0.254 0 0.00 x 2 0.00 This gives x 2 0.00 0.00 and x 0.254 0.659 0.93 0.443 (5/2 Roundoff Error Example Example from before is: ( 0.780 0.563 0.93 0.659 ( ( x 0.27 x 2 0.254 and 3-digit arithmetic gave the solution x 0.443 and x 2.000 The residual is easily calculated ( 0.27 0.780( 0.443 + 0.563(.000 r b Ax 0.254 0.93( 0.443 + 0.659(.000 which has each component < 0 3 It is easy to see that the actual solution is ( ( x.000 x 2.000 which is far from the computed solution, ( 0.000460 0.00054 (6/2, Roundoff Error Example Vector In this Example, the matrix is close to singular, so not typical. With 6 or more digits, the Gaussian Elimination with partial pivoting gives ( 0.93000 0.659000 0 0.00000 and the correct solution arises ( ( x.00000 x 2.00000 ( ( x 0.254000 x 2 0.00000 Because of the singularity almost any answer could arise for x 2. The computations produce a very small residual, and the two equations are very close to each other. If singular, then a single equation satisfies the problem. Thus, the first equation of the original problem is nearly parallel and close to the first equation of the problem with a little error. (7/2 Definition (l p Norm Consider an n-dimensional vector x [x,..., x n ] T. The l p norm for the vector x is defined by the following: ( n /p x p x i p. i Almost always the norms use p, p 2 (Euclidean or distance, or p (max ( x For x, we have x 2 ( x 2 + x 2 2 /2 x 2 (8/2
Unit Circles Vector Vector Consider x in different norms l Norm x2 x l2 Norm x2 x l Norm x2 x x x 2 x or or or ( x + x 2 x 2 + x 2 2 /2 max{ x, x 2 } Let x [x,..., x n ] T R n, then the norms for p, p 2, or p satisfy: n x x i Property (Norm x 2 i ( n i x i 2 2 x max{ x i } i Given an n-dimensional vector x [x,..., x n ] T, then: x > 0, if x i 0 for some i, x 0, if x i 0 for all i. (9/2 (0/2 Norm Example Vector Norm of a Matrix Vector Example: Consider x [0.2, 0.4, 0.6, 0.8]. For p, x 4 x i 0.2 + 0.4 + 0.6 + 0.8 2.0 i MatLab command is norm(x, For p 2, ( 4 /2 x 2 x i 2 0.04 + 0.6 + 0.36 + 0.64.0954 i MatLab command is norm(x or norm(x,2 For p, x max x i 0.8 i MatLab command is norm(x,inf (/2 Definition (Matrix Norm A matrix norm on the set of all n n matrices is a real-valued function,, defined on this set, satisfying for all n n matrices A and B and all real numbers α: A 0; 2 A 0, if and only if A is 0, the matrix with all entries 0; 3 αa α A ; 4 A + B A + B ; 5 AB A B ; (2/2
Norm of a Matrix Vector Range of Changes for a Matrix and Gaussian Elimination Norm of a Matrix: There are a number of norms on a matrix. The most common norm for a matrix is defined by the vector norms for R n Theorem (Matrix Norm If is a vector norm on R n, then is a matrix norm. It follows that for any x Ax A max Ax max x x 0 x As x is varied, Ax varies. The range satisfies: M Ax max x 0 x m Ax min x 0 x Note: If A is singular, then m 0 Definition ( Let M and m be defined as above for a matrix A. The Condition Number for A is given by: A Ax x or Ax A x (3/2 κ(a M m By the definition it follows that κ (4/2 Rounding Error and Gaussian Elimination and Gaussian Elimination Consider a system of equations Ax b Assume that A is known, and there is an error in x with x + δx A(x + δx b + δb This assumes a roundoff error in x and the result of A acting on x produces a roundoff error δb With norms we see for A(δx δb If A is invertible, then δx A (δb δx A (δb A δb Definition (Alternate: The for a matrix A satisfies: κ(a A A. or A(δx δb or δb A δx δx A δb The, κ(a, measures how much impact roundoff has (5/2 (6/2
Error in Solution and Gaussian Elimination Properties and Gaussian Elimination For the perturbation, δx A (δb A δb κ(a δb A From Ax b, we have b A x or A x b It follows that Rearranging δx κ(a δb A κ(a δb x b δx x κ(a δb b The relative error in b gives the relative error provided κ(a Roughly, κ(a A A measures the relative error magnification factor (7/2 Some properties of the Property (Basic Properties The satisfies κ(a 2 If P is a Permutation Matrix, then κ(p 3 If D is a Diagonal Matrix, then κ(d max d ii min d ii (8/2 Example with l -norm and Gaussian Elimination Example with l -norm and Gaussian Elimination Example with l -norm: Let ( 4. 2.8 A 9.7 6.6 Clearly, Ax b with Make a small perturbation, so The solution becomes, b ( 4. 9.7 b 3.8 and x b ( 4. 9.70 x Let δb b b and δx x x, then ( 0.34 0.97 (, x 0 δb 0.0 and δx.63 (9/2 Example with l -norm: From before δb 0.0 and δx.63 Thus, a small perturbation in b completely changes x The relative changes are δb b 0.0007246 and δx x.63 Because κ(a is the maximum magnification factor κ(a.63 0.0007246 2249.4 The b and δb were chosen to give the maximum, so κ(a 2249.4 (20/2
and Gaussian Elimination and Gaussian Elimination The plays a fundamental role in the analysis of roundoff errors during the solution of Gaussian Elimination Assume that A and b are exact and x is obtained from the floating point arithmetic of the machine with ɛ precision Assuming no singularities, the following inequalities can be established: b Ax A x x x x ρɛ, ρκ(aɛ, where ρ 0 The first inequality implies that the relative residual is approximately the same as the machine roundoff error The second inequality implies that the relative error requires A is nonsingular, and the roundoff error scales with the condition number, κ(a (2/2