Numerical Linear Algebra

Transcription

1 Numerical Linear Algebra R. J. Renka Department of Computer Science & Engineering University of North Texas 02/03/2015

2 Notation and Terminology R n is the Euclidean n-dimensional linear space over the set of real numbers R for n 1. Matrices are denoted by uppercase letters: A R m n. If m = n, A is square and has order n. Vectors are denoted by lowercase letters: x R n. A vector is an n by 1 matrix for n 1. Scalars are denoted by Greek letters: α R, or subscripted letters: x i, a ij = A ij.

3 Notation and Terminology continued The transpose of matrix A is denoted A T and defined by (A T ) ij = a ji for all i and j. Note that (αa) T = αa T and (A + B) T = A T + B T. The identity matrix I = I n of order n is defined by I ij = δ ij, where δ ij is the Kronecker delta function. The inverse of an order-n matrix A is A 1 defined by AA 1 = A 1 A = I. The scalar product = inner product = dot product of n-vectors x and y is x T y.

4 Multiplication Scalar multiplication is defined by (αa) ij = αa ij. Matrix multiplication is defined by c ij = m a ik b kj k=1 for i = 1,..., l and j = 1,..., n, where A R l m, B R m n C = AB R l n. Theorem: Matrix multiplication is associative but not commutative. In fact, AB may be well-defined while BA is not. Suppose A R l m and B R m n, Then A T R m l, B T R n m, B T A T R n l, and (AB) T R n l, but A T B T is well-defined if and only if l = n. Theorem: (AB) T = B T A T.

5 Special cases of Matrix multiplication A linear system of equations involves a matrix-vector product: Ax = b for A R m n x R n = R n 1, b R m = R m 1. Note that Ax = b iff x T A T = b T. The expression xa is nonsense. For x, y R n, the inner product is x T y, where x T R 1 n, y R n 1 x T y R 1 1 = R, and the outer product (rank-1 matrix) is xy T R n n. Theorem (AB) 1 = B 1 A 1 for invertible matrices A and B.

6 Operation Counts In numerical linear algebra, algorithms are usually compared for complexity by counting multiplies. The number of adds is about the same as the number of multiplies. Asymptotic operation counts are too crude. Exercise Assign operation counts to the following expressions. Assume N-vectors and N by N matrices. αx αa x T y xy T Ax xy T z AB A 2 αi ABx A T A

7 Linear Systems A system of m equations in n unknowns is written as Ax = b, where A R m n, x R n, b R m. Theorem There is a 1-1 correspondence (isomorphism) between a matrix A R m n and a linear transformation L : R n R m s.t. L(αx) = αlx and L(x + y) = Lx + Ly for all x, y R n, α R. A and L are often used interchangeably. Defn A set of vectors {v 1, v 2,..., v k } R n is linearly independent iff k α i v i = 0 α 1 = α 2 =... = α k = 0. i=1

8 Existence and Uniqueness Let R(A) denote the range of A (as a linear transformation). This coincides with the span (set of all linear combinations) of the columns of A and is a linear subspace of R m (closed under linear combinations). If m < n the system is underdetermined and has infinitely many solutions for b R(A). If m > n the system is overdetermined and has no solution (unless there are at most n linearly independent rows in the augmented system). This is a linear least squares problem: Minimize Ax b 2 2 over x Rn.

9 Existence and Uniqueness continued Theorem For m = n, the following are equivalent. 1 Ax = b has a unique solution x for all b R n 2 A is invertible (nonsingular): B s.t. AB = BA = I 3 det(a) 0 4 Ax = 0 x = 0 1 A has linearly independent columns 2 N (A) = {0} : Ax = 0 x = 0 3 null(a) = dim(n (A)) = 0 4 A has no zero eigenvalue: if x 0 s.t. Ax = λx then λ 0 5 As a linear operator, A is one-to-one 6 A solution to Ax = b is unique 5 R(A) {Ax : x R n } = R n 1 The columns of A span R n 2 rank(a) = dim(r(a)) = n 3 As a linear operator, A is onto: x R n s.t. Ax = b b R n 4 There exists a solution to Ax = b for all b R n 6 A T is invertible: (A T ) 1 = (A 1 ) T = A T

10 Existence and Uniqueness continued The above list can be expanded by replacing A by A T, giving additional characterizations such as 7 A has linearly independent rows 8 The rows of A span R n Recall that a function is invertible if and only if it is both one-to-one and onto. In the case of a linear operator, the three properties, invertible, one-to-one, and onto, are equivalent. Also, uniqueness of a solution is equivalent to existence of a solution for all b. In the case n = 1 all of the above characterizations of invertibility reduce to A 0.

11 Existence and Uniqueness continued In the case n = 2 there is a simple geometric interpretation. The equations are a1 T x = b 1 and a2 T x = b 2, where a1 T and at 2 are the rows of A. The equations correspond to lines in the x 1 x 2 plane, and a solution is a point of intersection of the lines. Suppose A is singular. Then a 1 = αa 2 for some α. There are two possibilities: 1 b 1 = αb 2, the lines coincide, and there are infinitely many solutions, or 2 b 1 αb 2, the lines are distinct, and there is no solution. Note the sensitivity of the solution to perturbations of A and b when A is nearly singular.

12 Cramer s Rule Cramer s Rule is a good method for not solving a linear system Ax = b of order n: x i = A i / A, (i = 1,..., n), where A i is the matrix obtained by replacing column i by b in A. Operation count: The determinant can be expressed as A = σ S n sign(σ)a σ(1),1 a σ(2),2 a σ(n),n, where S n is the set of permutations of {1, 2,..., n}, and sign(σ) = ( 1) m, m being the number of transpositions in a decomposition of σ. The number of permutations is n!, and the number of multiplies M is (n 1)n! for each of the n + 1 determinants: M = (n + 1)(n!)(n 1).

13 Timing a Linear Solver Suppose a multiply takes a nanosecond (1 Gigaflops processor). Then, dividing M by multiplies per year, we find that, using Cramer s Rule, we can solve a system of order n = 30 in a mere years. Gaussian elimination, on the other hand has an operation count of M = n 3 /3 + n 2 n/3 multiplies, resulting in a computation time under 10 microseconds for n = 30. n M µs µs sec hrs.

14 Matrix Inverse Using the matrix inverse is a second method for not solving a linear system. The most efficient method for computing A 1 consists of the following three steps. 1 Compute an LU factorization A = LU by Gaussian elimination: n 3 /3. 2 Compute L 1 and U 1 : n 3 /6 + n 3 /6 = n 3 /3. 3 Compute A 1 = U 1 L 1 : n 3 /3. The total operation count is n 3 multiplies to compute A 1 and then n 2 multiplies for the matrix-vector product x = A 1 b. This is three times as much work as is necessary to compute x = A 1 b = (U 1 L 1 )b = U 1 (L 1 b), and consequently involves more accumulated roundoff error. For example, with four decimal digits of precision 7x = 21 fl(x) = (7 1 )(21) = (.1429)(21) = whereas fl(21/7) =

15 Matrix Inverse continued It is often mistakenly thought that multiple right hand sides justifies the additional cost of computing a matrix inverse. If, for example, we need to evaluate C = A 1 B, we can partition B and C by columns to obtain n linear systems Ac j = b j, (j = 1,..., n) for B = [b 1 b 2 b n ], C = [c 1 c 2 c n ]. Given the LU factorization of A, the cost of solving each linear system is the same as it would be with a matrix inverse: n 2 multiplies. This is the reason for computing the factorization, and then solving two triangular systems, rather than carrying along the right hand side during Gaussian elimination applied to an augmented system. The right hand side is treated separately from the matrix so that the factorization does not have to be repeated for a second right hand side with the same matrix.

16 Elementary Row Operations The following operations applied to a system of linear equations or augmented matrix (matrix with right hand side appended as an additional column) leave the solution unaltered. 1 Scale a row (by a nonzero scalar) 2 Add one row to another 3 Interchange a pair of rows The basic idea for automating Gaussian elimination is to apply the elementary row operations in a systematic fashion that zeros out all the elements below the diagonal. The solution components are then easily computed in reverse order.

17 Gaussian Elimination Forward Elimination: elementary row operations that reduce A to an upper triangular matrix U. a 11 a 12 a 13 b 1 a 21 a 22 a 23 b 2 a 31 a 32 a 33 b 3 a 11 a 12 a 13 b 1 0 a 22 a 23 b 2 µ 21 = a 21 /a 11, a 22 = a 22 + µ 21 a 12 0 a 32 a 33 b 3 µ 31 = a 31 /a 11, a 32 = a 32 + µ 31 a 12 a 11 a 12 a 13 b 1 0 a 22 a 23 b a 33 b 3 µ 32 = a 32 /a 22, a 33 = a 33 + µ 32 a 23

18 Gaussian Elimination continued Back Substitution: solution of the upper triangular system with the ordering of the unknowns reversed. x 3 = b 3 /a 33 x 2 = (b 2 a 23 x 3 )/a 22 x 1 = (b 1 a 12 x 2 a 13 x 3 )/a 11 The diagonal elements of U, a 11, a 22, and a 33, are pivot elements and must be nonzero. Gaussian elimination applied to the augmented matrix is equivalent to computing an LU factorization of A, A = LU, where L is unit lower triangular and U is upper triangular, and solving Ax = LUx = b.

19 LU Factorization without Pivoting In order to prove the above assertion, we introduce some notation. An elementary row operation can be applied to a matrix A by applying the operation to the identity matrix I and then left-multiplying A by the resulting matrix. (A column operation is applied by right-multiplying.) For k = 1, 2,..., n 1, define M k as the elementary lower triangular matrix that introduces zeros below a kk. For the order-3 example we have Note that M 1 = µ µ M 1 A = a 11 a 12 a 13 0 a 22 a 23 0 a 32 a 33, M 2 = µ 32 1., and M 2 M 1 A = U.

20 LU Factorization without Pivoting continued We now have so that A = LU if and only if U = M n 1 M n 2 M 1 A L = (M n 1 M n 2 M 1 ) 1 = M 1 1 M 1 2 M 1 n 1. Note that a triangular matrix is nonsingular if and only if its diagonal elements are all nonzero, and hence L is invertible since it is the product of invertible matrices. Thus, A is invertible iff the pivots (elements of diag(u)) are nonzero. We need some additional notation: m T k = [0, 0,..., 0, µ k+1,k, µ k+2,k,..., µ n,k ] (k = 1,..., n 1). Then M k = I + m k e T k, where e k denotes column k of I.

21 LU Factorization without Pivoting continued For example 0 m 1 = µ 21 m 1 e1 T = µ 31 so that 0 µ 21 µ 31 M 1 = I + m 1 e1 T = [1 0 0] = µ µ µ µ Since M k adds multiples of row k to the following rows, its inverse must subtract the same multiples. Lemma 1: M 1 k = (I + m k e T k ) 1 = I m k e T k. proof M 1 k M k = (I m k e T k )(I + m ke T k ) = I + m ke T k m ke T k (m k e T k )(m ke T k ) = I m k(e T k m k)e T k = I..,

22 LU Factorization without Pivoting continued Lemma 2: M 1 j M 1 j+1 = (I m jej T )(I m j+1 ej+1 T ) = I m j ej T m j+1 ej+1 T. More generally, L = M 1 1 M 1 2 M 1 n 1 = I m 1e T 1 m 2 e T 2... m n 1 e T n 1. proof: e T j m k = 0 for k > j. In our example we have LU = L = µ µ 31 µ µ µ 31 µ 32 1 a 11 a 12 a 13 0 a 22 a a 33 = a 11 a 12 a 13 a 21 a 22 a 23 a 31 a 32 a 33.

23 LU Factorization without Pivoting continued Algorithm 1: overwrite A with L I + U. for k = 1:n-1 for i = k+1:n A(i,k) = A(i,k)/A(k,k); %negative multiplier for j = k+1:n A(i,j) = A(i,j) - A(i,k)*A(k,j); end end end Note that Matlab matrices are stored in column-major order, and the outer loop should therefore be on the column index j in order to maintain spatial locality (and thereby minimize cache misses and page faults). The loops should also be vectorized.

24 Operation Count Counting the divides as multiplies (or multiplies by prestored reciprocals of pivot elements), the number of multiplies is M = n 1 n k=1 i=k+1 n 1 n 1 [1 + (n k)] = (n + 1 k)(n k) = (j + 1)j, k=1 where we have substituted j = n k. Using j=1 we get n i=1 i 2 = 2n3 + 3n 2 + n 6 = n(2n + 1)(n + 1), 6 n 1 n 1 M = j + j 2 = j=1 j=1 (n 1)n 2 + (n 1)(2n 1)n 6 = n3 3 n 3.

25 Back Substitution Given the LU factorization, a solution is obtained from Ax = (LU)x = L(Ux) = b Ly = b for y = Ux. 1 Solve Ly = b unit lower triangular system 2 Solve Ux = y upper triangular system Note that Ly = b y = L 1 b = M n 1 M n 2 M 1 b; i.e,. solution of the lower triangular system is precisely the set of operations applied to b by forward elimination on the augmented matrix. Also, solution of Ux = y is the same back substitution operation used in the case of an augmented matrix. Hence the equivalence.

26 Back Substitution Algorithms Algorithm 2: overwrite b with L 1 b for k = 1:n-1 for i = k+1:n b(i) = b(i) - A(i,k)*b(k); end end The operation count is n 1 n 1 (n k) = j = k=1 j=1 (n 1)n 2 = n2 2 n 2.

27 Back Substitution Algorithms continued Algorithm 3: overwrite b with U 1 b b(n) = b(n)/a(n,n); for k = n-1:-1:1 t = 0; for j = k+1:n t = t + A(k,j)*b(j); end b(k) = (b(k)-t)/a(k,k); end The operation count is n (n k + 1) = n + k=1 (n 1)n 2 so that the total number of multiplies is = n2 2 + n 2, (n 2 /2 n/2) + (n 2 /2 + n/2) = n 2.

28 Back Substitution Algorithms continued The above algorithm with outer loop on column index (not so easily derived) is as follows: Algorithm 3 : overwrite b with U 1 b for j = n:-1:2 b(j) = b(j)/a(j,j); for i = 1:j-1 b(i) = b(i) - A(i,j)*b(j); end end b(1) = b(1)/a(1,1);

29 Stability Consider applying Gaussian elimination to the following matrix: [ ] 0 1 A =. 1 0 The zero pivot element leads to failure with multiplier µ 21 = 1/0 = even though the matrix is perfectly conditioned, and Ax = b has solution x 1 = b 2, x 2 = b 1. Since a zero pivot leads to failure, a small pivot must lead to numerical instability. In fact, small pivots lead to cancellation error caused by large multipliers (relative to A and b ). Consider the order-2 example: a 11 a 12 b 1 a 11 a 12 b 1 a 21 a 22 b 2 0 a 22 + µa 12 b 2 + µb 1 for µ = a 21 /a 11.

30 Stability continued Back substitution gives A large value µ implies x 2 = b 2 + µb 1 a 22 + µa 12, x 1 = b 1 a 12 x 2 a 11. fl(x 2 ) b 1 /a 12 fl(a 12 x 2 ) b 1 implying cancellation error in fl(x 1 ). In general, x n is computed accurately, but computation of the remaining solution components involves cancellation error. Partial Pivoting: row interchanges chosen to maximize pivot element magnitudes. Only full pivoting, involving both row and column interchanges (and requiring reordering of solution components), can guarantee stability, but partial pivoting is almost always sufficient in practice (unless the matrix is poorly scaled).

31 Orthogonal Matrix Defn An orthogonal matrix is a square matrix Q with orthonormal columns: q T i q j = δ ij, (i, j = 1,..., n) for Q = [q 1 q 2 q n ]; i.e., Q T Q = I. Equivalently, an orthogonal matrix is one whose transpose is its inverse. Note that AB = I does not imply that B = A 1 unless A and B are square. The complex analogue of an orthogonal matrix is a unitary matrix. An orthogonal matrix Q has the following properties. Q T is orthogonal, and thus Q has orthonormal rows. Inner products are preserved: (Qx) T (Qy) = x T y x, y R n. Euclidean norms are preserved: Qx 2 = x 2 x R n. The eigenvalues of Q have magnitude 1. Q is perfectly conditioned: κ 2 (Q) = ρ(q T Q) ρ(qq T ) = 1. det(q) = 1 (rotation) or det(q) = 1 (reflection).

32 Pivoting Defn An elementary permutation matrix P i is obtained by interchanging a pair of rows of the identity matrix. P i is symmetric and orthogonal so that P T i P i = P 2 i = I. A permutation matrix P is a product of elementary permutations. This is analogous to a permutation of the integers 1:n being a product of transpositions. P is not symmetric, but it is orthogonal permuting rows of an orthogonal matrix leaves them orthonormal. An LU factorization with partial pivoting produces LU = PA for P = P n 1 P n 2 P 2 P 1, where P k is the elementary permutation that interchanges row k with row m for some m k chosen to maximize the magnitude of the pivot element.

33 LU with Partial Pivoting: Example 1 A = , b = x = P 1 = , M 1 = P 2 = , M 2 =

34 LU with Partial Pivoting: Example 1 continued By storing the negative multipliers below the diagonal, we have overwritten the array with L I + U, from which we can read off L and U, and compute P = P 2 P 1 : L = , U = , P = We can now compute products LU and PA to verify that LU = = PA

35 LU with Partial Pivoting: Example 2 A = , b = Omitting the steps, the factors are L =.5 1 0, U = / / , P = We then obtain the solution by back substitution; i.e., we solve Ux = L 1 Pb, where L 1 Pb was computed by treating the augmented matrix: U = /3 x 1 x 2 x 3 = /3 x =

36 LU with Partial Pivoting Theorem The algorithm produces L, U, and P such that LU = PA. proof We need an expression for the unit lower triangular matrix L obtained by storing negative multipliers below the diagonal and applying each permutation to the entire row (including multipliers). We have negative multipliers in M 1 k, and they are interchanged (along with the rest of their rows) in P k+1 M 1 k. The trick is to then right-multiply by P k+1 in order to restore the correct structure in P k+1 M k P k+1. In the order-3 case, where P 2 interchanges rows 2 and 3, we have L = (P 2 M 1 1 P 2)M 1 2 = and U = M 2 P 2 M 1 P 1 A, so that µ µ 21 µ 32 1, LU = P 2 M 1 1 P 2M 1 2 M 2P 2 M 1 P 1 A = P 2 P 1 A = PA.

37 LU with Partial Pivoting continued The order-5 example is as follows: L = P 4 {P 3 [(P 2 M1 1 2)M2 1 3}M3 1 4M4 1 U = M 4 P 4 M 3 P 3 M 2 P 2 M 1 P 1 A, LU = P 4 P 3 P 2 [M1 1 2M2 1 3M3 1 4M4 1 4P 4 M 3 P 3 M 2 P 2 M 1 ]P 1 A = P 4 P 3 P 2 P 1 A = PA. The conclusion follows by finite induction. Note that an LU factorization is always possible, and L and P are invertible. U is invertible if and only if A is invertible. A zero pivot is encountered at step k iff U already has all zeros on and below the diagonal in column k. The multipliers in column k are taken to be zeros in this case.

38 LU with Partial Pivoting continued The above remarks suggest a test for singularity: perform Gaussian elimination with partial pivoting, and look for a zero pivot. This is a bad test. A zero pivot can be encountered in a nonsingular (but very ill-conditioned) matrix, and all nonzero pivots can be obtained with a singular matrix. In fact, a singular matrix might not even have a small pivot element. The only meaningful computational test is for singular to machine precision based on a condition number estimate.

39 LU with Partial Pivoting continued Back substitution now requires three steps. Ax = b iff PAx = Pb iff LUx = Pb. 1 Overwrite array b with b = Pb 2 Solve Ly = b for y, overwriting array b with y 3 Solve Ux = y for x, overwriting array b with x In Example 1 we have 7 b = 4, b = , y = , x =

40 LU with Partial Pivoting continued The algorithms are easily modified for pivoting. At each step of forward elimination, in addition to interchanging rows, a record of the interchange must be saved for use in back substitution. We initialize an index vector p = 1:n, and apply each interchange to p. Then Pb = b(p). The operation counts are not altered. An alternative method of storing permutations is to let p(k) be the index of the row to be interchanged with row k for k = 1,..., n 1. The last entry p(n) can be used to keep track of the sign of det(p) = ( 1) µ, where µ is the number of row interchanges, each of which changes the sign of det(a). Then det(a) = det(p T LU) = det(p) det(l) det(u) = ( 1) µ det(u), where, using an expression that avoids overflow and underflow, [ n n ] det(u) = u kk = exp log( u kk ). k=1 k=1

41 Vector Norms Defn A vector norm on R n is a function : R n R such that 1 x 0 and x = 0 iff x = 0 x R n 2 αx = α x α R, x R n 3 x + y x + y x, y R n Note that 0 = 0 by property (2). The most important and commonly employed norms are the p-norms with p = 1, p = 2, and p =. ( n ) (1/p) x p x i p, p = 1, 2,... i=1 x 1 = n i=1 x i cheap to compute n x 2 = i=1 x i 2 inner product Euclidean norm associated with an x = max 1 i n x i cheap; uniform approximation

42 Vector Norms continued Theorem x = lim p x p. proof Let x m = x max 1 i n x i. Then ( n ) (1/p) x i p = i=1 ( n i=1 ( n = x m i=1 since 1 x i /x m p n. (1/p) ( x i p xm p xm) p = xm p ) p (1/p) x i x m x m n i=1 ) p (1/p) x i x m The proof that p satisfies the triangle inequality is difficult in the general case, but not for the specific values p = 1, 2,.

43 Vector Norms continued The unit spheres are S p = {x R n : x p = 1}. These are easily sketched for n = 2 and p = 1, 2,. The shapes are diamond, circle, and square, respectively. Theorem All norms on R n are equivalent in the following sense. For norms M and N on R n, there exist constants c 1, c 2 > 0 such that c 1 M(x) N(x) c 2 M(x) x R n. Defn A sequence of vectors x 1, x 2,... converges to x R n iff x x k 0 as k. By the above theorem, the choice of norm is irrelevant.

44 Matrix Norms Defn A matrix norm on R n n is defined by the same three properties characterizing a vector norm along with the additional property 4 AB A B A, B R n n. A matrix norm is compatible with a vector norm v if Ax v A x v x R n, A R n n. The Frobenius norm n A F = i,j=1 a 2 ij (1/2) is compatible with 2.

45 Operator Norms Given a vector norm v, there is a corresponding operator norm induced by v : Ax v A = sup = sup Ax v x 0 x v 0< x 1 = max x 0 Ax v x v. The operator norm is clearly compatible with v, and is easily shown to satisfy the four defining properties. It is a measure of the extent to which a matrix stretches the unit sphere. The important examples are those induced by p-norms. A 1 = max 1 j n n i=1 a ij = max 1 j n a j 1 for A = (a 1 a 2... a n ). A 2 = ρ(a T A) for spectral radius ρ A = max 1 i n n j=1 a ij maximum absolute row sum

46 Operator Norms continued Theorem Let denote any operator norm on R n n. Then ρ A. Also, given ɛ > 0, there exists an operator norm ɛ such that A ɛ ρ(a) + ɛ. Corollary ρ(a) < 1 iff A < 1 for some operator norm. Theorem A m 0 as m iff ρ(a) < 1. Theorem ρ(a) < 1 (I A) 1 = I + A + A converges. Theorem If A < 1 for some operator norm, then (I A) 1 exists and (I A) 1 1/(1 A ). proof I = (I A)(I A) 1 = (I A) 1 A(I A) 1 (I A) 1 = I + A(I A) 1 (I A) 1 I + A(I A) A (I A) 1 (I A) 1 (1 A ) 1.

47 Condition Number Defn The condition number of a nonsingular matrix A is κ(a) = A A 1 for some operator norm. The condition number of a singular (square) matrix is κ(a) =. Note that A 1 = max y 0 A 1 y y = max x 0 [ x Ax = 1/ min x 0 ] Ax x (using y = Ax). Hence [ ] Ax κ(a) = max / x 0 x [ min x 0 ] [ ] [ ] Ax = max x Ax / min Ax 1. x =1 x =1 The condition number of a matrix is thus the extent to which it skews the unit sphere. In the case of the Euclidean norm, the unit sphere is mapped to an ellipsoid, and the condition number is the ratio of largest to smallest half-axis length.

48 Condition Number continued Theorem A bounded linear operator A (such as an order-n matrix) is continuous (and thus maps the unit sphere to a cts. surface). proof Given ɛ > 0, let δ = ɛ/ A. Then x y < δ Ax Ay = A(x y) A x y < A δ = ɛ Theorem Suppose Ax = b and a perturbation b in the data leads to solution x + x; i.e., A(x + x) = b + b. Then the relative change in the solution is bounded by κ(a) times the relative change in the data: x / x κ(a) b / b. proof Ax = b and A(x + x) = b + b A x = b x = A 1 b. Hence x A 1 b and b = Ax A x 1/ x A / b. Thus, x x A 1 b A / b = κ(a) b b

49 Condition Number continued Following are some examples. The identity is perfectly conditioned: κ(i ) = 1. Orthogonal matrices are perfectly conditioned: Q T Q = I κ 2 (Q) = Q 2 Q T 2 = ρ(q T Q) ρ(qq T ) = 1. D = diag(d 1, d 2,..., d n ) κ p (D) = max d i / min d i for p = 1, 2,. If A is symmetric and positive definite, its condition number is the ratio of largest to smallest eigenvalue: κ 2 (A) = λ max /λ min.