AN ITERATION In part as motivation, we consider an iteration method for solving a system of linear equations which has the form x Ax = b In this, A is an n n matrix and b R n.systemsof this form arise in a number of applications. One way in which such systems are solved is as follows. First rewrite the system as x = b + Ax () Choose and initial guess x (0). Then define a sequence of iterates x (),x (2),... by x (k+) = b + Ax (k), k =0,, 2,... (2) Do these iterates converge to x? Subtract (2) from (), obtaining x x (k+) = Ax Ax (k) = A ( x x (k))
By recursion, x x (k+) = A ( x x (k)), k 0 x x (k) = A k ( x x (0)), k 0 This error depends on both A k and the initial error x x (0), x x (k) A k x x (0) Generally we want a method which works for all possible initial guesses, in part because rounding error will eventually bring a randomness into the error. Consequently, in order to determine convergence of x (k) to x, we need to know whether A k 0ask.
THEOREM Let A be an n n matrix. Then if and only if A k 0 as k r σ (A) < PROOF: There are a number of ways to prove this; and in the book, I use the Jordan canonical form for A. To simplify things for this class presentation, we make the simplifying assumption that the Jordan canonical form for A is a diagonal matrix: there is a nonsingular matrix P for which P AP = D =diag[λ,..., λ n ] We can rewrite this as A = PDP
Then A k = ( PDP ) k = ( PDP )( PDP ) ( PDP ) = PD k P Then A k 0 if and only if D k 0. For a diagonal matrix D =diag[λ,..., λ n ], D k =diag [ λ k,..., λk n Thus D k 0 if and only if λ k i 0 as k 0 for i =, 2,..., n. This is true if and only if r σ (A) max i n λ i < COROLLARY ] Suppose A <. Then A k 0ask, because r σ (A) A <
SOLVABILITY OF A LINEAR SYSTEM Return to the linear system (I A) x = x Ax = b Is this system uniquely solvable? Equivalently, is I A an invertible matrix? The answer to this question turns out to be of fundamental importance in the numerical analysis of linear algebra problems. THEOREM Let A be an n n matrix. If r σ (A) <, then (I A) exists; and moreover, it can be expressed as the convergent infinite series (I A) = I + A + A 2 + A 3 + Conversely, if this series is convergent, then r σ (A) <
PROOF: Assume r σ (A) <. Then I A must be nonsingular. Otherwise there would be a vector x 0 for which (I A) x = 0 Ax = x and this says is an eigenvalue of A, contrary to assumption. To prove the series converges, look at its partial sums B k = I + A + A 2 + A 3 + + A k for k =, 2, 3,... Then (I A)B k = (I A) ( I + A + + A k) = I A k+ B k = (I A) ( I A k+) Then the partial sums B k converge to (I A) because A k+ 0.
If instead we assume the series I + A + A 2 + A 3 + is convergent, then the general term A k must converge to the zero matrix 0 as k. That in turn implies This result about r σ (A) < (I A) = I + A + A 2 + A 3 + is called the Geometric Series Theorem. Note its resemblance to the well-known geometric series z =+z + z2 + z 3 +, z < This in not an accidental resemblance. We can do similar things with series such as that for e z, leading to e A =+A + 2! A2 + 3! A3 +
THEOREM Let A be an n n matrix with A <. (I A) exists, with Then Moreover, (I A) = I + A + A 2 + A 3 + (I A) A PROOF: The only thing to be proven is the bound. Recall that Then B k = I + A + A 2 + A 3 + + A k = (I A) ( I A k+) B k I + A + A 2 + A 3 + + A k + A + A 2 + + A k = A k+ A A
Also, B k (I A) Bk = A (I A) B k (I A) A k+ (I A) A k+ 0 Combining these results proves the bound (I A) A
AN APPROXIMATION RESULT Suppose we are considering a system Ax = b where A is some nonsingular n n matrix. What happens to the solvability of this system if we change the matrix A by a small amount to a new square matrix B? And what is the relation of the solution of to the original solution x? Bz = b
THEOREM Let A and B be two n n matrices, and suppose A is nonsingular. Moreover, assume A B < A Then the matrix B is also nonsingular, with B A A A B For the solutions of Ax = b and Bz = b, x z B A B x This theorem tells us that if A is invertible, then all nearby matrices B are also invertible.
PROOF: Write B = A (A B) = A [ I A (A B) ] Examine the matrix I A (A B). A (A B) A A B < Then by the Geometric Series Theorem, [ I A (A B) ] exists, and [ I A (A B) ] A (A B) A A B
Returning to B = A [ I A (A B) ] we have B is the product of invertible matrices, and therefore it is itself invertible: B = [ I A (A B) ] A B [ I A (A B) ] A The bound for B follows from this formula. For the equations Ax = b and Bz = b, write x z = A b B b = ( A B ) b = B (B A) A b = B (B A) x x z B A B x
EXAMPLE Let B = c 0 0 0 c 0 0 0 c 0 0 0 c 0 0 0 c Approximate this by the matrix A = ci. Then B A = Using the row norm, 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 B A =2
Also, A = A c I, = The condition A B < A becomes c. 2 < c Then B is invertible if this is true; and then the bound B A A A B becomes B c 2 c = c 2 (3)
This also tells us something about the eigenvalues of B. LetBx = λx for some x 0. Then λ B = c +2 (4) Multiply both sides of Bx = λx by both B and λ to get λ x = B x Therefore the eigenvalues of B are the reciprocals of those of B. Recall that Then (3) implies With (4), r σ (B ) B λ c 2 λ c 2 c 2 λ c +2 for all eigenvalues λ of the matrix B.