Numerical methods for solving linear systems

Chapter 2 Numerical methods for solving linear systems Let A C n n be a nonsingular matrix We want to solve the linear system Ax = b by (a) Direct methods (finite steps); Iterative methods (convergence) (See Chapter 4) 2 Elementary matrices x ȳ x ȳ n Let X = K n and x, y X Then y x K, xy = The eigenvalues x n ȳ x n ȳ n of xy are {0,, 0, y x}, since rank(xy ) = by (xy )z = (y z)x and (xy )x = (y x)x Definition 2 A matrix of the form is called an elementary matrix I αxy (α K, x, y K n ) (2) The eigenvalues of (I αxy ) are {,,,, αy x} Compute If αy x 0 and letβ = (I αxy )(I βxy ) = I (α + β αβy x)xy (22) α αy x, then α + β αβy x = 0 We have (I αxy ) = (I βxy ), α + β = y x (23) Example 2 Let x K n, and x x = Let H = {z : z x = 0} and Q = I 2xx (Q = Q, Q = Q) Then Q reflects each vector with respect to the hyperplane H Let y = αx + w, w H Then, we have Qy = αqx + Qw = αx + w 2(x w)x = αx + w

24 Chapter 2 Numerical methods for solving linear systems Example 22 Let y = e i = the i-th column of unit matrix and x = l i = [0,, 0, l i+,i,, l n,i ] T Then, I + l i e T i = (24) l i+,i l n,i Since e T i l i = 0, we have From the equality (I + l i e T i ) = (I l i e T i ) (25) follows that (I + l e T )(I + l 2e T 2 ) = I + l e T + l 2e T 2 + l (e T l 2)e T 2 = I + l e T + l 2e T 2 (I + l e T ) (I + l ie T i ) (I + l n e T n ) = I + l e T + l 2e T 2 + + l n e T n l = 2 0 (26) l n l n,n Theorem 2 A lower triangular with on the diagonal can be written as the product of n elementary matrices of the form (24) Remark 2 (I + l e T + + l n e T n ) = (l l n e T n ) (I l e T ) which can not be simplified as in (26) 22 LR-factorization Definition 22 Given A C n n, a lower triangular matrix L and an upper triangular matrix R If A = LR, then the product LR is called a LR-factorization (or LRdecomposition) of A Basic problem: Given b 0, b K n Find a vector l = [0, l 2,, l n ] T and c K such that Solution: (I l e T )b = ce { b = c, b i l i b = 0, i = 2,, n { b = 0, it has no solution (since b 0), b 0, then c = b, l i = b i /b, i = 2,, n

22 LR-factorization 25 Construction of LR-factorization: Let A = A (0) = [a (0) a (0) n ] Apply basic problem to a (0) : If a (0) 0, then there exists L = I l e T such that (I l e T )a (0) = a (0) e Thus The i-th step: A () = L A (0) = [L a (0) L a (0) n ] = a (0) a (0) 2 a (0) n 0 a () 22 a () 2n (22) 0 a () n2 a () nn A (i) = L i A (i ) = L i L i L A (0) a (0) a (0) n 0 a () 22 a () 2n 0 = a (i ) ii a (i ) in 0 a (i) i+,i+ a (i) i+,n 0 0 a (i) n,i+ a (i) nn (222) If a (i ) ii 0, for i =,, n, then the method is executable and we have that A (n ) = L n L A (0) = R (223) is an upper triangular matrix Thus, A = LR Explicit representation of L: L i = I l i e T i, L i = I + l i e T i L = L L n = (I + l e T ) (I + l n e T n ) = I + l e T + + l n e T n (by (26)) Theorem 22 Let A be nonsingular Then A has an LR-factorization (A=LR) if and only if k i := det(a i ) 0, where A i is the leading principal matrix of A, ie, a a i A i =, a i a ii for i =,, n Proof: (Necessity ): Since A = LR, we have a a i l = O r r i O r ii a i a ii l i l ii

26 Chapter 2 Numerical methods for solving linear systems From det(a) 0 follows that det(l) 0 and det(r) 0 Thus, l jj 0 and r jj 0, for j =,, n Hence k i = l l ii r r ii 0 (Sufficiency ): From (222) we have A (0) = (L L i )A (i) Consider the (i + )-th leading principle determinant From (223) we have a a i,i+ a i+ a i+,i+ = 0 l 2 l i+, l i+,i a (0) a (0) 2 a () 22 a (i ) ii a (i ) i,i+ 0 a (i) i+,i+ 0 Therefore, the LR- Thus, k i = a (0) a() 22 a(i) i+,i+ factorization of A exists 0 which implies a(i) i+,i+ Theorem 222 If a nonsingular matrix A has an LR-factorization with A = LR and l = = l nn =, then the factorization is unique Proof: Let A = L R = L 2 R 2 Then L 2 L = R 2 R = I Corollary 22 If a nonsingular matrix A has an LR-factorization with A = LDR, where D is diagonal, L and R T are unit lower triangular (with one on the diagonal) if and only if k i 0 Theorem 223 Let A be a nonsingular matrix Then there exists a permutation P, such that P A has an LR-factorization (Proof): By construction! Consider (222): There is a permutation P i, which interchanges the i-th row with a row of index large than i, such that 0 a (i ) ii ( P i A (i ) ) This procedure is executable, for i =,, n So we have L n P n L i P i L P A (0) = R (224) Let P be a permutation which affects only elements i +,, n It holds P (I l i e T i )P = I (P l i )e T i = I l i e T i = L i, (e T i P = e T i ) where L i is lower triangular Hence we have Now write all P i in (224) to the right as P L i = L i P (225) L n Ln 2 L P n P A (0) = R Then we have P A = LR with L = L n Ln 2 L and P = P n P

23 Gaussian elimination 27 23 Gaussian elimination 23 Practical implementation Given a linear system Ax = b (23) with A nonsingular We first assume that A has an LR-factorization ie, A = LR Thus LRx = b We then (i) solve Ly = b; (ii) solve Rx = y These imply that LRx = Ly = b From (224), we have L n L 2 L (A b) = (R L b) Algorithm 23 (without permutation) For k =,, n, if a kk = 0 then stop ( ); else ω j := a kj (j = k +,, n); for i = k +,, n, η := a ik /a kk, a ik := η; for j = k +,, n, a ij := a ij ηω j, b j := b j ηb k For x: (back substitution!) x n = b n /a nn ; for i = n, n 2,,, x i = (b i n j=i+ a ijx j )/a ii Cost of computation (one multiplication + one addition one flop): (i) LR-factorization: n 3 /3 n/3 flops; (ii) Computation of y: n(n )/2 flops; (iii) Computation of x: n(n + )/2 flops For A : 4/3n 3 n 3 /3 + kn 2 (k = n linear systems) Pivoting: (a) Partial pivoting; (b) Complete pivoting From (222), we have A (k ) = a (0) n 0 a (k 2) k,k a (k 2) k,n 0 a (k ) kk a (k ) kn 0 0 a (k ) nk a (k ) nn a (0)

28 Chapter 2 Numerical methods for solving linear systems For (a): Find a p {k,, n} such that a pk = max k i n a ik (r k = p) (232) swap a kj, b k and a pj, b p respectively, (j =,, n) Replacing ( ) in Algorithm 23 by (232), we have a new factorization of A with partial pivoting, ie, P A = LR (by Theorem 22) and l ij for i, j =,, n For solving linear system Ax = b, we use P Ax = P b L(Rx) = P T b b It needs extra n(n )/2 comparisons For (b): Find p, q {k,, n} such that a pq max a ij, (r k := p, c k := q) k i,j n swap a kj, b k and a pj, b p respectivly, (j = k,, n), swap a ik and a iq (i =,, n) (233) Replacing ( ) in Algorithm 23 by (233), we also have a new factorization of A with complete pivoting, ie, P AΠ = LR (by Theorem 22) and l ij, for i, j =,, n For solving linear system Ax = b, we use P AΠ(Π T x) = P b LR x = b x = Π x It needs n 3 /3 comparisons [ ] 0 4 Example 23 Let A = be in three decimal-digit floating point arithmetic Then κ(a) = A A 4 A is well-conditioned Without pivoting: [ ] 0 L = fl(/0 4, fl(/0 4 ) = 0 4, ) [ ] 0 4 R = 0 fl( 0 4, fl( 0 4 ) = 0 4 ) [ ] [ ] [ ] [ ] 0 0 4 0 4 0 4 LR = 0 4 0 0 4 = = A 0 [ ] Here a 22 entirely lost from computation It is numerically unstable Let Ax = [ ] [ ] 2 Then x But Ly = solves y 2 = and y 2 = fl(2 0 4 ) = 0 4, Rˆx = y solves ˆx 2 = fl(( 0 4 )/( 0 4 )) =, ˆx = fl(( )/0 4 ) = 0 We have an erroneous solution with cond(l), cond(r) 0 8 Partial pivoting: L = R = L and R are both well-conditioned [ ] [ 0 fl(0 4 = /) [ ] 0 fl( 0 4 = ) 0 0 4 [ 0 ] ],

23 Gaussian elimination 29 232 LDR- and LL T -factorizations Let A = LDR as in Corollary 22 Algorithm 232 (Crout s factorization or compact method) For k =,, n, for p =, 2,, k, r p := d p a pk, ω p := a kp d p, d k := a kk k p= a kpr p, if d k = 0, then stop; else for i = k +,, n, a ik := (a ik k p= a ipr p )/d k, a ki := (a ki k p= ω pa pi )/d k Cost: n 3 /3 flops With partial pivoting: see Wilkinson EVP pp225- Advantage: One can use double precision for inner product Theorem 23 If A is nonsingular, real and symmetric, then A has a unique LDL T - factorization, where D is diagonal and L is a unit lower triangular matrix (with one on the diagonal) Proof: A = LDR = A T = R T DL T It implies L = R T Theorem 232 If A is symmetric and positive definite, then there exists a lower triangular G R n n with positive diagonal elements such that A = GG T Proof: A is symmetric positive definite x T Ax 0, for all nonzero vector x R n n k i 0, for i =,, n, all eigenvalues of A are positive From Corollary 22 and Theorem 23 we have A = LDL T From L AL T = D follows that d k = (e T k L )A(L T e k ) > 0 Thus, G = Ldiag{d /2,, d /2 n } is real, and then A = GG T Algorithm 233 (Cholesky factorization) Let A be symmetric positive definite To find a lower triangular matrix G such that A = GG T For k =, 2,, n, a kk := (a kk k p= a2 kp )/2 ; for i = k +,, n, a ik = (a ik k p= a ipa kp )/a kk Cost: n 3 /6 flops Remark 23 For solving symmetric, indefinite systems: See Golub/ Van Loan Matrix Computation pp 59-68 P AP T = LDL T, D is or 2 2 block-diagonal matrix, P is a permutation and L is lower triangular with one on the diagonal

30 Chapter 2 Numerical methods for solving linear systems 233 Error estimation for linear systems Consider the linear system and the perturbed linear system Ax = b, (234) (A + δa)(x + δx) = b + δb, (235) where δa and δb are errors of measure or round-off in factorization Definition 23 Let be an operator norm and A be nonsingular Then κ κ(a) = A A is a condition number of A corresponding to Theorem 233 (Forward error bound) Let x be the solution of the (234) and x+δx be the solution of the perturbed linear system (235) If δa A <, then ( δx x κ δa κ δa A + δb ) (236) b A Proof: From (235) we have Thus, (A + δa)δx + Ax + δax = b + δb δx = (A + δa) [(δa)x δb] (237) Here, Corollary 27 implies that (A + δa) exists Now, (A + δa) = (I + A δa) A A A δa On the other hand, b = Ax implies b A x So, x A b (238) From (237) follows that δx inequality (236) is proved A ( δa x + δb ) By using (238), the A δa Remark 232 If κ(a) is large, then A (for the linear system Ax = b) is called illconditioned, else well-conditioned 234 Error analysis for Gaussian algorithm A computer in characterized by four integers: (a) the machine base β; (b) the precision t; (c) the underflow limit L; (d) the overflow limit U Define the set of floating point numbers F = {f = ±0d d 2 d t β e 0 d i < β, d 0, L e U} {0} (239)

23 Gaussian elimination 3 Let G = {x R m x M} {0}, where m = β L and M = β U ( β t ) are the minimal and maximal numbers of F \ {0} in absolute value, respectively We define an operator fl : G F by One can show that fl satisfies fl(x) = the nearest c F to x by rounding arithmetic fl(x) = x( + ε), ε eps, (230) where eps = 2 β t (If β = 2, then eps = 2 t ) It follows that or where ε eps and = +,,, / fl(a b) = (a b)( + ε) fl(a b) = (a b)/( + ε), Algorithm 234 Given x, y R n The following algorithm computes x T y and stores the result in s s = 0, for k =,, n, s = s + x k y k Theorem 234 If n2 t 00, then n fl( x k y k ) = k= n x k y k [ + 0(n + 2 k)θ k 2 t ], θ k k= Proof: Let s p = fl( p k= x ky k ) be the partial sum in Algorithm 234 Then s = x y ( + δ ) with δ eps and for p = 2,, n, s p = fl[s p + fl(x p y p )] = [s p + x p y p ( + δ p )]( + ε p ) with δ p, ε p eps Therefore fl(x T y) = s n = n x k y k ( + γ k ), k= where ( + γ k ) = ( + δ k ) n j=k ( + ε j), and ε 0 Thus, n fl( x k y k ) = k= n x k y k [ + 0(n + 2 k)θ k 2 t ] (23) k= The result follows immediately from the following useful Lemma

32 Chapter 2 Numerical methods for solving linear systems Lemma 235 If ( + α) = n k= ( + α k), where α k 2 t and n2 t 00, then n ( + α k ) = + 0nθ2 t with θ k= Proof: From assumption it is easily seen that ( 2 t ) n n ( + α k ) ( + 2 t ) n (232) Expanding the Taylor expression of ( x) n as < x <, we get Hence ( x) n = nx + k= Now, we estimate the upper bound of ( + 2 t ) n : If 0 x 00, then n(n ) ( θx) n 2 x 2 nx 2 ( 2 t ) n n2 t (233) e x = + x + x2 2! + x3 3! + = + x + x 2 x( + x 3 + 2x2 + ) 4! + x e x + x + 00x 2 ex + 0x (234) (Here, we use the fact e 00 < 2 to the last inequality) Let x = 2 t Then the left inequality of (234) implies ( + 2 t ) n e 2 t n (235) Let x = 2 t n Then the second inequality of (234) implies From (235) and (236) we have e 2 tn + 0n2 t (236) ( + 2 t ) n + 0n2 t Let the exact LR-factorization of A be L and R (A = LR) and let L, R be the LR-factorization of A by using Gaussian Algorithm (without pivoting) There are two possibilities: (i) (ii) Forward error analysis: Estimate L L and R R Backward error analysis: Let L R be the exact LR-factorization of a perturbed matrix à = A + F Then F will be estimated, ie, F?

23 Gaussian elimination 33 235 Àpriori error estimate for backward error bound of LRfactorization From (222) we have a (k+) ij = A (k+) = L k A (k), for k =, 2,, n (A () = A) Denote the entries of A (k) by a (k) ij and let l ik = fl(a (k) ik /a(k) kk ), i k + From (222) we know that 0; for i k +, j = k fl(a (k) ij a (k) fl(l ik a (k) kj )); for i k +, j k + (237) ij ; otherwise From (230) we have l ik = (a (k) ik /a(k) kk )( + δ ik) with δ ik 2 t Then a (k) ik Let a (k) ik δ ik ε (k) From (230) we also have ik with δ ij, δ ij 2 t Then a (k+) ij = a (k) ij l ika (k) kk + a(k) ij δ ik = 0, for i k + (238) a (k+) ij = fl(a (k) ij l ik a (k) kj = (a (k) ij Let ε (k) ij l ik a (k) kj δ ij + a (k+) ij (237), (238) and (2320) we obtain where a (k+) ij = ε (k) ij = a (k) a (k) ij a (k) ij a (k) ij fl(l ik a (k) kj )) (239) (l ik a (k) kj ( + δ ij)))/( + δ ij ) l ika (k) kj δ ij + a (k+) ij δ ij, for i, j k + (2320) δ ij which is the computational error of a (k) ij in A (k+) From l ik a (k) kk + ε(k) ij ; for i k +, j = k l ik a (k) kj + ε(k) ij ; for i k +, j k + (232) + ε (k) ij ; otherwise, ij δ ij; for i k +, j = k, l ik a (k) kj δ ij a (k+) ij δ ij ; for i k +, j k + 0; otherwise Let E (k) be the error matrix with entries ε (k) ij Then (232) can be written as where (2322) A (k+) = A (k) M k A (k) + E (k), (2323) 0 M k = 0 l k+,k l n,k 0 (2324)

34 Chapter 2 Numerical methods for solving linear systems For k =, 2, n, we add the n equations in (2323) together and get M A () + M 2 A (2) + + M n A (n ) + A (n) = A () + E () + + E (n ) From (237) we know that the k-th row of A (k) is equal to the k-th row of A (k+),, A (n), respectively and from (2324) we also have Thus, Then where M k A (k) = M k A (n) = M k R (M + M 2 + + M n + I) R = A () + E () + + E (n ) L R = A + E, (2325) l 2 O L = and E = E() + + E (n ) (2326) l n l n,n Now we assume that the partial pivotings in Gaussian Elimination are already arranged such that pivot element a (k) kk has the maximal absolute value So, we have l ik Let ρ = max i,j,k a(k) ij / A (2327) Then From (2322) and (2328) follows that Therefore, ε (k) From (2326) we get ij ρ A a (k) ij ρ A (2328) 2 t ; for i k +, j = k, 2 t ; for i k +, j k +, 0; otherwise (2329) 0 0 0 0 E (k) ρ A 2 t 0 2 2 (2330) 0 2 2 E ρ A 2 t Hence we have the following theorem 0 0 0 0 0 2 2 2 2 3 4 4 4 3 5 2n 4 2n 4 3 5 2n 3 2n 2 (233)

23 Gaussian elimination 35 Theorem 236 The LR-factorization L and R of A using Gaussian Elimination with partial pivoting satisfies L R = A + E, where Proof: E n 2 ρ A 2 t (2332) n E ρ A 2 t ( (2j ) ) < n 2 ρ A 2 t j= Now we shall solve the linear system Ax = b by using the factorization L and R, ie, Ly = b and Rx = y For Ly = b: From Algorithm 23 we have y = fl(b /l ), ( ) li y l i2 y 2 l i,i y i + b i y i = fl, (2333) l ii for i = 2, 3,, n From (230) we have y = b /l ( + δ ), with δ 2 t y i = fl( fl( l iy l i2 y 2 l i,i y i )+b i l ii (+δ ii ) ) (2334) = fl( l iy l i2 y 2 l i,i y i )+b i l ii (+δ ii )(+δ ii ), with δ ii, δ ii 2 t Applying Theorem 234 we get where fl( l i y l i2 y 2 l i,i y i ) = l i ( + δ i )y l i,i ( + δ i,i )y i, So, (2334) can be written as δ i (i )0 2 t ; for i = 2, 3,, n, { i = 2, 3,, n, δ ij (i + j)0 2 t ; for j = 2, 3,, i l ( + δ )y = b, l i ( + δ i )y + + l i,i ( + δ i,i )y i + l ii ( + δ ii )( + δ ii )y i = b i, for i = 2, 3,, n (2335) (2336) or (L + δl)y = b (2337)

36 Chapter 2 Numerical methods for solving linear systems From (2335) (2336) and (2337) follow that l 0 l 2 2 l 22 δl 0 2 t 2 l 3 2 l 32 2 l 33 3 l 4 3 l 42 2 l 43 (n ) l n (n ) l n2 (n 2) l n3 2 l n,n 2 l nn (2338) This implies, δl n(n + ) 2 0 2 t max l ij i,j n(n + ) 2 0 2 t (2339) Theorem 237 For lower triangular linear system Ly = b, if y is the exact solution of (L + δl)y = b, then δl satisfies (2338) and (2339) Applying Theorem 237 to the linear system Ly = b and Rx = y, respectively, the solution x satisfies ( L + δ L)( R + δ R)x = b or ( L R + (δ L) R + L(δ R) + (δ L)(δ R))x = b (2340) Since L R = A + E, substituting this equation into (2340) we get The entries of L and R satisfy [A + E + (δ L) R + L(δ R) + (δ L)(δ R)]x = b (234) l ij, and r ij ρ A Therefore, we get L n, R nρ A, δ L n(n+) 2 0 2 t, δ R n(n+) 2 0ρ2 t In practical implementation we usually have n 2 2 t << So it holds (2342) Let Then, (2332) and (2342) we get δ L δ R n 2 ρ A 2 t δa = E + (δ L) R + L(δ R) + (δ L)(δ R) (2343) δa E + δ L R + L δ R + δ L δ R 0(n 3 + 3n 2 )ρ A 2 t (2344)

23 Gaussian elimination 37 Theorem 238 For a linear system Ax = b the solution x computed by Gaussian Elimination with partial pivoting is the exact solution of the equation (A + δa)x = b and δa satisfies (2343) and (2344) Remark 233 The quantity ρ defined by (2327) is called a growth factor The growth factor measures how large the numbers become during the process of elimination In practice, ρ is usually of order 0 for partial pivot selection But it can be as large as ρ = 2 n, when A = 0 0 0 0 0 Better estimates hold for special types of matrices For example in the case of upper Hessenberg matrices, that is, matrices of the form A = 0 the bound ρ (n ) can be shown (Hessenberg matrices arise in eigenvalus problems) For tridiagonal matrices α β 2 0 γ 2 A = βn 0 γ n α n it can even be shown that ρ 2 holds for partial pivot selection elimination is quite numerically stable in this case For complete pivot selection, Wilkinson (965) has shown that Hence, Gaussian with the function a k ij f(k) max a ij i,j f(k) := k 2 [2 3 2 4 3 k (k ) ] 2 This function grows relatively slowly with k: k 0 20 50 00 f(k) 9 67 530 3300

38 Chapter 2 Numerical methods for solving linear systems Even this estimate is too pessimistic in practice Up until now, no matrix has been found which fails to satisfy a (k) ij (k + ) max a ij k =, 2,, n, i,j when complete pivot selection is used This indicates that Gaussian elimination with complete pivot selection is usually a stable process Despite this, partial pivot selection is preferred in practice, for the most part, because: (i) Complete pivot selection is more costly than partial pivot selection (To compute A (i), the maximum from among (n i + ) 2 elements must be determined instead of n i + elements as in partial pivot selection) (ii) Special structures in a matrix, ie the band structure of a tridiagonal matrix, are destroyed in complete pivot selection 236 Improving and Estimating Accuracy Iterarive Improvement: Suppose that the linear system Ax = b has been solved via the LR-factorization P A = LR Now we want to improve the accuracy of the computed solution x We compute Then in exact arithmatic we have r = b Ax, Ly = P r, Rz = y, x new = x + z Ax new = A(x + z) = (b r) + Az = b (2345) Unfortunately, r = fl(b Ax) renders an x new that is no more accurate than x It is necessary to compute the residual b Ax with extended precision floating arithmetic Algorithm 235 Compute P A = LR (t-digit) Repeat: r := b Ax (2t-digit) Solve Ly = P r for y (t-digit) Solve Rz = y for z (t-digit) Update x = x + z (t-digit) This is referred to as an iterative improvement From (2345) we have r i = b i a i x a i2 x 2 a in x n (2346) Now, r i can be roughly estimated by 2 t max j a ij x j That is r 2 t A x (2347)

23 Gaussian elimination 39 Let e = x A b = A (Ax b) = A r Then we have From (2347) follows that Let Then we have e A r (2348) e A 2 t A x = 2 t cond(a) x cond(a) = 2 p, 0 < p < t, (p is integer) (2349) e / x 2 (t p) (2350) From (2350) we know that x has q = t p correct significant digits Since r is computed by double precision, so we can assume that it has at least t correct significant digits Therefore for solving Az = r according to (2350) the solution z (comparing with e = A r) has q-digits accuracy so that x new = x + z has usually 2q-digits accuracy From above discussion, the accuracy of x new is improved about q-digits after one iteration Hence we stop the iteration, when the number of the iterates k (say!) satifies kq t From above we have z / x e / x 2 q = 2 t 2 p (235) From (2349) and (235) we have By (235) we get cond(a) = 2 t ( z / x ) q = log 2 ( x z ) and k = t log 2 ( x z ) In the following we shall give a further discussion of convergence of the iterative improvement From Theorem 238 we know that z in Algorithm 55 is computed by (A+δA)z = r That is A(I + F )z = r, (2352) where F = A δa Theorem 239 Let the sequence of vectors {x v } be the sequence of improved solutions in Algorithm 55 for solving Ax = b and x = A b be the exact solution Assume that F k in (2352) satisfies F k σ < /2 for all k Then {x k } converges to x, ie, lim v x k x = 0 Proof: From (2352) and r k = b Ax k we have A(I + F k )z k = b Ax k (2353) Since A is nonsingular, multiplying both sides of (2353) by A we get (I + F k )z k = x x k

40 Chapter 2 Numerical methods for solving linear systems From x k+ = x k + z k we have (I + F k )(x k+ x k ) = x x k, ie, Subtracting both sides of (2354) from (I + F k )x we get Applying Corollary 2 we have Hence, Let τ = σ/( σ) Then (I + F k )x k+ = F k x k + x (2354) (I + F k )(x k+ x ) = F k (x k x ) x k+ x = (I + F k ) F k (x k x ) x k+ x F k x k x F k σ σ x k x x k x τ k x x But σ < /2 follows τ < This implies convergence of Algorithm 235 Corollary 23 If then Algorithm 235 converges 0(n 3 + 3n 2 )ρ2 t A A < /2, Proof: From (2352) and (2344) follows that F k 0(n 3 + 3n 2 )ρ2 t cond(a) < /2 237 Special Linear Systems Toeplitz Systems Definition 232 (i) T R n n is called a Toeplitz matrix if there exists r n+,, r 0,, r n such that a ij = r j i for all i, j eg, T = r 0 r r 2 r 3 r r 0 r r 2 r 2 r r 0 r r 3 r 2 r r 0, (n = 4) (ii) B R n n is called a Persymmetric matrix if it is symmetric about northest-southwest diagonal, ie, b ij = b n j+,n i+ for all i, j That is, B = EB T E, where E = [e n, e ]

23 Gaussian elimination 4 Given scalars r,, r n such that the matrices r r 2 r k r T k = r r k are all positive definite, for k =,, n Three algorithms will be described: (a) Durbin s Algorithm for the Yule-Walker problem T n y = (r,, r n ) T (b) Levinson s Algorithm for the general right hand side T n x = b (c) Trench s Algorithm for computing B = T n To (a): Let E k = [e (k) k,, e(k) ] Suppose the k-th order Yule-Walker system T k y = (r,, r k ) T = r T has been solved Consider the (k + )-st order system [ ] [ ] [ Tk E k r z r r T = E k α r k+ can be solved in O(k) flops Observe that and Since T k we get z = T k ( r αe kr) = y αt k E kr (2355) ] α = r k+ = r T E k z (2356) is persymmetric, T k E k = E k T k and z = y +αe k y Substituting into (2356) α = r k+ r T E k (y + αe k y) = (r k+ + r T E k y)/( + r T y) Here ( + r T y) is positive, because T k+ is positive definite and [ ] T [ ] [ ] [ I Ek y Tk E k r I Ek y Tk 0 0 r T = E k 0 0 + r T y Algorithm 236 (Durbin Algorithm, 960) Let T k y (k) = r (k) = (r,, r k ) T For k =,, n, y () = r, for k =,, n, β k = + r (k)t y (k), α k = (r k+ + r (k)t E k y (k) )/β k, z (k) = y (k) [ + α k E] k y (k), z y (k+) (k) = α k ]

42 Chapter 2 Numerical methods for solving linear systems This algorithm requires 3 2 n2 flops to generate y = y (n) Further reduction: β k = + r (k)t y (k) [ ] y = + [r (k )T, r (k) ] (k ) + α k E k y (k ) α k = + r (k )T y (k ) + α k (r (k )T E k y (k ) + r k ) = β k + α k ( β k α k ) = ( α 2 k )β k To (b): Want to solve T k x = b = (b,, b k ) T, for k n (2357) [ Tk E k r r T E k ] [ ν µ ] [ b b k+ ], (2358) where r = (r,, r k ) T Since ν = T k (b µe kr) = x + µe k y, it follows that µ = b k+ r T E k ν = b k+ r T E k x µr T y = (b k+ r T E k x)/( + r T y) We can effect the transition form (2357) to (2358) in O(k) flops We can solve the system T n x = b by solving and T k x (k) = b (k) = (b,, b k ) T T k y (k) = r (k) = (r,, r k ) T It needs 2n 2 flops See Algorithm Levinson (947) in Matrix Computations, pp28-29 for details To (c): [ A Er Tn = r T E ] [ B ν = ν T γ where A = T n, E = E n and r = (r,, r n ) T From the equation [ ] [ ] [ ] A Er ν 0 r T = = E γ follows that Aν = γer = γe(r,, r n ) T and γ = r T Eν If y is the solution of (n )-st Yule-Walker system Ay = r, then Thus the last row and column of Tn have γ = /( + r T y) and ν = γey ], are readily obtained Since AB + Erν T = I n, we B = A (A Er)ν T = A + ννt γ

23 Gaussian elimination 43 Since A = T n is nonsingular and Toeplitz, its inverse is persymmetric Thus b ij = (A ) ij + ν iν j γ = b n j,n i ν n iν n j γ = (A ) n j,n i + ν iν j γ + ν iν j γ = b n j,n i γ (ν iν j ν n i ν n j ) It needs 7 4 n2 flops See Algorithm Trench (964) in Matrix Computations, pp32 for details Banded Systems Definition 233 Let A be a n n matrix A is called a (p, q)-banded matrix, if a ij = 0 whenever i j > p or j i > q A has the form A = O O } {{ } p where p and q are the lower and upper band widthes, respectively q, Example 232 (, ): tridiagonal matrix; (, n ): upper Hessenberg matrix; (n, ): lower Hessenberg matrix Theorem 230 Let A be a (p, q)-banded matrix Suppose A has a LR-factorization (A = LR) Then L = (p, 0) and R = (0, q)-banded matrix, respectively Algorithm 237 See Algorithm 43 in Matrix Computations, pp50 Theorem 23 Let A be a (p, q)-banded nonsingular matrix If Gaussian Elimination with partial pivoting is used to compute Gaussian transformations L j = I l j e T j, for j =,, n, and permutations P,, P n such that L n P n L P A = R is upper triangular, then R is a (0, p + q)-banded matrix and l ij = 0 whenever i j or i > j + p (Since the j-th column of L is a permutation of the Gaussian vector l j, it follows that L has at most p + nonzero elements per column)

44 Chapter 2 Numerical methods for solving linear systems Symmetric Indefinite Systems Consider the linear system Ax = b, where A R n n is symmetric but indefinite There are a method using n 3 /6 flops due to Aasen (97) that computes the factorization P AP T = LT L T, where L = [l ij ] is unit lower triangular, P is a permutation chosen such that l ij, and T is tridiagonal Rather than the above factorization P AP T = LT L T we have the calculation of P AP T = LDL T, where D is block diagonal with by and 2 by 2 blocks on diagonal, L = [l ij ] is unit lower triangular, and P is a permutation chosen such that l ij Bunch and Parlett (97) has proposed a pivot strategy to do this, n 3 /6 flops are required Unfortunately the overall process requires n 3 /2 n 3 /6 comparisons A better method described by Bunch and Kaufmann (977) requires n 3 /6 flops and O(n 2 ) comparisons A detailed discussion of this subsection see p59-68 in Matrix Computations