Orthogonalization and least squares methods

Transcription

1 Chapter 3 Orthogonalization and least squares methods 31 QR-factorization (QR-decomposition) 311 Householder transformation Definition 311 A complex m n-matrix R = [r ij is called an upper (lower) triangular matrix, if r ij = for i > j (i < j) Example 311 r 11 r 1n (1) m = n : R = r nn r 11 r 1n (3) m > n : R = r nn, (2) m < n : R = r 11 r 1n r mm r mn Definition 312 Given A C m n, Q C m m unitary and R C m n upper triangular as in Examples such that A = QR Then the product is called a QR-factorization of A Basic problem: Given b, b C n Find a vector w C n with w w = 1 and c C such that Solution (Householder transformation): (1) b = : w arbitrary (in general w = ) and c = (I 2ww )b = ce 1 (311), (2) b : { b 1 c = b b 1 2, if b 1, b 2, if b 1 =, { w = 1 (b 2k 1 c, b 2,, b n ) T := 1 u 2k with2k = 2 b 2 ( b 2 + b 1 ) (312) (313)

2 46 Chapter 3 Orthogonalization and least squares methods Theorem 311 Any complex m n matrix A can be factorized by the product A = QR, where Q is m m-unitary R is m n upper triangular Proof: Let A () = A = [a () 1 a() 2 a() n Find Q 1 = (I 2w 1 w1 ) such that Q 1a () 1 = ce 1 Then A (1) = Q 1 A () = [Q 1 a () 1, Q 1a () 2,, Q 1a () n = c 1 a (1) 2 a (1) n 1 Find Q 2 = such that (I 2w 2 w I w 2 w2 2 )a(1) 2 = c 2 e 1 Then c 1 c 2 A (2) = Q 2 A (1) = a (2) 3 a (2) n (314) We continue this process Then after l = min(m, n) steps A (l) is an upper triangular matrix satisfying A (l 1) = R = Q l 1 Q 1 A Then A = QR, where Q = Q 1 Q l 1 Remark 311 We usually call the method in Theorem 311 as Householder method (Algorithm??) Theorem 312 Let A be a nonsingular n n matrix Then the QR- factorization is essentially unique That is, if A = Q 1 R 1 = Q 2 R 2, then there is a unitary diagonal matrix D = diag(d i ) with d i = 1 such that Q 1 = Q 2 D and DR 1 = R 2 Proof: matrix Let A = Q 1 R 1 = Q 2 R 2 Then Q 2Q 1 = R 2 R 1 1 = D must be a diagonal unitary Remark 312 The QR-factorization is unique, if it is required that the diagonal elements of R are positive Corollary 311 A is an arbitrary m n-matrix The following factorizations exist: (i) A = LQ, where Q is n n unitary and L is m n lower triangular (ii) A = QL, where Q is m m unitary and L is m n lower triangular (iii) A = RQ, where Q is n n unitary and R is m n upper triangular

3 31 QR-factorization (QR-decomposition) 47 Proof: (i) A has a QR-factorization Then A = QR A = R Q (i) O 1 (ii) Let P m = Then by Theorem 311 we have P m AP n = QR This implies 1 O A = (P m QP m )(P m RP n ) QL (ii) (iii) A has a QL-factorization (from (ii)), ie, A = QL This implies A = L Q (iii) Cost of Householder method Consider that the multiplications in (314) can be computed in the form (I 2w 1 w 1 )A = (I u 1 b b 1 b 2 u 1 )A = (I vu 1 )A = A vu 1 A := A vw So the first step for a m n-matrix A requires; c 1 : m multiplications, 1 root; 4k 2 : 1 multiplication; v: m divisions (= multiplications); w: mn multiplications; A (1) = A vw : m(n 1) multiplications Similarly, for the j-th step m and n are replaced by m j+1 and n j+1, respectively Let l = min(m, n) Then the number of multiplications is l 1 [2(m j + 1)(n j + 1) + (m j + 2) (315) j=1 = l(l 1)[ 2l 1 (m + n) 5/2 + (l 1)(2mn + 3m + 2n + 4) 3 (= mn 2 1/3n 3, if m n) Especially, for m = n, it needs n 1 [2(n j + 1) 2 + m j + 2 = 2/3n 3 + 3/2n /6n 4 (316) j=1 flops and (l + n 2) roots To compute Q = Q 1 Q l 1, it requires 2[m 2 n mn 2 + n 3 /3 multiplications (m n) (317) Remark 313 Let A = QR be a QR-factorization A Then we have A A = R Q QR = R R If A has full column rank and we require that the diagonal elements of R are positive, then we obtain the Cholesky factorization of A A

4 48 Chapter 3 Orthogonalization and least squares methods 312 Gram-Schmidt method Remark 314 Theorem 311 (or Algorithm??) can be used to solved orthonormal basis (OB) problem (OB) : Given linearly independent vectors a 1,, a n R n 1 Find an orthonormal basis for span{a 1,, a n } If A = [a 1,, a n = QR with Q = [q 1,, q n, and R = [r ij, then a k = k r ik q i (318) By assumption rank(a) = n and (318) it implies r kk So, we have q k = 1 k 1 (a k r ik q i ) (319) r kk The vector q k can be thought as a unit vector in the direction of z k = a k k 1 s ikq i To ensure that z k q 1,, q k 1 we choose s ik = q T i a k, for i = 1,, k 1 This leads to the Classical Gram-Schmidt (CGS) Algorithm for solving (OB) problem Algorithm 311 (Classical Gram-Schmidt (CGS) Algorithm) Given A R m n with rank(a) = n We compute A = QR, where Q R m n has orthonormal columns and R R n n For i = 1,, n, q i = a i ; For j = 1,, i 1 r ji = q T j a i, q i = q i r ji q j, end for r ii = q i 2, q i = q i /r ii, end for Disadvantage : The CGS method has very poor numerical properties, if some columns of A are nearly linearly independent Advantage : The method requires mn 2 multiplications (m n) Remark 315 Modified Gram-Schmidt (MGS): Write A = n q ir T i Define A (k) by k 1 [, A (k) = A q i ri T = n q i ri T (311) It follows that if A (k) = [z, B, z R m, B R m (n k) then r kk = z 2 and q k = z/r kk by (319) Compute [r k,k+1,, r kn = q T k B Next step: A (k+1) = B q k [r k,k+1,, r kn i=k

5 31 QR-factorization (QR-decomposition) 49 Algorithm 312 (MGS) Given A R m n with rank(a) = n We compute A = QR, where Q R m n has orthonormal columns and R R n n is upper triangular For i = 1,, n, q i = a i ; For j = 1,, i 1 r ji = q T j q i, q i = q i r ji q j, end for r ii = q i 2, q i = q i /r ii, end for The MGS requires mn 2 multiplications Remark 316 MGS computes the QR factorization at the kth step, the kth column of Q and the kth row of R are computed CGS at the kth step, the kth column of Q and the kth column of R are computed Advantage for OB problem (m n): (i) Householder method requires mn 2 n 3 /3 flops to get factorization A = QR and mn 2 n 3 /3 flops to get the first n columns of Q But MGS requires only mn 2 flops Thus for the problem of finding an orthonormal basis of range(a), MGS is about twice as efficient as Householder orthogonalization (ii) MGS is numerically stable 313 Givens method Basic problem: Given (a, b) T R 2, find c, s R with c 2 + s 2 = 1 such that [ [ [ c s a k =, s c b where c = cosα and s = sinα Solution: { c = 1, s =, k = a; if b =, Let c = a a, s = 2 +b 2 1 G(i, j, α) = b a, k = a 2 + b 2 ; if b 2 +b 2 cos α sin α sin α cos α 1 (3111) Then G(i, j, α) is called a Givens rotation in the (i, j)-coordinate plane In the matrix à = G(i, j, α)a, the rows with index i, j are the same as in A and ã ik = cos(α)a ik + sin(α)a jk, for k = 1,, n, ã jk = sin(α)a ik + cos(α)a jk, for k = 1,, n

6 5 Chapter 3 Orthogonalization and least squares methods Algorithm 313 (Givens orthogonalization) Given A R m n The folllowing Algorithm overwrites A with Q T A = R, where Q is orthonormal and R is upper triangular For q = 2,, m, for p = 1, 2,, min{q 1, n}, Find c = cos α and s = sin α as in (3111) such that [ [ [ c s app = s c A := G(p, q, α)a This algorithm requires 2n 2 (m n/3) flops Fast Givens method (See Matrix Computations, pp25-29): A modification of Givens method bases on the fast Givens rotations and requires about n 2 (m n/3) flops 32 Overdetermined linear Systems - Least Squares Methods Given A R m n, b R m and m > n Consider the least squares(ls) problem: a qp min x R n Ax b 2 (321) Let X be the set of minimizers defined by X = {x R n Ax b 2 = min!} It is easy to see the following properties: x X A T (b Ax) = (322) X is convex (323) X has a unique element x LS having minimal 2-norm (324) X = {x LS } rank(a) = n (325) For x R n, we refer to r = b Ax as its residual A T (b Ax) = is refered to as the normal equation The minimum sum is defined by ρ 2 LS = Ax LS b 2 2 If we let ϕ(x) = 1 Ax 2 b 2 2, then ϕ(x) = AT (Ax b) Theorem 321 Let A = r σ i u i v T i, with r =rank(a), U = [u 1,, u m and V = [v 1,, v n be the SVD of A R m n (m n) If b R m, then and x LS = r (u T i b/σ i )v i (326) ρ 2 LS = m (u T i b) 2 (327) i=r+1

7 32 Overdetermined linear Systems - Least Squares Methods 51 Proof: For any x R n we have Ax b 2 2 = U T AV (V T x) U T b 2 2 = r m (σ i α i u T i b)2 + (u T i b)2, i=r+1 where α = V T x Clearly, if x solves the LS-problem, then α i = (u T i b/σ i), for i = 1,, r If we set α r+1 = = α n =, then x = x LS Remark 321 If we define A + by A + = V Σ + U T, where Σ + = diag(σ 1 1,, σ 1 r,,, ) R n m then x LS = A + b and ρ LS = (I AA + )b 2 A + is refered to as the pseudo-inverse of A A + is defined to be the unique matrix X R n m that satisfies Moore-Penrose conditions : (i)axa = A, (ii)xax = X, (iii) (AX) T = AX, (iv) (XA) T = XA (328) Existence of X is easy to check by taking X = A + Now, we show the uniqueness of X Suppose X and Y satisfying the conditions (i) (iv) Then X = XAX = X(AY A)X = X(AY A)Y (AY A)X = (XA)(Y A)Y (AY )(AX) = (XA) T (Y A) T Y (AY ) T (AX) T = (AXA) T Y T Y Y T (AXA) T = A T Y T Y Y T A T = Y (AY A)Y = Y AY = Y If rank(a) = n (m n), then A + = (A T A) 1 A T If rank(a) = m (m n), then A + = A T (AA T ) 1 If m = n = rank(a), then A + = A 1 For the case rank(a)=n: Algorithm 321 (Normal equations) Given A R m n (m n) with rank(a) = n and b R m This Algorithm computes the solution to the LS-problem: min{ Ax b 2 ; x R n } Compute d := A T b, and form C := A T A by computing the Cholesky factorization C = R T R (see Remark 61) Solve R T y = d and Rx LS = y Algorithm 322 (Householder and Givens orthogonalizations) Given A R m n (m n) with rank(a) = n and b R m This Algorithm computes the solutins to the LS-problem: min{ Ax b 2 ; x R n } [ Compute QR-factorization Q T R1 A = by using Householder and Givens methods respectively (Here R 1 is upper triangular) Then where Q T b = [ c d Ax b 2 2 = Q T Ax Q T b 2 2 = R 1 x c d 2 2, Thus, x LS = R 1 1 c, (since rank(a) =rank(r 1 ) = n) and ρ 2 LS = d 2 2

8 52 Chapter 3 Orthogonalization and least squares methods Algorithm 323 (Modified Gram-Schmidt) Given A R m n (m n) with rank(a) = n and b R m The solution of min Ax b 2 is given by: Compute A = Q 1 R 1, where Q 1 R m n with Q T 1 Q 1 = I n and R 1 R n n upper triangular Then the normal equation (A T A)x = A T b is transformed to the linear system R 1 x = Q T 1 b x LS = R1 1 Q1 T b For the case rank(a) < n: Problem: (i) How to find a solution to the LS-problem? (ii) How to find the unique solution having minimal 2-norm? (iii) How to compute x LS reliably with infinite conditioned A? Definition 321 Let A be a m n matrix with rank(a) = r (r m, n) The factorization A = BC with B R m r and C R r n is called a full rank factorization, provided that B has full column rank and C has full row rank Theorem 322 If A = BC is a full rank factorization, then A + = C + B + = C T (CC T ) 1 (B T B) 1 B T (329) Proof: From assumption follows that B + B = (B T B) 1 B T B = I r, CC + = CC T (CC T ) 1 = I r We calculate (328) with A(C + B + )A = BCC + B + BC = BC = A, (C + B + )A(C + B + ) = C + B + BCC + B + = C + B +, A(C + B + ) = BCC + B + = BB + symmetric, (C + B + )A = C + B + BC = C + C symmetric These imply that X = C + B + satisfies (328) It follows A + = C + B + Unfortunately, if rank(a) < n, then the QR-factorization does not necessarily produce a full rank factorization of A For example A = [a 1, a 2, a 3 = [q 1, q 2, q Fortunately,we have the following two methods to produce a full rank factorization of A

9 32 Overdetermined linear Systems - Least Squares Methods Rank Deficiency I : QR with column pivoting Algorithm?? can be modified in a simple way so as to produce a full rank factorization of A [ R11 R AΠ = QR, R = 12 }r, (321) }m-r }{{} r }{{} n r where r = rank(a) < n (m n), Q is orthogonal, R 11 is nonsingular upper triangular and Π is a permuatation Once (321) is computed, then the LS-problem can be readily solved by Ax b 2 2 = (QT Aπ)(π T x) Q T b 2 2 = R 11y (c R 12 z) d 2 2, [ [ y where Π T }r c x = and Q z T }r b = }n-r d Thus if Ax b 2 = min!, then we }m-r must have [ R 1 11 (c R x = Π 12 z) z If z is set to be zero, then we obtain the basic solution [ R 1 11 c x B = π The basic solution is not the solution with minimal 2-norm, unless the submatrix R 12 is zero Since [ x LS 2 = min R 1 z R x 11 R n r B π 12 z I n r (3211) 2 We now solve the LS-problem (3211) by using Algorithms 321 to 323 Algorithm 324 Given A R m n, with rank(a) = r < n The following algorithm computes the factorization AΠ = QR defined by (321) The element a ij is overwritten by r ij (i j) The permutation Π = [e c1,, e cn is determined according to choosing the maximum of column norm in the current step c j := j (j = 1, 2,, n), r j := m (j = 1,, n), a 2 ij For k = 1,, n, Detemine p with (k p n) so that r p = max r j k j n If r p = then stop; else Interchange c k and c p, r k and r p, and a ik and a ip, for i = 1,, m Determine a Householder ˆQ k such that ˆQ k a kk a mk = A := diag(i k 1, ˆQ k )A; r j := r j a 2 kj (j = k + 1,, n)

10 54 Chapter 3 Orthogonalization and least squares methods This algorithm requires 2mnr r 2 (m + n) + 2r 3 /3 flops Algorithm 324 produces the full rank factorization (321) of A We have the following important relations: { r11 r 22 r rr, r jj =, j = r + 1,, n, r ii r ik, i = 1,, r, k = i + 1,, n (3212) Here, r = rank(a) < n, and R = (r jj ) In the following we show another application of the full rank factorization for solving the LS-problem Algorithm 325 (Compute x LS = A + b directly) ( (i) Compute (321): AΠ = QR (Q }{{} (1) Q (2) R1 ) r ) }r }m-r, AΠ = Q(1) R 1 (ii) (AΠ) + = R + 1 Q (1)+ = R + 1 Q (1)T (iii) Compute R + 1 : Either: R 1 + = RT 1 (R 1R1 T ) 1 (since R 1 has full row rank) (AΠ) + = R1 T (R 1 R1 T ) 1 Q (1)T Or: [ Find Q using Householder transformation (Algorithm??) such that QR1 T = T, where T R r r is upper triangular Let Q T := ( ˆQ (1), ˆQ (2) ) R T 1 = ˆQ (1) T + ˆQ (2) = ˆQ (1) T R 1 = T T ˆQ(1) T R + 1 = ( ˆQ (1)T ) + (T T ) + = ˆQ (1) (T T ) 1 (AΠ) + = ˆQ (1) (T T ) 1 Q (1)T (iv) Since min Ax b 2 = min AΠ(Π T x) b 2 (Π T x) LS = (AΠ ) + b x LS = Π (AΠ ) + b Remark 322 Unfortunately, QR with column pivoting is not entirely reliable as a method for detecting near rank deficiency For example: T n (c) = diag(1, s,, s n 1 ) 1 c c c 1 c c 1 c2 + s 2 = 1, c, s > If n = 1, c = 2, then σ n =3679e 8 But this matrix is unaltered by Algorithm 324 However,the degree of unreliability is somewhat like that for Gaussian elimination with partial pivoting, a method that works very well in practice

11 32 Overdetermined linear Systems - Least Squares Methods Rank Deficiency II : The Singular Value Decomposition Algorithm 326 (Householder Bidiagonalization) Given A R m n (m n) The following algorithm overwrite A with U T B AV B = B, where B is upper bidiagonal and U B and V B are orthogonal For k = 1,, n, Determine a Householder matrix Ũk of order n k + 1 such that U k a kk a mk =, A := diag(i k 1, U k )A, If k 2, then determine a Householder matrix V k of order n k + 1 such that [a k,k+1,, a kn V k = (,,, ), A := Adiag(I k, V k ) This algorithm requires 2mn 2 2/3n 3 flops Algorithm 327 (R-Bidiagonalization) when m n we can use the following faster method of bidiagonalization [ (1) Compute an orthogonal Q 1 R m m such that Q T 1 A = R1, where R 1 R n n is upper triangular (2) Applying Algorithm 326 to R 1, we get Q T 2 R 1 V B = B 1, where Q 2, V B R n n orthogonal and B 1 R n n upper bidiagonal [ (3) Define U B = Q 1 diag(q 2, I m n ) Then UB T AV B1 B = B bidiagonal This algorithm require mn 2 + n 3 It involves fewer compuations comparing with Algorithm 76 (2mn 2 2/3n 3 ) whenever m 5/3n Once the bidiagonalization of A has been achieved,the next step in the Golub-Reinsch SVD algorithm is to zero out the super diagonal elements in B Unfortunately, we must defer our discussion of this iteration until Chapter 5 since it requires an understanding of the symmetric QR algorithm for eigenvalues That is, it computes orthogonal matrices U Σ and V Σ such that UΣ T BV Σ = Σ = diag(σ 1,, σ n ) By defining U = U B U Σ and V = V B V Σ, we see that U T AV = Σ is the SVD of A

12 56 Chapter 3 Orthogonalization and least squares methods Algorithms Flop Counts Algorithm 321 Normal equations mn 2 /2 + n 3 /6 Algorithm 322 Householder orthogonalization mn 2 n 3 /3 rank(a)=n Algorithm 323 Modified Gram-Schmidt mn 2 Algorithm 313 Givens orthogonalization 2mn 2 2/3n 3 Algorithm 326 Householder Bidiagonalization 2mn 2 2/3n 3 Algorithm 327 R-Bidiagonalization mn 2 + n 3 LINPACK Golub-Reinsch SVD 2mn 2 + 4n 3 rank(a) < n Algorithm 325 QR-with column pivoting 2mnr mr 2 + 1/3r 3 Alg 327+SVD Chan SVD mn /2n 3 Table 31: Solving the LS problem (m n) Remark 323 If the LINPACK SVD Algorithm is applied with eps=1 17 to 1 c c c T 1 (2) = diag(1, s,, s n 1 1 c c ), 1 then ˆσ n = Remark 324 As we mentioned before, when solving the LS problem via the SVD, only Σ and V have to be computed (see (326)) Table 31 compares the efficiency of this approach with the other algorithms that we have presented 323 The Sensitivity of the Least Squares Problem Corollary 321 (of Theorem 123) Let U = [u 1,, u m, V = [v 1,, v n and U AV = Σ = diag(σ 1,, σ r,,, ) If k < r = rank(a) and A k = k σ i u i vi T, Then min A B 2 = A A k 2 = σ k+1 rank(b)=k Proof: Since U T A k V = diag(σ 1,, σ k,,, ), it follows rank(a k ) = k and that A A k 2 = U T (A A k )V 2 = diag(,,, σ k+1,, σ r ) 2 = σ k+1 Suppose B R m n and rank(b) = k, ie, there are orthogonal vectors x 1,, x n k such that N (B) = span{x 1,, x n k } This implies span{x 1,, x n k } span{v 1,, v k+1 } {}

13 32 Overdetermined linear Systems - Least Squares Methods 57 Let z be a unit vector in the intersection set Then Bz = and Az = k+1 σ i (vi T z)u i Thus, k+1 A B 2 2 (A B)z 2 2 = Az 2 2 = σi 2 (vi T z) 2 σk Condition number of a Rectangular Matrix Let A R m n, rank(a) = n, κ 2 (A) = σ max (A)/σ min (A) (i) The method of normal equation: (a) C = A T A, d = A T b min Ax b x R n 2 A T Ax = A T b (b) Compute the Cholesky factorization C = GG T (c) Solve Gy = d and G T x LS = y Then x LS x LS 2 x LS epsκ 2 (A T A) = epsκ 2 (A) 2 x x x κ(a) where (A + F ) x = b + f and Ax = b (ii) LS solution via QR factorization ( ε F ) A + ε f + o(ε 2 ), b Ax b 2 2 = QT Ax Q T b 2 2 = R 1x c d 2 2, x LS = R 1 1 c, ρ LS = d 2 Numerically, trouble can be expected wherever κ 2 (A) = κ 2 (R) 1/eps But this is in contrast to normal equation, Cholesky factorization becomes problematical once κ 2 (A) is in the neighborhood of 1/ eps Remark 325 A 2 (A T A) 1 A T 2 = κ 2 (A), A 2 2 (AT A) 1 2 = κ 2 (A) 2 Theorem 323 Let A R m n, (m n), b Suppose that x, r, x, r satisfy Ax b = min!, r = b Ax, ρ LS = r 2, (A + δa) x (b + δb) 2 = min!, r = (b + δb) (A + δa) x

14 58 Chapter 3 Orthogonalization and least squares methods If ε = max { δa 2, δb 2 } < σ n(a) A 2 b 2 σ 1 (A) and then and x x 2 x 2 sin θ = ρ LS b 2 1, ε{ 2κ 2(A) cos θ + tan θκ 2(A) 2 } + O(ε 2 ) r r 2 b 2 ε(1 + 2κ 2 (A)) min(1, m n) + O(ε 2 ) Proof: Let E = δa/ε and f = δb/ε Since δa 2 < σ n (A), by previous Corollary follows that rank(a + εe) = n for t [, ε [t = ε A + te = A + δa If rank(a + δa) = k < n, then A (A + δa) 2 = δa 2 A A k 2 = σ k+1 δ n Contradiction! So min A B 2 = A A k 2 rank(b)=k = A k σ i u i vi T 2 = σ k+1 Hence we have, (A + te) T (A + te)x(t) = (A + te) T (b + tf) (3213) Since x(t) is continuously differentiable for all t [, ε, x = x() and x = λ(ε), it follows that x = x + εẋ() + O(ε 2 ) and x x x = ε ẋ() 2 x Differentiating (3213) and setting t = then we have Thus, + O(ε 2 ) E T Ax + A T Ex + A T Aẋ() = A T f + E T b ẋ() = (A T A) 1 A T (f Ex) + (A T A) 1 E T r From f 2 b 2 and E 2 A 2 follows x x 2 ε{ A 2 (A T A) 1 A T b 2 2 ( + 1) x 2 A 2 x 2 ρ LS + A 2 2 A 2 x (AT A) 1 2 } + O(ε 2 ) 2 Since A T (Ax LS b) =, Ax LS Ax LS b and then b Ax Ax 2 2 = b 2 2 and A 2 2 x 2 2 b 2 2 ρ2 LS

15 32 Overdetermined linear Systems - Least Squares Methods 59 Thus, x x 2 1 eps{κ 2 (A)( x 2 cos θ + 1) + κ 2(A) 2 sin θ cos θ } + O(ε2 ) Furthermore, by sin θ cos θ = ρ LS, we have b 2 2 ρ 2 LS x x 2 x 2 eps(κ 2 (A) + κ 2 (A) 2 ρ LS ) (θ : small ) Remark 326 Normal equation: eps κ 2 (A) 2 QR-approach: eps(κ 2 (A) + ρ LS κ 2 (A) 2 ) (i) If ρ LS is small and κ 2 (A) is large, then QR is better than the normal equation (ii) The normal equation approach involves about half of the arithmetic when m n and does not requires as much storage (iii) The QR approach is applicable to a wider class of matrices because the Cholesky to A T A break down before the back substitution process on Q T A = R 325 Iterative Improvement r + Ax = b, [ f (k) g (k) [ Im A A T [ r x = [ b A T r = A T Ax = A T b Thus, [ [ [ b I A r (k) = A T and x (k), b Ax 2 = min! [ I A A T [ p (k) z (k) [ f (k) = g (k) This implies, [ R1 If A = QR = Q where Q T f = [ f1 f 2, then [ h Q T p = f 2 [ [ [ r (k+1) r (k) p (k) = + x (k+1) [ I A A T I n R 1 I m n R1 T [ p z x (k) = h f 2 z [ f g z (k) implies that = f 1 f 2 g, Thus, R T 1 h = g h = R T 1 g Then [ h z = R1 1 (f 1 h), P = Q f2

16 6 Chapter 3 Orthogonalization and least squares methods