STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9

Transcription

1 STAT 309: MATHEMATICAL COMPUTATIONS I FALL 2018 LECTURE 9 1. qr and complete orthogonal factorization poor man s svd can solve many problems on the svd list using either of these factorizations but they are much cheaper to compute there are direct algorithms for computing qr and complete orthogonal factorization in a finite number of arithmetic steps recall that svd is spectral in nature only iterative algorithms in general by Galois Abel, although for any fixed precision (fixed number of decimal places), we can compute svd in finitely many steps there are several versions of qr factorization version 1: for any A C m n with n m, there exist a unitary matrix Q C m m (i.e., Q Q = QQ = I n ) and an upper-triangular matrix R C m n (i.e., r ij = 0 whenver i > j) such that R1 A = QR = Q (1.1) 0 R 1 C n n is an upper-triangular square matrix in general if A has full column rank, i.e., rank(a) = n, then R 1 is nonsingular this is called the full qr factorization of A version 2: for any A C m n with n m, there exist a unitary matrix Q 1 C m n (i.e., Q 1 Q 1 = I n but Q 1 Q 1 I m unless m = n) and an upper-triangular square matrix R 1 C n n such that A = Q 1 R 1 (1.2) R 1 here is in fact the same R 1 as in (1.1) Q 1 is the first n columns of Q in (1.1), i.e., Q = [Q 1, Q 2 ] where Q 2 C m (m n) is the last m n columns of Q in fact we obtain (1.2) from (1.1) by simply multiplying out R1 A = QR = [Q 1, Q 2 ] = Q 0 1 R 1 + Q 2 0 = Q 1 R 1 as before, if A has full column rank, i.e., rank(a) = n, then R 1 is nonsingular this is called the reduced qr factorization of A version 3: for any A C m n with rank(a) = r, there exist a permutation matrix Π C n n, a unitary matrix Q C m m, and a nonsingular, upper-triangular square matrix R 1 C r r such that R1 S AΠ = Q (1.3) S C r (n r) is just some matrix with no special properties this is called the rank-retaining qr decomposition of A form Date: October 22, 2018, version 1.0. Comments, bug reports: lekheng@galton.uchicago.edu. 1

2 we may also write (1.3) as A = QRΠ T = Q R1 S Π T (1.4) version 4: for any A C m n with rank(a) = r, there exist a unitary matrix Q C m m, a unitary matrix U C n n, and a nonsingular, lower-triangular square matrix L C r r such that L 0 A = Q U (1.5) this is called the complete orthogonal factorization [ of] A it can be obtained from a full qr factorization of R 1 C m r, which has full column S rank, R 1 R2 S = Z (1.6) 0 where Z C m m is unitary and R 2 C r r is nonsingular, upper-triangular square matrix observe from (1.4) and (1.6) that A = Q [ R1 S ] Π T = Q R 2 0 Z Π T = Q L 0 U where we set L = R2 and U = ΠZ. note that for a matrix that is not of full column rank, a qr decomposition would necessarily mean either versions 3 or 4 there are yet other variants of qr factorizations that can be obtained using essentially the same algorithms (Givens and Householder qr): A = QR, A = LQ, A = RQ, A = QL where Q is unitary, R is upper triangular, and L is lower triangular using such variants, we could for instance make the lower triangular matrix L in (1.5) an upper-triangular matrix instead the qr factorization is sometimes regarded as a generalization of the polar form of a complex number a C, a = re iθ to matrices, we will see later that we may always choose our R so that r ii 0 2. aside: permutation matrices the permutation matrix Π in (1.3) comes from performing column pivoting in the algorithm recall that a permutation matrix is a simply the identity matrix with the rows and columns permuted, e.g. Π = 1 (2.1) 1 multiplying a matrix A C m n by an n n permutation matrix on the right, i.e., AΠ, has the effect of permuting the columns of A according to precisely the way the columns of Π are permuted from the identity, e.g. a b c d e f 1 = c a b f d e g h i 1 i g h 2

3 multiplying a matrix A C m n by an m m permutation matrix on the left, i.e., ΠA, has the effect of permuting the rows of A according to precisely the way the rows of Π are permuted from the identity, e.g. 1 a b c d e f = d e f g h i 1 g h i a b c multiplying a square matrix A C n n by an n n permutation matrix on the left and its transpose on the right, i.e., ΠAΠ T, has the effect of permuting the diagonal of A entries on the diagonal stays on the diagonal and entries off the diagonal stays off diagonal, e.g. 1 a b c d e f T 1 = e f d h i g 1 g h i 1 b c a note that a, e, i stays on the diagonal as expected permutation matrices are always orthogonal (also unitary since it has real entries), i.e. Π T Π = ΠΠ T = I or Π 1 = Π T = Π we don t store permutation matrices as matrices of floating point numbers, we store just the permutation, e.g. (2.1) can be stored as since it takes column 3 to column 1, column 1 to column 2, column 2 to column 3 3. existence and uniqueness of qr if A C m n has full column rank, i.e., rank(a) = n m, then we will show existence and (some kind of) uniqueness of its reduced qr factorization uniqueness is easy if m = n suppose A = Q 1 R 1 = Q 2 R 2 for Q 1, Q 2 C n n are unitary and R 1, R 2 C n n are nonsingular then Q 2Q 1 = R 2 R 1 1 note that the left-hand side is unitary and right hand side is upper-triangular the only matrix that is both unitary and upper-triangular is a diagonal matrix of the form D = diag(e iθ 1,..., e iθn ) so we get Q 2 = Q 1 D, R 2 = DR 1 qr factorization is unique up to such unimodular scaling more generally, we could also get uniqueness without requiring m = n this follows from Gram Schmidt, which we could also use to establish existence 4. Gram Schmidt orthogonalization suppose A C n n is square and full-rank so all the column vectors of A are linearly independent consider the qr factorization A = [ r 11 r 1n ] a 1 a n = q1 q n r nn = QR

4 from this matrix equation, we get a 1 = r 11 q 1 a 2 = r 12 q 1 + r 22 q 2. a n = r 1n q 1 + r 2n q r nn q n and from which we can deduce an algorithm first note that a 1 = r 11 q 1, and so next, from a 2 = r 12 q 1 + r 22 q 2 we get r 11 = a 1 2, q 1 = 1 a 1 2 a 1 r 12 = q 1a 2, r 22 = a 2 r 12 q 1 2, q 2 = 1 (a 2 r 12 q 1 ) r 22 in general, we get k a k = r jk q j and hence q k = 1 a k r jk q j, r jk = q ja k note that 0: since a 1,..., a n are linearly independent and so and so and so a k / span{a 1,..., a k 1 } = span{q 1,..., q k 1 } a k r jk q j 0 = a k r jk q j 0 (4.1) 2 this is the Gram Schmidt algorithm, there are two ways to see it given a list of linearly independent vectors a 1,..., a n C n, it produces a list of orthogonormal vectors q 1,..., q n that spans the same subspace given a matrix A C n n of full rank, it produces a qr factorization A = QR so we have established the existence of qr in fact, it is clear that if we started from a list of linearly independent vectors a 1,..., a n C m where n m or equivalently a matrix A C m n of full column rank rank(a) = n m, the Gram Schmidt algorithm would still produce a list of orthogonormal vectors q 1,..., q n or equivalently a matrix Q C m n with orthonormal columns the only difference is that the algorithm would terminate at step n when it runs out of input vectors note that this is a special qr factorization since > 0 for all k = 1,..., n (because is chosen to be a norm) in fact, requiring > 0 gives us uniqueness (not just uniqueness up to unimodular scaling) now what if A C m n is not full rank, i.e., a 1,..., a n are not linearly independent in this case Gram Schmidt could fail since in (4.1) can now be 0 4

5 we need to modify Gram Schmidt so that it finds a subset of a 1,..., a n that is linearly independent this is equivalent to finding a permutation matrix Π so that the first r = rank(a) columns of AΠ are linearly independent this can be done adaptively and corresponds to column pivoting we will discuss this later when we discuss Givens and Householder qr algorithms, which are what used in practice the truth is that Gram Schmidt is really a lousy algorithm it is numerically unstable for example, if a 1 and a 2 are almost parallel, then a 2 r 12 q 1 is almost zero and roundoff error becomes significant because of such numerical instability the computed q 1, q 2,..., q k gradually lose their orthogonality however it is not difficult to fix Gram Schmidt by reorthogonalization, essentially by applying Gram Schmidt a second time to the output of the first round of Gram Schmidt q 1, q 2,..., q k in exact arithmetic, q 1, q 2,..., q k is already orthogonal and applying Gram Schmidt a second time has no effect but in the presence of rounding error, reorthogonalization has real effect making the output of the second round orthogonal the nice thing is that there is no need to do a third round of Gram Schmidt twice suffices (for subtle reasons) 5. modified Gram Schmidt algorithm we didn t discuss this in lectures but I m adding this discussion of modified Gram Schmidt, a way to improve the numerical stability of Gram Schmidt note that q k can be rewritten as q k = 1 a k (q ja k )q j = 1 a k q j q ja k = 1 I q j q j a k if we define P i = q i q i Cn n, then P i is an orthogonal projector that satisfies Pi 2 = P i and P i P j = 0 if i j we can write q k = 1 I j=0 P j a k = 1 k 1 (I P j )a k although the classical Gram Schmidt process is numerically unstable, the modified Gram Schmidt method partially alleviates this difficulty note that A = QR = [ r 11 q 1 r 12 q 1 + r 22 q 2 ] we define which means A (k) = q i r T i, r T i = r i1 r i2 r ii i=1 A q i r T i = [ 0 A (k)] i=1 5

6 if we write then we then compute A (k) = [ z B ] = z 2, q k = 1 z [ rk,k+1 r k,n ] = q T k B which yields A (k+1) = B q k [ r1k ] this process is numerically more stable than Gram Schmidt although still not as good as Householder or Givens qr 6. back substitution backsolve or back substitution refers to a simple, intuitive way of solving linear systems of the form Rx = y or Lx = y where R is upper-triangular and L is lower-triangular take Rx = y for illustration r 11 r 1n..... r nn start at the bottom and work out way up y n = r nn x n x 1 x n = y 1. y n y n 1 = r n 1,n x n + r n 1,n 1 x n 1. we get y 1 = r 11 x 1 + r 12 x r 1n x n x n = y n r nn x n 1 = y n 1 r n 1,n (y n /r n ) r n 1,n 1. this requires that 0 for all k = 1,..., n, which is guaranteed if R is nonsingular for example we could use qr factorization given A C n n nonsingular and b C n step 1: find qr factorization A = QR step 2: form y = Q b step 3: backsolve Rx = y to get x 7. general principle for factoring matrices it is easy to solve Ax = b if A is unitary or orthogonal (includes permutation matrices) A is upper- or lower-triangular (includes diagonal matrices) Ax = b with such A can be solved with O(n 2 ) flops 6

7 if A represents a special orthogonal matrix like the discrete Fourier or wavelet transforms, then Ax = b can in fact be solved in O(n log n) flops using algorithms like fast Fourier or fast wavelet transforms if A is not one of these forms, we factorize A into a product of matrices of these forms this includes all the basic matrix factorizations lu, qr, svd, evd actually to the above list, we could also add A is bidiagonal/tridiagonal (or banded, i.e., a ij = 0 if i j > b for some bandwidth b n) A is Toeplitz or Hankel, i.e., a ij = a i j or a ij = a i+j constant on the diagonals or the opposite diagonals A is semiseparable Ax = b with bidiagonal or tridiagonal A can be solved in O(n) flops Ax = b with Toeplitz or Hankel A can be solved in O(n 2 log n) flops these are often called structured matrices for example, a tridiagonal system b 1 c 1 0 x 1 d 1 a 2 b 2 c 2. x 2 d 2 a 3 b.. 3 x 3 = d cn a n b n x n d n may be solved by first computing c i i = 1, c b i = i c i b i a i c i = 2, 3,..., n 1, i 1 and i = 1, d b i i = d i a i d i 1 b i a i c i = 2, 3,..., n, i 1 followed by back substitution x n = d n, d i x i = d i c ix i+1, i = n 1, n 2,..., 1 in this course we will just restrict ourselves to unitary and triangular factors but we will discuss a general principle for solving linear systems and least squares problems based on rank-retaining factorizations that works with any structured matrices 7