AMS526: Numerical Analysis I (Numerical Linear Algebra)

Transcription

1 AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 7: More on Householder Reflectors; Least Squares Problems Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 15

2 Outline 1 Householder Reflectors 2 Linear Least Squares Problems Xiangmin Jiao Numerical Analysis I 2 / 15

3 Householder Triangularization Method introduced by Alston Scott Householder in 1958 It multiplies unitary matrices to make column triangular, e.g. x x x Q 1 0 x x 0 x x Q 2 x x 0 x Q 3 x 0 x x 0 x 0 0 x x 0 x 0 A Q 1 A Q 2 Q 1 A Q 3 Q 2 Q 1 A After n steps, we get a product of unitary matrices Q n Q 2 Q }{{} 1 A = R Q and in turn we get full QR factorization A = QR Q k introduces zeros below diagonal of kth column while preserving zeros below diagonal in preceding columns The key question is how to find Q k Xiangmin Jiao Numerical Analysis I 3 / 15

4 Householder Reflectors First, consider Q 1 : Q 1 a 1 = a 1 e 1, where e 1 = (1, 0,, 0) T. Why the length is a 1? Q 1 reflects a 1 across hyperplane H orthogonal to v = a 1 e 1 a 1, and therefore Q 1 = I 2 vv v v More generally, Q k = [ I 0 0 F where I is (k 1) (k 1) and F is (m k + 1) (m k + 1) such that F x = x 2 e 1, where x is (a k,k, a k,k+1,, a k,m ) T What is F? It has similar form as Q 1 with v = x e 1 x. ] Xiangmin Jiao Numerical Analysis I 4 / 15

5 Choice of Reflectors We could choose F such that F x = x e 1 instead of F x = x e 1, or more generally, F x = z x e 1 with z = 1 for z C This leads to an infinite number of possible QR factorizations of A If we require z R, we still have two choices Numerically, it is undesirable for v v to be close to zero for v = z x e 1 x, and v is larger if z = sign(x 1 ) Therefore, v = sign(x 1 ) x e 1 x. Since I 2 vv v v is independent of sign, clear out the factor 1 and obtain v = sign(x 1 ) x e 1 + x For completeness, if x 1 = 0, set z to 1 (instead of 0) Xiangmin Jiao Numerical Analysis I 5 / 15

6 Householder Algorithm Householder QR Factorization for k = 1 to n x = A k:m,k v k = sign(x 1 ) x e 1 + x v k = v k / v k A k:m,k:n = A k:m,k:n 2v k (v k A k:m,k:n) Note that sign(x) = 1 if x 0 and = 1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector v k is stored Xiangmin Jiao Numerical Analysis I 6 / 15

7 Householder Algorithm Householder QR Factorization for k = 1 to n x = A k:m,k v k = sign(x 1 ) x e 1 + x v k = v k / v k A k:m,k:n = A k:m,k:n 2v k (v k A k:m,k:n) Note that sign(x) = 1 if x 0 and = 1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector v k is stored Question: Can A be reused to store both R and v k completely? Xiangmin Jiao Numerical Analysis I 6 / 15

8 Householder Algorithm Householder QR Factorization for k = 1 to n x = A k:m,k v k = sign(x 1 ) x e 1 + x v k = v k / v k A k:m,k:n = A k:m,k:n 2v k (v k A k:m,k:n) Note that sign(x) = 1 if x 0 and = 1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector v k is stored Question: Can A be reused to store both R and v k completely? Answer: We can use lower diagonal portion of A to store all but one entry in each v k. So an additional array of size n is needed. Xiangmin Jiao Numerical Analysis I 6 / 15

9 Householder Algorithm Householder QR Factorization for k = 1 to n x = A k:m,k v k = sign(x 1 ) x e 1 + x v k = v k / v k A k:m,k:n = A k:m,k:n 2v k (v k A k:m,k:n) Note that sign(x) = 1 if x 0 and = 1 if x < 0 Leave R in place of A Matrix Q is not formed explicitly but reflection vector v k is stored Question: Can A be reused to store both R and v k completely? Answer: We can use lower diagonal portion of A to store all but one entry in each v k. So an additional array of size n is needed. Question: What happens if v k is 0 in line 3 of the loop? Xiangmin Jiao Numerical Analysis I 6 / 15

10 Applying or Forming Q Compute Q b = Q n Q 1 b Implicit calculation of Q b for k = 1 to n b k:m = b k:m 2v k (v k b k:m) Xiangmin Jiao Numerical Analysis I 7 / 15

11 Applying or Forming Q Compute Q b = Q n Q 1 b Implicit calculation of Q b for k = 1 to n b k:m = b k:m 2v k (v k b k:m) Compute Qx = Q 1 Q 2 Q n x Implicit calculation of Qx for k = n downto 1 x k:m = x k:m 2v k (v k x k:m) Question: How to form Q and ˆQ, respectively? Xiangmin Jiao Numerical Analysis I 7 / 15

12 Applying or Forming Q Compute Q b = Q n Q 1 b Implicit calculation of Q b for k = 1 to n b k:m = b k:m 2v k (v k b k:m) Compute Qx = Q 1 Q 2 Q n x Implicit calculation of Qx for k = n downto 1 x k:m = x k:m 2v k (v k x k:m) Question: How to form Q and ˆQ, respectively? Answer: Apply x = I m m or first n columns of I, respectively Xiangmin Jiao Numerical Analysis I 7 / 15

13 Operation Count Most work done at step A k:m,k:n = A k:m,k:n 2v k (v k A k:m,k:n) Flops per iteration: 2(m k)(n k) for dot products v k A k:m,k:n (m k)(n k) for outer product 2v k ( ) (m k)(n k) for subtraction 4(m k)(n k) total Including outer loop, total flops is n 4(m k)(n k) = 4 k=1 n (mn km kn + k 2 ) k=1 4mn 2 4(m + n)n 2 /2 + 4n 3 /3 = 2mn n3 If m n, it is more efficient than Gram-Schmidt method, but if m n, similar to Gram-Schmidt Xiangmin Jiao Numerical Analysis I 8 / 15

14 Givens Rotations Instead of using reflection, we can rotate x to obtain x e 1 [ ] cos θ sin θ A Given rotation R = rotates x R sin θ cos θ 2 counterclockwise by θ Choose θ to be angle between (x i, x j ) T and (1, 0) T, and we have [ ] [ ] cos θ sin θ xi = sin θ cos θ x j [ x 2 i 0 + x 2 j ] where cos θ = x 2 i x i + x 2 j, sin θ = x j xi 2 + xj 2 Xiangmin Jiao Numerical Analysis I 9 / 15

15 Givens QR Introduce zeros in column bottom-up, one zero at a time (4,5) (3,4) x x x x x x 0 x x 0 x x... To zero a ij, left-multiply matrix F with F i:i+1,i:i+1 being rotation matrix and F kk = 1 for k i, i + 1 Flop count of Givens QR is 3mn 2 n 3, which is about 50% more expensive than Householder QR Xiangmin Jiao Numerical Analysis I 10 / 15

16 Outline 1 Householder Reflectors 2 Linear Least Squares Problems Xiangmin Jiao Numerical Analysis I 11 / 15

17 Linear Least Squares Problems Overdetermined system of equations Ax b, where A has more rows than columns and has full rank, in general has no solutions Example application: Polynomial least squares fitting In general, minimize the residual r = b Ax In terms of 2-norm, we obtain linear least squares problem: Given A C m n, m n, and b C m, find x C n such that b Ax 2 is minimized If A has full rank, the minimizer x is the solution to the normal equation A Ax = A b or in terms of the pseudoinverse A +, x = A + b, where A + = (A A) 1 A C n m Xiangmin Jiao Numerical Analysis I 12 / 15

18 Geometric Interpretation Ax is in range(a), and the point in range(a) closest to b is its orthogonal projection onto range(a) Residual r is then orthogonal to range(a), and hence A r = A (b Ax) = 0 Ax is orthogonal projection of b, where x = A + b, so P = AA + = A(A A) 1 A is orthogonal projection (recall lecture 5) Xiangmin Jiao Numerical Analysis I 13 / 15

19 Solution of Lease Squares Problems One approach is to solve normal equation A Ax = A b directly using Cholesky factorization Is unstable, but is very efficient if m n (mn n3 ) More robust approach is to use QR factorization A = ˆQ ˆR b can be projected onto range(a) by P = ˆQ ˆQ, and therefore ˆQ ˆRx = ˆQ ˆQ b Left-multiply by ˆQ and we get ˆRx = ˆQ b (note A + = ˆR 1 ˆQ ) Least squares via QR Factorization Compute reduced QR factorization A = ˆQ ˆR Compute vector c = ˆQ b Solve upper-triangular system ˆRx = c for x Computation is dominated by QR factorization (2mn n3 ) Question: If Householder QR is used, how to compute ˆQ b? Xiangmin Jiao Numerical Analysis I 14 / 15

20 Solution of Lease Squares Problems One approach is to solve normal equation A Ax = A b directly using Cholesky factorization Is unstable, but is very efficient if m n (mn n3 ) More robust approach is to use QR factorization A = ˆQ ˆR b can be projected onto range(a) by P = ˆQ ˆQ, and therefore ˆQ ˆRx = ˆQ ˆQ b Left-multiply by ˆQ and we get ˆRx = ˆQ b (note A + = ˆR 1 ˆQ ) Least squares via QR Factorization Compute reduced QR factorization A = ˆQ ˆR Compute vector c = ˆQ b Solve upper-triangular system ˆRx = c for x Computation is dominated by QR factorization (2mn n3 ) Question: If Householder QR is used, how to compute ˆQ b? Answer: Compute Q b (where Q is from full QR factorization) and then take first n entries of resulting Q b Xiangmin Jiao Numerical Analysis I 14 / 15

21 Solution by SVD Using A = Û ˆΣV, b can be projected onto range(a) by P = ÛÛ, and therefore Û ˆΣV x = ÛÛ b Left-multiply by Û and we get ˆΣV x = Û b Least squares via SVD Compute reduced SVD factorization A = Û ˆΣV Compute vector c = Û b Solve diagonal system ˆΣw = c for w Set x = V w Work is dominated by SVD, which is 2mn n 3 flops, very expensive if m n Best numerical stability Question: If A is rank deficient, how to solve Ax b? Xiangmin Jiao Numerical Analysis I 15 / 15

22 Solution by SVD Using A = Û ˆΣV, b can be projected onto range(a) by P = ÛÛ, and therefore Û ˆΣV x = ÛÛ b Left-multiply by Û and we get ˆΣV x = Û b Least squares via SVD Compute reduced SVD factorization A = Û ˆΣV Compute vector c = Û b Solve diagonal system ˆΣw = c for w Set x = V w Work is dominated by SVD, which is 2mn n 3 flops, very expensive if m n Best numerical stability Question: If A is rank deficient, how to solve Ax b? Answer: x is no longer unique. Constrain x to be orthogonal to null space of A. Xiangmin Jiao Numerical Analysis I 15 / 15