MATH2071: LAB #7: Factorizations

Transcription

1 1 Introduction MATH2071: LAB #7: Factorizations Introduction Exercise 1 Orthogonal Matrices Exercise 2 A Set of Questions Exercise 3 The Gram Schmidt Method Exercise 4 Gram-Schmidt Factorization Exercise 5 Householder Matrices Exercise 6 Householder Factorization Exercise 7 The QR Method for Linear Systems Exercise 8 The singular value decomposition Exercise 9 Exercise 10 We have seen one factorization PLU that can be used to solve a linear system provided that the system is square, and that it is nonsingular, and that it is not too badly conditioned. However, if we want to handle problems with a bad condition number, or that are singular, or even rectangular, we are going to need to come up with a different approach. In this lab, we will look at two versions of the QR factorization: A = QR where Q is an orthogonal matrix, and R is upper triangular, and also the singular value decomposition (SVD) that factors a matrix into the form A = USV T. We will see that the QR factorization can be used to solve the same problems that the PLU factorization handles, but can be extended to do several other tasks for which the PLU factorization is useless. In situations where we are concerned about controlling roundoff error, the QR factorization can be more accurate than the PLU factorization, and is even a little easier to use when solving linear systems, since the inverse of Q is simply Q T. We will also see that the SVD can be used to compress data, to identify the range and nullspace of a linear operator, and even to solve non-square systems. You may find it convenient to print the pdf version of this lab rather than the web page itself. This lab will take three sessions. 2 Orthogonal Matrices Definition: An orthogonal matrix is a real matrix whose inverse is equal to its transpose. By convention, an orthogonal matrix is usually denoted by the symbol Q. The definition of an orthogonal matrix immediately implies that QQ T = Q T Q = I. From the definition of an orthogonal matrix, and from the fact that the L 2 vector norm of x can be defined by: x 2 = (x T x) and the fact that (Ax) T = x T A T you should be able to deduce that, for orthogonal matrices, Qx 2 = x 2 1

2 If I multiply x by Q and its L 2 norm doesn t change, then Qx must lie on the circle whose radius is x. In other words, the only thing that has happened is that Qx has been rotated around the origin by some angle, or reflected through the origin. That means that an orthogonal matrix represents a rotation or reflection. When matrices are complex, the term unitary is an analog to orthogonal. A matrix is unitary if UU H = I, where the H superscript refers to the Hermitian or conjugate-transpose of the matrix. In Matlab, the prime operator constructs the Hermitian and the dot-prime operator constructs the transpose. A real matrix that is unitary is actually orthogonal. 3 A Set of Questions Suppose we have N vectors x i, each of dimension M. Here are some common and related questions: Can we determine if the vectors are linearly independent? Can we determine the dimension of the space the vectors span? (This is the rank.) Can we construct an orthonormal basis for the space spanned by the vectors? Can we construct an orthonormal basis for the space of vectors perpendicular to all the x i? Given a new vector x N+1, can we determine whether this vector lies in the space spanned by the other vectors? If a new vector lies in the space, what specific linear combination of the old vectors is equal to the new one? If we create an M by N matrix using the vectors as columns, what is the rank of the matrix? How can we solve a rectangular system of linear equations? Since we actually plan to do these calculations numerically we also need to be concerned about how we deal with numerical error. For instance, in exact arithmetic, it s enough to say a matrix is not singular. In numerical calculations, we would like to be able to say that a particular matrix, while not singular, is nearly singular. 4 The Gram Schmidt Method The Gram Schmidt method can be thought of as a process that analyzes a set of vectors X, producing an equivalent (and possibly smaller) set of vectors Q that span the same space, have unit L 2 norm, and are pairwise orthogonal. Note that if we can produce such a set of vectors, then we can easily answer many of the questions in our set of problems. In particular, the size of the set Q tells us whether the set X was linearly independent, and the dimension of the space spanned and the rank of a matrix constructed from the vectors. And of course, the vectors in Q are the orthonormal basis we were seeking. So you should believe that being able to compute the vectors Q is a valuable ability. In fact, the Gram-Schmidt method can help us with all the problems on our list. We will be using the Gram-Schmidt process to factor a matrix, but the process itself pops up repeatedly whenever sets of vectors must be made orthogonal. You may see, for example, it in Krylov space methods for iteratively solving systems of equations and for eigenvalue problems. In words, the Gram-Schmidt process goes like this: 1. Start with no vectors in Q; 2. Consider x, the next (possibly the first) vector in X ; 2

3 3. For every vector q i already in Q, compute the projection of q i on x (i.e., q i x) and subtract it from x to get a vector orthogonal to q i ; 4. Compute the norm of (what s left of) x. If the norm is zero (or too small), discard x from Q; otherwise, divide x by its norm and move it from X to Q. 5. If there are more vectors in X, return to step When you get here, X is empty and you have an orthonormal set of vectors Q spanning the same space as the original set X. Here is a sketch of the Gram Schmidt process as an algorithm. Assume that n x is the number of x vectors: n q = 0 % n q will become the number of q vectors for k = 1 to n x for l = 1 to n q % if n q = 0 the loop is skipped r lk = q l x k x k = x k - r lk q l end r kk = x k x k if r kk > 0 n q = n q + 1 q nq = x k /r kk end end You should be able to match this algorithm to the previous verbal description. Can you see how the L 2 norm of a vector is being computed? Exercise 1: (a) Implement the Gram-Schmidt process in an m-file called gram schmidt.m. Your function should have the signature: function Q = gram_schmidt ( X ) % Q = gram_schmidt ( X ) % more comments % your name and the date Assume the x vectors are stored as columns of the matrix X. Be sure you understand this data structure because it is a potential source of confusion. In the algorithm description above, the x vectors are members of a set X, but here these vectors are stored as the columns of a matrix X. Similarly, the vectors in the set Q are stored as the columns of a matrix Q. The first column of X corresponds to the vector x 1, so that the Matlab expression X(k,1) refers to the k th component of the vector x 1, and the Matlab expression X(:,1) refers to the Matlab column vector corresponding to x 1. Similarly for the other columns. Include your code in your summary report. (b) It is always best to test your code on simple examples for which you know the answers. Test your code using the following input: X = [ ] You should be able to see that the correct result is the identity matrix. If you do not get the identity matrix, find your bug before continuing. 3

4 (c) Another simple test you can do in your head is the following: X = [ ] The columns of Q should have L 2 norm 1, and be pairwise orthogonal, and you should be able to confirm these facts by inspection. (d) Test your Gram-Schmidt code to compute a matrix Q using the following input: X = [ ] If your code is working correctly, you should compute approximately: [ ] You should verify that the columns of Q have L 2 norm 1, and are pairwise orthogonal. If you like programming problems, think of a way to do this check in a single line of Matlab code. (e) Show that the matrix Q you just computed is not an orthogonal matrix, even though its columns form an orthornormal set. Are you surprised? 5 Gram-Schmidt Factorization We need to take a closer look at the Gram-Schmidt process. Recall how the process of Gauss elimination could actually be regarded as a process of factorization. This insight enabled us to solve many other problems. In the same way, the Gram-Schmidt process is actually carrying out a different factorization that will give us the key to other problems. Just to keep our heads on straight, let me point out that we re about to stop thinking of X as a bunch of vectors, and instead regard it as a matrix. Since it is traditional for matrices to be called A, that s what we ll call our set of vectors from now on. Now, in the Gram-Schmidt algorithm, the numbers that we called r lk and r kk, that we computed, used, and discarded, actually record important information. They can be regarded as the nonzero elements of an upper triangular matrix R. The Gram-Schmidt process actually produces a factorization of the matrix A of the form: A = QR Here, the matrix Q has the same M N shape as A, so it s only square if A is. The matrix R will be square (N N), and upper triangular. Now that we re trying to produce a factorization of a matrix, we need to modify the Gram-Schmidt algorithm of the previous section. Every time we consider a vector x k at the beginning of a loop, we will now always produce a vector q k at the end of the loop, instead of dropping some out. Hence, if r kk, the norm of vector x k, is zero, we will simply set q k to the zero vector. Exercise 2: (a) Make a copy of the m-file gram schmidt.m and call it gs factor.m. Modify it to compute the Q and R factors of a (possibly) rectangular matrix A. It should have the signature function [ Q, R ] = gs_factor ( A ) % [ Q, R ] = gs_factor ( A ) % more comments 4

5 % your name and the date Include a copy of your code with your summary. (b) Compute the QR factorization of the following matrix. A = [ ] You computed Q above, and you should get the same matrix Q again. Check that A = QR: if so, your code is correct. (c) Compute the QR factorization of a matrix of random numbers (A=rand(100,100)). (Hint: use norms to check the equalities.) Is it true that Q T Q = I? (Hint: Does norm(q *Q-eye(100), fro ) equal 0?) Is it true that QQ T = I? Is Q orthogonal? Is the matrix R upper triangular? (Hint: the Matlab functions tril or triu can help, so you don t have to look at all those numbers.) Is it true that A = QR? (d) Compute the QR factorization of the following matrix. A = [ ] Is it true that Q T Q = I? Is it true that QQ T = I? Is Q orthogonal? Is the matrix R upper triangular? Is it true that A = QR? Exercise 3: One step in the Gram-Schmidt process avoids dividing by zero by checking that r kk > 0. This is, in fact, very poor practice! The reason is that r kk might be of roundoff size. Find the matrices Q and R for the matrix A = [ *eps ] Are the columns of your Q matrix orthonormal? (Recall that eps is the smallest number you can add to 1.0 and change its value.) The solution to this problem is more art than science, and is beyond the scope of this lab. One feasible way to fix the problem would be to replace zero with, say, eps*max(max(r)). 5

6 6 Householder Matrices It turns out that it is possible to take a given vector d and find a matrix H so that the product Hd is proportinal to the vector e 1 = (1, 0,..., 0) T. That is, it is possible to find a matrix H that zeros out all but the first entry of d. The matrix is given by v = d d e 1 (d d e 1 ) H = I 2vv T (1) Note that v T v is a scalar and vv T is a matrix. To see why Equation (1) is true, denote d = (d 1, d 2,...) T and do the following calculation. Hd = d 2 (d d e 1)(d d e 1 ) T (d d e 1 ) T (d d e 1 ) d = d 2 d 2 2d 1 d ) + d 2 (d d e 1)( d 2 d d 1 ) = d 2( d 2 d d 1 ) (2 d 2 2d 1 d ) (d d e 1) = d e 1 You can find further discussion in Atkinson, page 601, and in several places on the web. For example, a Planet Math (planetmath.org) encyclopedia article. Atkinson, on page 611, describes a way to use this idea to take any column of a matrix and make those entries below the diagonal entry be zero. Repeating this sequentially for each column will result in an upper triangular matrix. Atkinson s discussion can be summarized in the following way: Starting with a column vector of length n [ ] c b =, d where c is a column vector of length k 1 and d is a column vector of length n k + 1. The objective is to construct a matrix H with the property that c H b = 0,. where the k th component of the product, (H b) k, is non-zero. The Householder matrix H will be constructed from a vector [ ] 0 w = v where the vector v is the same size as d. The algorithm for constructing v and hence w can be summarized as follows. 1. Set α = ± d with signum(α) = signum(d 1 ) (Atkinson s Equations (9.3.5) and (9.3.9).) (The sign is arbitrary and chosen to minimize roundoff errors.) [ 1 2. Set v 1 = 2 1 d 1 ] α (Atkinson s Equation (9.3.8).) 6

7 3. Set p = αv 1 (Atkinson s Equation (9.3.7).) 4. Set v j = dj 2p for j = 2, 3,... (Atkinson s Equation (9.3.10).) [ ] 0 5. Set w =. v 6. Set H = I 2ww T. In the following exercise you will write a Matlab function to implement this algorithm. Exercise 4: (a) Start a function m-file named householder.m with signature and beginning lines function H = householder(b, k) % H = householder(b, k) % more comments % your name and the date n = length(b); d(:,1) = b(k:n); if d(1)>=0 alpha = -norm(d); else alpha = norm(d); end (b) Add comments at the beginning, being sure to explain the use of the variable k. (c) In the case that alpha is exactly zero, the algorithm will fail because of a division by zero. For the case that alpha is zero, return H = eye(n); otherwise, complete the above algorithm. (d) Test your function on the vector b=[10;9;8;7;6;5;4;3;2;1]; with k=1 and again with k=5. Check that H should be orthogonal and H*b should have zeros in positions below k. (Hint: Orthogonality of H is equivalent to the vector w being a unit vector. If H is not orthogonal, check w.) This householder function can be used for the QR factorization of a matrix by proceeding through a series of partial factorizations A = Q k R k, where Q 0 is the identity matrix, and R 0 is the matrix A. When we begin the k th step of factorization, our factor R k 1 is only upper triangular in columns 1 to k 1. Our goal on the k-th step is to find a better factor R k that is upper triangular through column k. If we can do this process n 1 times, we re done. Suppose, then, that we ve partially factored the matrix A, up to column k 1. In other words, we have a factorization A = Q k 1 R k 1 for which the matrix Q k 1 is orthogonal, but for which the matrix R k 1 is only upper triangular for the first k 1 columns. To proceed from our partial factorization, we re going to consider taking a Householder matrix H, and inserting it into the factorization as follows: A = Q k 1 R k 1 = Q k 1 H T HR k 1 7

8 Then, if we define Q k = Q k 1 H T it will again be the case that: R k = HR k 1 A = Q k R k and we re guaranteed that Q k is still orthogonal. Our construction of H guarantees that R k is actually upper triangular all the way through column k. Exercise 5: In this exercise you will see how the Householder matrix can be used in the steps of an algorithm that passes through a matrix, one column at a time, and turns the bottom of each column into zeros. (a) Define the matrix A to be the transpose of the Frank matrix of order 5. (b) Compute H1, the Householder matrix that knocks out the subdiagonal entries in column 1 of A, and then compute A1=H1*A. Are there any non-zero values below the diagonal in the first column? (c) Now let s compute H2, the Householder matrix that knocks out the subdiagonal entries in column 2 of A1, and compute A2=H2*A1. This matrix should have subdiagonal zeros in column 2. You should be convinced that you can zero out any subdiagonal column you want. Since zeroing out one subdiagonal column doesn t affect the previous columns, you can proceed sequentially to zero out all subdiagonal columns. 7 Householder Factorization For a rectangular M by N matrix A, the Householder QR factorization has the form A = QR where the matrix Q is M M (hence square and truly orthogonal) and the matrix R is M N, and upper triangular (or upper trapezoidal if you want to be more accurate.) If the matrix A is not square, then this definition is different from the Gram-Schmidt factorization we discussed before. The obvious difference is the shape of the factors. Here, it s the Q matrix that is square. The other difference, which you ll have to take on faith, is that the Householder factorization is generally more accurate (smaller arithmetic errors), and easier to define compactly. Householder QR Factorization Algorithm: Q = I; R = A; for k = 1:min(m,n) Construct the Householder matrix H for column k of the matrix R; Q = Q * H ; R = H * R; end Exercise 6: (a) Write an m-file h factor.m. It should have the form function [ Q, R ] = h_factor ( A ) % [ Q, R ] = h_factor ( A ) % more comments % your name and the date 8

9 Use the routine householder.m in order to compute the H matrix that you need at each step. (b) A simple test is the matrix A = [ ] You should see that simply interchanging the first and second columns of A turns it into an upper triangular matrix, so you can guess what the Householder matrices will be and what the solution is. Be sure that Q*R is A. (c) Test your code by computing the QR factorization of the Hilbert matrix of order 4. Compare your results with those of the Matlab qr function. 8 The QR Method for Linear Systems If we have computed the Householder QR factorization of a matrix without encountering any singularities, then it is easy to solve linear systems. We use the property of the Q factor that Q T Q = I: Ax = b QRx = b Q T QRx = Q T b Rx = Q T b (2) so all we have to do is form the new right hand side Q T b and then solve the upper triangular system. And that s easy because it s the same as the u solve step of our old PLU solution algorithm from lab 6. Exercise 7: Make a copy of your file u solve.m from lab 6 (upper-triangular solver), or get a copy of my version. Write a file called h solve.m. It should have the signature function x = h_solve ( Q, R, b ) % x = h_solve ( Q, R, b ) % more comments % your name and the date The matrix R is upper triangular, so you can use u solve to solve (2) above. Assume that the QR factors come from the h factor routine. Set up your code to compute Q T b and then solve the upper triangular system Rx = Q T b using u solve. When you think your solver is working, test it out on a system as follows: n = 5 A = frank ( n ) x = [ 1 : n ] b = A * x [ Q, R ] = h_factor ( A ) x2 = h_solve ( Q, R, b ) norm(x - x2) % should be close to zero. 9

10 9 The singular value decomposition One of the most important matrix product decompositions is the singular value decomposition. According to Atkinson s Theorem 7.5 (page 478), if A is an m n matrix, then there are matrices U, S and V, with A = USV T, (3) where U is n n and unitary, and V is m m and unitary. Recall that unitary matrices are like orthogonal matrices but they could have complex entries. Unitary matrices with all real entries are orthogonal matrices. The matrix S is n m and is of the form { σ k for k = l S kl = 0 for k l and σ 1 σ 2 σ 3 σ r > 0 = 0 = = 0. The number r is the rank of the matrix A and is necessarily no larger than min(m, n). The singular values, σ k, are real and positive and are the eigenvalues of the Hermitian matrix A H A. The singular value decomposition provides much useful information about the matrix A. This information includes: Information about the range and nullspace of A regarded as a linear operator, including the dimensions of these spaces. A method of solving non-square matrix systems. A method of compressing the matrix A. You will see these facts illustrated below. Atkinson gives a proof of the SVD but it does not provide a numerically useful algorithm. Further, the obvious algorithm involves finding the eigenvalues of A H A, and this is not really practical because of roundoff difficulties. Matlab includes a function called svd with signature [U S V]=svd(A) to compute the singular value decomposition and we will be using it in the following exercises. This function uses the Lapack subroutine dgesvd, so if you were to need it in a Fortran or C program, it would be available by linking against the Lapack library. Supposing that r is the smallest integer so that σ r = 0, then the SVD implies that the rank of the matrix is r 1. Similarly, if the matrix A is n n, then the rank of the nullspace of A is n r + 1. The first r 1 columns of U are an orthonormal basis of the range of A, and the last n r + 1 columns of V are an orthonormal basis for the nullspace of A. Of course, in the presence of roundoff some judgement must be made as to what constitutes a zero singular value, but the SVD is the best (if expensive) way to discover the rank and nullity of a matrix. This is especially true when there is a large gap between the smallest of the large singular values and the largest of the small singular values. The SVD can also be used to solve a matrix system. Assuming that the matrix is non-singular, all singular values are strictly positive, and the SVD can be used to solve a system. b = Ax b = USV H x U H b = SV H x S + U H b = V H x (4) V S + U H b = x Where S + is the diagonal matrix whose entries are 1/σ k. It turns out that Equation (4) is not a good way to solve nonsingular systems, but is an excellent way to solve systems that are singular or almost singular! 10

11 To this end, the matrix S + is the matrix with the same shape as S T whose elements are given as { S + kl = 1/σ k for k = l and σ k > 0 0 otherwise Further, if A is close to singular, a similar definition but with diagonal entries 1/σ k for σ k > ɛ for some ɛ can work very nicely. In these latter cases, the solution is the least-squares best fit solution and the matrix A + is called the Moore-Penrose pseudo-inverse of A. Exercise 8: In this exercise you will use the Matlab svd function to solve for the best-fit linear function of several variables through a set of points. This is an example of solving a rectangular system. Imagine that you have been given many samples of related data involving several variables and you wish to find a linear relationship among the variables that approximates the given data in the best-fit sense. This can be written as a rectangular system a 1 d 11 + a 2 d 12 + a 3 d d 1n = 0 a 1 d 21 + a 2 d 22 + a 3 d d 2n = 0 a 1 d 31 + a 2 d 32 + a 3 d d 3n = where d ij represents the data and a i the desired coefficients. The best-fit solution can be found using the SVD. As we have done several times before, we will first generate a data set with known solution and then use svd to recover the known solution. Place Matlab code for the following steps into a script m-file called exer8.m (a) Generate a data set consisting of twenty samples of each of four variables using the following Matlab code. N=20; d1=rand(n,1); d2=rand(n,1); d3=rand(n,1); d4=4*d1-3*d2+2*d3-1; It should be obvious that these vectors satisfy the equation 4d 1 3d 2 + 2d 3 d 4 = 1, but, if you were solving a real problem, you would not be aware of this equation, and it would be your objective to discover it, i.e., to find the coefficients in the equation. (b) Introduce small errors into the data. The rand function produces numbers between 0 and 1. EPSILON=1.e-5; d1=d1.*(1+epsilon*rand(n,1)); d2=d2.*(1+epsilon*rand(n,1)); d3=d3.*(1+epsilon*rand(n,1)); d4=d4.*(1+epsilon*rand(n,1)); (c) Regard the four vectors d1,d2,d3,d4 as data given to you, and construct the matrix consisting of the four column vectors, A=[d1,d2,d3,d4]. (d) Use the Matlab svd function [U S V]=svd(A);. 11

12 Please include the four non-zero values of S in your summary, but not the matrices U and V. (e) Just to confirm that you have everything right, compute norm(a-u*s*v, fro ). This number should be roundoff-sized. (f) Construct the matrix S + (call it Splus) so that S + kl = { 1/S(k, k) for k = l and S(k, k) > 0 0 for k l or S(k, k) = 0 (g) We are seeking the coefficient vector x that x 1 d 1 + x 2 d 2 + x 3 d 3 + x 4 d 4 = 1. Find this vector by setting b=ones(n,1) and then using Equation (4) above. Include your solution with your summary. It should be close to the known solution x=[4;-3;2;-1]. Sometimes the data you are given turns out to be deficient because the variables supposed to be independent are actually related. This dependency will cause the coefficient matrix to be singular or nearly singular. Dealing with dependencies of this sort is one of the greatest difficulties in using least-squares fitting methods. In the following exercise you will construct a deficient set of data and see that the singular value decomposition can find the solution nonetheless. Exercise 9: (a) Copy your m-file exer8.m to exer9.m. Replace the line d3=rand(n,1); with the line d3=d1+d2; This introduces nonuniqueness into the solution for the coefficients. (b) After using svd to find U, S, and V, print S(1,1), S(2,2), S(3,3), S(4,4). You should see that S(4,4) is substantially smaller than the others. Treat it as if it were zero and find the coefficient vector x as before. The solution you found is probably not x=[4;-3;2;-1]. (c) Look at V(:,4). This is the vector associated with the singular value that you set to zero. This vector should be associated with the extra relationship d1+d2-d3=0 that was introduced in the first step. What multiple of V(:,4) can be added to your solution to yield [4;-3;2;-1]? In the following exercise you will see how the singular value decomposition can be used to compress a graphical figure by representing the figure as a matrix and then using the singular value decomposition to find the closest matrix of lower rank to the original. This approach can form the basis of efficient compression methods. Exercise 10: (a) This photo from NASA is one of the pictures the Mars rovers sent back. Download it. (b) Matlab provides various image processing utilities. In this case, read the image in and convert it to a matrix of real ( double ) numbers using the following commands. mars=double(imread( mars.jpg )); The variable mars will be a matrix of integers between 0 and 255. (c) Matlab has the notion of a colormap that determines how the values in the matrix will be colored. Use the command 12

13 colormap( bone ); to set a reasonable colormap. Now you can show the picture with the command image(mars) Please do not send me this plot. (d) Use the command [U S V]=svd(mars); to perform the singular value decomposition. Do not include these matrices with your summary! (e) Plot the singular values: plot(diag(s)). You should see that the final 300 singular values are essentially zero, and the singular values larger than 50 are still noticable. Please send me this plot with your summary. (f) Construct the three new matrices mars200=u(:,1:200)*s(1:200,1:200)*v(:,1:200) ; mars100=u(:,1:100)*s(1:100,1:100)*v(:,1:100) ; mars25=u(:,1:25)*s(1:25,1:25)*v(:,1:25) ; These matrices are of lower rank than the mars matrix, and can be stored in a more efficient manner. Do not include these matrices with your summary! (g) Plot all three files as images using the following commands; figure(1); colormap( bone ); image(mars); title( Original mars ); figure(2); colormap( bone ); image(mars25); title( mars with 25 singular values ); figure(3); colormap( bone ); image(mars100); title( mars with 100 singular values ); figure(4); colormap( bone ); image(mars200); title( mars with 200 singular values ); You should see essentially no difference between the original and the one with 200 singular values, while you should see degradation of the image in the case of 25 singular values. Please only include the 25 singular value case with your summary. 13