y = A T x = A T (x r + x c ) = A T x r

Transcription

1 APPM/MATH 4660 Homework 1 Solutions This assignment is due on Google Drive by 11:59pm on Friday, January 30th Submit your solution writeup and any (well-commented) code that you produce according to the submission guidelines found on the course webpage Minimal credit will be given for incomplete solutions or solutions that do not provide details on how the solution is found You may discuss the problems with your classmates, but all work (analysis and code) must be your own 1 In class we talked etensively about the solution of the least-squares problem min b A for the case when A has full rank In this eercise you ll investigate what happens when A does not have full rank (a) Let A be a real m n matri Prove that for any system of equations, A = b, the normal equations A T A = A T b (and by etension, the least-squares problem) have a solution Make no assumptions about the sizes of m and n Argue that when A does not have full rank the least-squares problem has an infinite number of solutions Solution Method 1: First we need to remind ourselves of the Fundamental Theorem of Linear Algebra The FTLA lays out relationships between the four fundamental subspaces of a matri Let A be a real m n matri, then R m = range(a) coker(a) with range(a) coker(a) R n = corange(a) ker(a) with corange(a) ker(a) We ll show that A T A = A T b always has a solution by showing that A T b range ( A T A ) First we need to convince ourselves that range ( A T A ) = range ( A T ) We have range ( A T ) = { all y A T = y for some R m} but, by the FTLA, R m can be uniquely decomposed into r range(a) and c coker(a) Then y = A T = A T ( r + c ) = A T r Since r range(a) we have y = A T r = A T Az for some z R n Thus range ( A T ) = range ( A T A ) The remainder of the proof is straight forward A T b range ( A T ) = range ( A T A ) and so the normal equations have a solution Solution Method : We reformulate the normal equations as A T (A b) = 0 and argue that this system always has a solution First notice that b R m which means, by the FTLA, it can be decomposed uniquely into a vector b r range(a) and a vector b c coker(a) Since b r range(a) there eists particular solution such that A = b r Plugging this solution into the factored linear system we have A T (A b) = A T (A b r b c ) = A T b c = 0 where the last step follows from the fact that b c coker(a)

2 Note that the n n matri A T A is a Gram matri, so rank ( A T A ) = rank(a) If A has full-rank (ie rank(a) = n ) then rank ( A T A ) = n and thus A T A is nonsingular Since A T A is nonsingular the normal equations have a unique solution If A is not full-rank (ie rank(a) < n or m < n) then the normal equations have infinitely many solutions (b) The method of regularized least squares is often used in practice to solve the least squares problem when A does not have full rank The regularized least squares problem has the following form: min b A + λ where λ > 0 is a (usually small) parameter Show that the regularized least squares problem is equivalent to the following min b 0 A µi where µ = λ Solution: We start by writing out the sum of squared-norms in the first form of the regularized least-squares problem term by term We have b A = (b 1 a 11 1 a 1 a 1n n ) + (b a 1 1 a a n n ) + (b m a m1 1 a m a mn n ) λ = λ ( 1 ) + λ ( ) + λ ( ) = (µ 1 ) + (µ ) + (µ n ) Stacking these on top of each other we have b A + λ = (b 1 a 11 1 a 1 a 1n n ) + (b a 1 1 a a n n ) + (b m a m1 1 a m a mn n ) + (0 µ 1 ) + (0 µ ) + (0 µ n ) = b A 0 µi

3 (c) Derive the normal equations for the regularized least squares problem given in part (b) Solution: The regularized least-squares problem has the form b min ˆb  where ˆb = 0 and  = A µi Then the equivalent normal equations are given by  T  = ÂT ˆb A T µi A µi = A T µi b 0 ( A T A + λi ) = A T b (d) Prove that the normal equations of the regularized problem have a unique solution for every positive value of λ Solution Method 1: The matri A T A is a Gram matri and is thus symmetric positive semidefinite If λ > 0 then ( A T A + λi ) is symmetric positive definite and thus nonsingular Since ( A T A + λi ) is nonsingular the associated normal equations have a unique solution Solution Method : Notice that the matri the argument from part (a) A λi necessarily has full column-rank and repeat

4 In this problem you will eplore the stability of various methods for computing the QR Factorization of a matri (a) Write a computer code that takes an m n matri A and uses Gram-Schmidt to compute its condensed QR Factorization (b) Write a computer code that takes an m n matri A and uses Modified Gram-Schmidt to compute its condensed QR Factorization (c) Compute the QR Factorization of random n n matrices for n =, 4, 8, 18 using (i) Gram- Schmidt, (ii) Modified Gram-Schmidt, and (iii) a canned QR function from your favorite scripting language (which almost certainly uses Householder s method) For each matri and factorization, measure the orthogonality of the columns of Q using η = ˆQ T ˆQ I On a single set of aes, produce a log-log plot of n vs η for each factorization method What can you conclude from your results? Solution: The following figure shows n vs η for Gram-Schmidt, Modified Gram-Schmidt, and Householder QR From the figure we see that for all three methods the orthogonality of the columns of Q degrades only slightly as n increases We also notice that Naive Gram-Schmidt and Modified Gram-Schmidt perform only slightly worse than Householder QR This is because random matrices are typically well-conditioned, and thus even Naive Gram-Schmidt is an effective orthogonalization technique Remember that a matri A is ill-conditioned matrices when it s columns are nearly linearly dependent The odds of generating a random matri linearly dependent columns is infinitesimal

5 (d) Consider the matri A = ɛ ɛ ɛ Repeat the general idea of part (c), but this time keep the size of the matri fied as shown, and let ɛ decrease like ɛ = 10 l for l =, 3,, 10 Produce a log-log plot of ɛ vs η for each factorization What can you conclude about your results this time? Solution: The following figure shows ɛ vs η for Gram-Schmidt, Modified Gram-Schmidt, and Householder QR From the figure we can see that Gram-Schmidt does a terrible job of producing an orthogonal basis for any value of ɛ < 10 5 For much smaller epsilon the columns of Q produced by Naive Gram-Schmidt are not remotely orthogonal Modified Gram-Schmidt does considerably better, but still eperiences a minor loss of orthogonality for small ɛ Finally, Householder QR is entirely stable, producing a basis that is orthogonal to machine precision

6 3 Consider the data in the following table t i y i (a) Use the QR Factorization code you developed in Problem 1 to find the best least-squares fit of the given data using polynomials of degree 1,, and 3 For each model, report the least-squares error b A and plot the fitted curve along with the data Solution: The fitted models along with the data can be seen in the figure at the bottom of the page The least-squares errors for each polynomial fits are as follows poly degree least-squares error 1 39e+ 144e e-4 From the table (and the figures) we see that the quadratic and cubic models fit the data significantly better than the linear model As epected, the cubic model fits the data better than the quadratic model, but only slightly in terms of least-squares error (b) It is sometimes desirable to fit data using models other than polynomials An etremely popular choice is to fit the data to an eponential model of the form y = Ae Bt where here A and B are the model parameters you re trying to determine It turns out that you can can do this using nearly the same method as you used in part (a) Describe in detail how you can accomplish this (This method is called eponential regression in statistics and is covered in nearly every tetbook on numerical analysis See if you can work it out on your own, but don t be afraid to use Google Hint: Think logarithmically) Solution: Taking the natural log of both sides of the eponential model we have ln y = ln A + Bt We see that the logarithmic equation is linear in t and thus fits the form of the typical linear least-squares problem We can then write the minimization problem as min b M where M = 1 t 1 1 t 1 t n, = ln A B, and b = ln y 1 ln y ln y n Forming M and b we can solve the linear least-squares problem for using the usual method by QR decomposition of M Then the fitted model is y = Ae Bt where A = e 1 and B =

7 (c) Using the method you described in part (b) determine the best least-squares fit of the form y = Ae Bt to the given data Plot the fitted curve along with the data and report the least-squares error, which in this case is given by n i=1 ( yi Ae Bti ) Solution: Proceeding by the method described in (b) we find that A 46 and B 0374 The eponential least-squares error is found to be n ( yi Ae Bti) = 418e+ i=1 Notice that this is even worse than the linear polynomial fit found in part (a) Indeed, we can see from the figure that the eponential model does not fit the data particularly well This should not be particularly surprising The data clearly did not come from an eponential process (evidenced by the fact that a simple quadratic model eplained the data etremely well) so there is no reason to epect that an eponential model would eplain the data well