UCSD ECE269 Handout #8 Prof. Young-Han Kim Wednesday, February 7, Homework Set #4 (Due: Wednesday, February 21, 2018)

Transcription

1 UCSD ECE269 Handout #8 Prof. Young-Han Kim Wednesday, February 7, 2018 Homework Set #4 (Due: Wednesday, February 21, 2018) 1. Almost orthonormal basis. Let u 1,u 2,...,u n form an orthonormal basis for an inner product space V and let v 1,v 2,...,v n be a set of vectors in V such that Show that v 1,v 2,...,v n form a basis for V. u j v j < 1 n, j = 1,2,...,n. 2. Matrix inversion lemmas. Let A F n n, B F n k, C F k n, and D F k k. Suppose that A, D, and D CA 1 B are invertible. (a) Show that A 1 B(D CA 1 B) 1 = (A BD 1 C) 1 BD 1. (b) Show that (A BD 1 C) 1 = A 1 +A 1 B(D CA 1 B) 1 CA 1. (c) Similarly, suppose that A, D, and D +CA 1 B are invertible. Show that (A+BD 1 C) 1 = A 1 A 1 B(D +CA 1 B) 1 CA 1, which is sometimes referred to as the Woodbury matrix identity. (d) Let à Fn n be identical to A except that the (i,j)-th entry differs by δ, i.e., à = A+δe i e T j. Show that à 1 = A 1 1 1/δ +(A 1 f i gj T, ) ji where f i is the i-th column of A 1 and g T j is its j-th row. (Hint: Consider the block matrix and its inverse.) [ ] A B C D 3. Moore Penrose pseudoinverse. A pseudoinverse of A R m n is defined as a matrix A + R n m that satisfies and AA + and A + A are symmetric. AA + A = A, A + AA + = A +, 1

2 (a) Show that A + is unique. (b) Show that (A T A) 1 A T is the pseudoinverse and a left inverse of a full-rank tall matrix A (c) ShowthatA T (AA T ) 1 isthepseudoinverseandarightinverseofafull-rankfatmatrixa. (d) Show that A 1 is the pseudoinverse of a full-rank square matrix A. (e) Show that A is the pseudoinverse of itself for a projection matrix A (cf. Question 4 in Homework Set #3). (f) Show that (A T ) + = (A + ) T. (g) Show that (AA T ) + = (A + ) T A + and (A T A) + = A + (A + ) T. (h) Suppose that A has a rank decomposition A = BC, for example, B = Q R m r and C = R R r n as in the QR decomposition. Find A + in terms of B and C. (i) Show that R(A + ) = R(A T ) and N(A + ) = N(A T ). (j) Show that P = AA + and Q = A + A are projection matrices. (k) Show that y = Px and z = Qx are the projections of x onto R(A) and R(A T ), respectively, where P and Q are defined as in 3j. (l) Show that A + = lim δ 0 (A T A+δI) 1 A T = lim δ 0 A T (AA T +δi) 1. (m) Show that x = A + b is a least-squares solution to the linear equation Ax = b, i.e., Ax b Ax b for every other x. (n) Show that x = A + b is the least-norm solution to the linear equation Ax = b, i.e., x x for every other solution x, provided that a solution exists. 4. Least squares for an alternative inner product. (a) Let A R m n be full-rank and tall. Show that x,y A = Ax,Ay = x T A T Ay is a valid inner product. (b) Let B R n k be full-rank and tall. Find the unique solution to the following leastsquares problem minimize y Bx A, where v A = v,v A. Your answer should be in terms of y and B. 5. Recursive least squares. The least-squares problem for y = Ax can be viewed as finding the best fit for noisy observations y 1,y 2,...,y m from linear measurements ã T 1 x,ãt 2 x,...,ãt m x, where ã T 1,ãT 2,...,ãT m are rows of the measurement matrix A. We know that if A is full-rank and tall, ( m ) 1 m x m = (A T A) 1 A T y = ã i ã T i y i ã i is the least-squares solution. Now suppose that there is an additional measurement ã T m+1 x, which results in a new observation y m+1. The new least-squares solution can be found from scratch as x m+1 = ( m+1 2 ã i ã T i ) 1 m+1 y i ã i,

3 which is computational inefficient. This problem explores a low-complexity alternative that can compute the new solution x m+1 based on the old one x m, and can incorporate subsequent measurement outcomes recursively. (a) Let P m = ( m ãiã T i ) 1. Using Problem 2, show that P m+1 = [P 1 m +ã m+1 ã T m+1] 1 = P m P mã m+1 ã T m+1 P m (1+ã T m+1 P mã m+1 ). (b) Show that the solution x m+1 at the (m+1)-st iteration can be obtained as where x m+1 = x m +ǫ m+1 q m+1, q m+1 = P m+1 ã m+1 ǫ m+1 = y m+1 ã T m+1x m. 3

4 Programming Assignment Write down your code as clearly as possible and add suitable comments. 1. Least-qquares polynomial approximation. Consider a function φ : [a, b] R. Suppose that we wish to approximate φ(t) by a polynomial p(t) of degree n, using evaluations of the function y 1 = φ(t 1 ), y 2 = φ(t 2 ),. y m = φ(t m ) at distinct values t 1,t 2,...,t m [a,b]. The goodness of the approximation is measured by the squared error m J = (y i p(t i )) 2, and the goal is to find the polynomial p(t) = α 0 +α 1 t+α 2 t 2 + +α n t n that minimizes J. Let 1 t 1 t 2 1 t n 1 1 t 2 t 2 2 t n 2 A =..... Rm (n+1), 1 t m t 2 m tn m x = (α 0,α 1,α 2,...,α n ) R n+1, and y = (y 1,y 2,...,y m ) R m. Then the problem is equivalent to finding the least-squares solution to y = Ax. (1) The matrix A is a Vandermonde matrix and is full-rank (provided that t 1,t 2,...,t m are distinct. Depending on m and n, there are three possibilities. If m > n+1 (i.e., A is tall), there is no solution to (1) and we fit a large number of data points (t 1,y 1 ),...,(t m,y m ) to a low-degree polynomial. If m = n+1, A is invertible and there is a unique polynomial passing through all the data points. If m < n+1, there are infinitely many polynomials passing through all the data points. (a) Write a Julia function lspoly(t,y,n) that takes as input vectors y and t, each of length m and a positive integer n, and outputs the coefficients α 0,...,α n of a polynomial p of degree n, such that the squared error m (y i p(t i )) 2 is minimized. For m < n +1, your function should output the polynomial p that passes through all the data points such that n j=0 α2 j is minimized. 4

5 (b) Using the function in part (a), we now fit a polynomial to φ(t) = sint on [0,2π]. For this purpose, choose 45 points uniformly at random on [0,2π], using the seed 1234 for the random number generator. The seed can be set by the command srand(1234). We then divide this data into three subsets. The first 15 points will be used as the training set, the next 15 points as test set #1, and the last 15 points as test set #2 (to be used in a subsequent problem). Find a polynomial p n of degree at most n = 3, such that the error m (sint i p n (t i )) 2 over the training set is minimized, and plot the training set data points, the fitted polynomial q n, and the function φ together. Repeat the experiment for n = 5,6,7,10,12. Plot the relative training error m (sint i p n (t i )) 2 m (sint i) 2 (2) as a function of n. For plotting, you can choose a few more values of n other than the ones mentioned above. (Hint: the training error can be written as y Ax / y.) Comment on the plot. (c) For each n, using the polynomial p n found by training in part (b), compute the relative test error t i test set # 1 (sint i p(t i )) 2 t i test set # 1 (sint i) 2. (3) Plot the training and test errors as functions of n on the same plot. What do you observe? (d) Repeat parts (b) and (c) for φ(t) = tlnt on t [0.25,1.75], for n = 3,5,6,7,10,12, Regularization. As noted in Problem 1, if m n+1, then there is at least one polynomial passing through all the points. This sounds like a desirable result in theory, but in practice this usually means that our model is too complex for the amount of data that we have, or in other words, we are drawing strong conclusions based on insufficient evidence. Even for under-determined problems, our model might try to approximate the training data too closely, so that performance on a new test data suffers as a consequence of trying to improve performance on the training data. This phenomenon is called over-fitting and a popular way to mitigate this effect is to regularize the solution by penalizing models that are too complex. In this problem, we use the Tychonov regularization min x R n (4) ( y Ax 2 +λ x 2) (5) to trade off the goodness of the fit and the complexity of the solution. (a) Write a Julia function rlspoly(t,y,n,lambda) that takes as input vectors y and t, each of length m, a positive integer n, and a regularization parameter lambda, and outputs 5

6 the coefficients α 0,...,α n of a polynomial p of degree n, which is the solution to the Tychonov-regularized least squares polynomial fitting. (b) Fit φ(t) = sint using the training set defined in Problem 1(b), by the Tychonovregularized least-squares solution of degree 15. Find the relative error over test set #1 for lambda = [0.1, 0.2, 0.5, 0.7, 1.0, 2.0, 5.0, 5.5]. On the same figure, plot the relative error over test set #1 and the relative error over the training set, as functions of lambda (you can use more lambda-values, if necessary.) Comment on the plots. For the best lambda (as judged by the relative error over test set #1), plot the training set data points, the test set data points, and the fitted polynomial q n. Also report this best lambda (call it λ sin.) (c) Repeat part (b) for φ(t) = tlnt on [0.25,1.75], with n = 17 and lambda = [1.0, 2.0, 2.3, 2.5, 2.7, 3.5, 4.0, 4.5]. Let λ entropy be the optimal lambda that you obtain. 3. Recursive least-squares. Sometimes we would like to update an already-existing model to take into account new observations. Performing the whole computation again for adding a single extra data point is not very economical. In this scenario, we can update our model recursively to reflect the changes, rather than rebuilding it from scratch. (a) Write a Julia function lspolyupdate(tnew, ynew, xold, P) that takes as input scalars tnew and ynew, a vector of coefficients xold of length n + 1, and a matrix P of size (n + 1) (n + 1), and outputs the updated coefficient vector xnew and the updated matrix Pnew in accordance with Problem 5. (b) Starting from the polynomial p n (for n = 15) obtained by solving the (non-regularized) least-squares problem in Problem 1(b) and applying the function lspolyupdate() recursively, compute the polynomial p n of degree n = 15 that solves the least-squares problem over all the data points in the training set and test set #1 combined. We can consider this as an alternative way (in contrast to regularization) of utilizing the data in test set #1. Now, on the same figure, plot the training set data points, the test set #1 data points, the test set #2 data points, the polynomial p n, and the polynomial q n obtained in problem 2(b) (for lambda = λ sin.). Comment on the plots. Compare the relative errors of q n and p n over test set #2. Which approach is more effective in this case: adding more training data, or regularizing the solution? (c) Repeat part (b) for φ(t) = tlnt on [0.25,1.75] with n = 17. 6