LEAST SQUARES SOLUTION TRICKS

Transcription

1 LEAST SQUARES SOLUTION TRICKS VESA KAARNIOJA, JESSE RAILO AND SAMULI SILTANEN Abstract This handout is for the course Applications of matrix computations at the University of Helsinki in Spring 2018 We define the linear least squares solutions and minimum norm solutions We study their basic properties such as the relation to the normal equation and singular values Fitting of a linear model to noisy data is also considered as an example Contents 1 Least squares solution and minimum norm solution 1 2 Normal equation and the least squares solution 2 3 Fitting a linear model to noisy data 5 4 Computing the minimum norm solution 6 Appendix A The singular value decomposition 7 1 Least squares solution and minimum norm solution Let us define the least squares solution and minimum norm solution of the matrix equation Ax = b, where x 1 x 2 x = x n and the matrix A has size k n Date: Rn, b = b 1 b 2 b k Rk, Version 1 Suggestions and corrections could be send to jesserailo@helsinkifi 1

2 0 = d dt A( x + tw) b 2 t=0 2 VESA KAARNIOJA, JESSE RAILO AND SAMULI SILTANEN Definition 11 A vector x R n is called a least-squares solution of the equation Ax = b if (11) A x b = min Ax b x Rn Furthermore, we give a special name for the shortest least-squares solution (in general there may be many least-squares solutions) A vector x 0 is called the minimum norm solution of Ax = b if x 0 is a least-squares solution of Ax = b and additionally satisfies (12) x 0 = min{ x : x is a least-squares solution of Ax = b} The vector norm above is the Euclidean norm x 2 = x 2 1 +x x 2 n In the next two sections we explain how to compute these solutions in practice 2 Normal equation and the least squares solution Recall the definitions of the following linear subspaces related to the matrix A: Ker(A) = {x R n : Ax = 0}, Range(A) = {b R k : there exists x R n such that Ax = b}, Coker(A) = (Range(A)) R k Consider the quadratic functional Q : R n R defined by Q(x) = Ax b 2 We want to find a minimizer x R n for Q In other words, we look for a vector x for which it holds that (21) Q( x) = min x R n Q(x) Note that Q is continuously differentiable in any variable x j Therefore, since x is a minimizer, we have for any w R n (Why?) We use the notation x, y for the inner product between two vertical vectors x R n and ỹ R n The definition is Note that x, y = x T y = x 1 y 1 + x 2 y x n y n x, x = x T x = x x x 2 n = x 2

3 LEAST SQUARES SOLUTION TRICKS 3 Also, use matrix algebra to see that Ax, b = (Ax) T b = (x T A T )b = x T (A T b) = x, A T b Now use the linearity of the inner product to compute 0 = d dt A( x + tw) b 2 t=0 = d dt t=0 A x + taw b, A x + taw b = d { A x 2 + 2t A x, Aw + t 2 Aw 2 dt 2t b, Aw 2 A x, b + b 2 } t=0 { = 2 A x, Aw + 2t Aw 2 2 b, Aw } t=0 =2 A x, Aw 2 b, Aw =2 A T A x, w 2 A T b, w We conclude that the identity A T A x, w = A T b, w holds for any nonzero w R n Therefore, the minimizing vector must satisfy (22) A T A x = A T b The identity (22) is called the normal equation Recall that if V R n is a linear subspace (ie a vector subspace), then V is defined using the rule: x V if and only if x, w for every w V Then also V is a linear subspace and every x R n has the unique decomposition as x = x 0 + x where x 0 V and x V One then says that R n = V V is an orthogonal sum of V and V Lemma 21 Let A : R n R k be a linear map Then Ker(A T ) = Range(A) Proof This is a simple observation using only definitions and y Range(A) y, x = 0 y Ker(A T ) A T y = 0, x Range(A) and the formula y, Aw = A T y, w Details are left as an exercise Theorem 21 Let A : R n R k be a linear map Then there exists at least one solution to the normal equation (22) for any b R k

4 4 VESA KAARNIOJA, JESSE RAILO AND SAMULI SILTANEN Proof It is sufficient to show that A T b Range(A T A) for every b R k, ie Range(A T ) Range(A T A) (Notice that it is a trivial observation that Range(A T A) Range(A T ) since Range(A) R k ) Let us decompose x R k into x = x 0 + x where x 0 Range(A) and x Range(A) Now A T x = A T (x 0 + x ) = A T x 0 + A T x = A T x 0 since x Ker(A T ) by Lemma 21 Hence A T (R k ) A T (Range(A)), ie Range(A T ) Range(A T A) This shows that the normal equation (22) A T Ax = A T b has always a solution Theorem 22 Let x R n Then x is a least squares solution of Ax = b if and only if x solves the normal equation (22) Proof We already proved the direction using calculus Let us prove the other direction now This proof is based on the orthogonal decomposition R k = Range(A) Range(A) and Lemma 21 If x R n solves the normal equation (22), then A T (A x b) = 0 Therefore A x b Ker(A T ) = Range(A) by Lemma 21 This means that A x b, y = 0 for every y Range(A) Let us decompose b = b 0 + b where b 0 Range(A) and b Range(A) We can conclude that since A x b 0 Range(A), and A x b, A x b 0 = 0 A x (b 0 + b ), b = b 2 since b Range(A) Adding up these two formulas gives directly that A x b 2 = b 2 The proof is complete if we show that Ax b 2 b 2 for every x R n Using that Ax + b 0 Range(A) and b Range(A), we see that Ax + b 0 and b are orthogonal to each others Therefore as desired Ax b 2 = Ax b b 2 b 2

5 LEAST SQUARES SOLUTION TRICKS 5 The above proof may seem a bit abstract at first sight, it however has a geometric nature A solution of the normal equation approximates perfectly the part of b in Range(A), and the total error is equal to the part of b in Range(A) Moreover, we showed in the end of the proof that this is the best approximation of b that can be obtained in the range of A, ie a least squares solution One can also notice that the formula (42) gives a concrete equivalent way to look at least squares solutions Corollary 21 If the n n matrix A T A is invertible, the least squares solution of Ax = b is given by (23) x = (A T A) 1 A T b If A T A is not invertible, there is no unique minimizer for Q and we cannot use formula (23) But even in that case we can compute the minimum norm solution! 3 Fitting a linear model to noisy data Consider the following linear model describing the relationship between two scalar quantities x R and x R: (31) y = a 0 x + b 0, where a 0, b 0 R are parameters Assume given noisy data y 1, y 2,, y n at points x 1, x 2,, x n More precisely, (32) y j = ax j + b + ε j, where ε j is some unknown error in the measurement We can solve for the parameters a, b R that give the model of the form (31) that best fits the data in the least-squares sense Namely, write x 1 1 y 1 x 2 1 y 2 A = Rn, y = x n 1 y n and consider the linear system of equations defined by [ ] a (33) A = y b Rn, Now in general the equation (33) has no solutions because of the errors in (32) But if the matrix (A T A) is invertible, then we can use (23)

6 6 VESA KAARNIOJA, JESSE RAILO AND SAMULI SILTANEN to compute the least-squares solution as [ ] ã = (A b T A) 1 A T y 4 Computing the minimum norm solution We need a method for computing minimum norm solutions For this, write A in the form of its svd A = UDV T as explained in Section A Recall that the singular values are ordered from largest to smallest as shown in (A4), and let r be the largest index for which the corresponding singular value is nonzero: (41) r = max{j 1 j min(k, n), d j > 0} The definition of index r is essential in the following analysis, so we will be extra-specific: d 1 > 0, d 2 > 0, d r > 0, d r+1 = 0, d min(k,n) = 0 Of course, it is also possible that all singular values are zero, in which case r is not defined and A is the zero matrix, or none of the singular values may be zero The next result gives a method to determine the minimum norm solution Theorem 41 Let A be a k n matrix and denote by A = UDV T the singular value decomposition of A The minimum norm solution of the equation Ax = b is given by A + b where A + b = V D + U T b, and where 1/d /d 2 D + = 1/d r R n k Proof Write the singular matrix V in the form V = [V 1 V 2 V n ] and note that the column vectors V 1,, V n form an orthogonal basis for R n We write x R n as a linear combination x = n j=1 a jv j = V a, and our goal is to find such coefficients a 1,, a n that x becomes a minimum norm solution

7 LEAST SQUARES SOLUTION TRICKS 7 Set b = U T b R k and compute (42) Ax b 2 = UDV T V a Ub 2 = Da b 2 = r k (d j a j b j) 2 + (b j) 2, j=1 j=r+1 where we used the orthogonality of U (namely, Ux = x for any vector x R k ) Now since d j and b j are given and fixed, the expression (42) attains its minimum when a j = b j/d j for j = 1,, r So any x of the form d 1 1 b 1 x = V d 1 r b r a r+1 is a least-squares solution The smallest norm x is clearly given by the choice a j = 0 for r < j n, so the minimum norm solution is uniquely determined by the formula a = D + b a n Definition 41 The matrix A + Moore-Penrose inverse of A is called the pseudoinverse, or the Appendix A The singular value decomposition We know from matrix algebra that any matrix A R k n can be written in the form (A1) A = UDV T, where U R k k and V R n n are orthogonal matrices, that is, U T U = UU T = I, V T V = V V T = I, and D R k n is a diagonal matrix The right side of (A1) is called the singular value decomposition (svd) of matrix A, and the diagonal elements d j are the singular values of A The properties of d j, and the columns u i of U, and the columns V i of V correspond to those of the SVE

8 8 VESA KAARNIOJA, JESSE RAILO AND SAMULI SILTANEN In the case k = n the matrix D is square-shaped: D = diag(d 1,, d k ) If k > n then d d 2 [ ] diag(d1,, d (A2) D = n ) = 0 (k n) n 0 d n, and in the case k < n the matrix D takes the form (A3) D = [diag(d 1,, d k ), 0 k (n k) ] d d = 2 0 d k 0 0 The diagonal elements d j are nonnegative and in decreasing order: (A4) d 1 d 2 d min(k,n) 0 Note that some or all of the d j can be equal to zero