Notes on Solving Linear Least-Squares Problems

Transcription

1 Notes on Solving Linear Least-Squares Problems Robert A. van de Geijn The University of Texas at Austin Austin, TX 7871 October 1, 14 NOTE: I have not thoroughly proof-read these notes!!! 1 Motivation For a motivation of the linear least-squares problem, read Week 1 Sections of Linear Algebra: Foundations to Frontiers - Notes to LAFF With. The Linear Least-Squares Problem Let A C m n and y C m. Then the linear least-square problem LLS is given by Find x s.t. Ax y = min z C n Az y. In other words, x is the vector that minimizes the expression Ax y. Equivalently, we can solve Find x s.t. Ax y = min z C n Az y. If x solves the linear least-squares problem, then Ax is the vector in CA the column space of A closest to the vector y. 3 Method of Normal Equations Let A R m n have linearly independent columns which implies m n. Let f : R n R m be defined by fx = Ax y = Ax y T Ax y = x T A T Ax x T A T y y T Ax + y T y = x T A T Ax x T A T y + y T y. This function is minimized when the gradient is zero, fx =. Now, fx = A T Ax A T y. If A has linearly independent columns then A T A is nonsingular. Hence, the x that minimizes Ax y solves A T Ax = A T y. This is known as the method of normal equations. Notice that then x = A T A 1 A T y, }{{} A where A is known as the pseudo inverse or Moore-Penrose pseudo inverse. In practice, one performs the following steps: 1

2 Form B = A T A, a symmetric positive-definite SPD matrix. Cost: approximately mn floating point operations flops, if one takes advantage of symmetry. Compute the Cholesky factor L, a lower triangular matrix, so that B = LL T. This factorization, discussed in Week 8 Section 8.4. of Linear Algebra: Foundations to Frontiers - Notes to LAFF With and to be revisited later in this course, exists since B is SPD. Cost: approximately 1 3 n3 flops. Compute ŷ = A T y. Cost: mn flops. Solve Lz = ŷ and L T x = z. Cost: n flops each. Thus, the total cost of solving the LLS problem via normal equations is approximately mn n3 flops. Remark 1. We will later discuss that if A is not well-conditioned its columns are nearly linearly dependent, the Method of Normal Equations is numerically unstable because A T A is ill-conditioned. The above discussion can be generalized to the case where A C m n. In that case, x must solve A H Ax = A H y. A geometric explanation of the method of normal equations for the case where A is real valued can be found in Week 1 Sections of Linear Algebra: Foundations to Frontiers - Notes to LAFF With. 4 Solving the LLS Problem Via the QR Factorization Assume A C m n has linearly independent columns and let A = Q L R T L be its QR factorization. We wish to compute the solution to the LLS problem: Find x C n such that 4.1 Simple derivation of the solution Ax y = min z C n Az y. Notice that we know that, if A has linearly independent columns, the solution is given by x = A H A 1 A H y the solution to the normal equations. Now, x = [A H A] 1 A H y Solution to the Normal Equations = [ Q L R T L H Q L R T L ] 1 QL R T L H y A = Q L R T L = [ R H T L QH L Q LR T L ] 1 R H T L Q H L y BCH = C H B H = [ RT H L R 1 T L] R H T L Q H L y QH L Q L = I = R 1 T L R H T L RH T L QH L y = R 1 T L QH L y Thus, the x that solves R T L x = Q H L y solves the LLS problem. BC 1 = C 1 B 1 R H T L RH T L = I

3 4. Alternative derivation of the solution We know that then there exists a matrix Q R such that Q = Q L Q R is unitary. Now, min z C n Az y = min z C n Q L R T L z y substitute A = Q L R T L = min z C n Q H Q L R T L z y two-norm is preserved since Q H is unitary Q H = min L Q H z C n Q Q H L R T L z L y partitioning, distributing R Q H R RT L z Q H = min z C n L y Q H R y partitioned matrix-matrix multiplication RT L z Q H L = min z C n y Q H R y partitioned matrix addition = min z C n R T L z Q H L y + QH R y property of the -norm: x = x + y y = min z C n RT L z Q H L y + Q H R y Q H R y is independent of z = Q H R y minimized by x that satisfies R T L x = Q H L y Thus, the desired x that minimizes the linear least-squares problem solves R T L x = Q H L y. The solution is unique because R T L is nonsingular because A has linearly independent columns. In practice, one performs the following steps: Compute the QR factorization A = Q L R T L. If Gram-Schmidt or Modified Gram-Schmidt are used, this costs mn flops. Form ŷ = Q H L y. Cost: mn flops. Solve R T L x = ŷ triangular solve. Cost: n flops. Thus, the total cost of solving the LLS problem via Modified Gram-Schmidt QR factorization is approximately mn flops. Notice that the solution computed by the Method of Normal Equations generalized to the complex case is given by A H A 1 A H y = Q L R T L H Q L R T L 1 Q L R T L H y = RT H LQ H L Q L R T L 1 RT H LQ H L y = RT H LR T L 1 RT H LQ H L y = R 1 T L R H T L RH T LQ H L y = R 1 T L QH L y = R 1 T Lŷ = x where R T L x = ŷ. This shows that the two approaches compute the same solution, generalizes the Method of Normal Equations to complex valued problems, and shows that the Method of Normal Equations computes the desired result without requiring multivariate calculus. 3

4 5 Via Householder QR Factorization Given A C m n with linearly independent columns, the Householder QR factorization yields n Householder transformations, H,..., H n 1, so that H n 1 H A = }{{} H Q = Q L Q R RT L. We wish to solve R T L x = ŷ = Q H L y = = Q H L y }{{} ŷ. But [ ] Q H H L I y = I Q L Q R y = I I Q H R H n 1 H y. = I H n 1 H y = w T. }{{ } w = wt w B Q H y This suggests the following approach: RT L Compute H,..., H n 1 so that H n 1 H A =, storing the Householder vectors that define H,..., H n 1 over the elements in A that they zero out see Notes on Householder QR Factorization. Cost: mn 3 n3 flops. Form w = H n 1 H y see Notes on Householder QR Factorization. Partition w = wt w B where w T C n. Then ŷ = w T. Cost: 4m n flops. See Notes on Householder QR Factorization regarding this. Solve R T L x = ŷ. Cost: n flops. Thus, the total cost of solving the LLS problem via Householder QR factorization is approximately mn 3 n3 flops. This is cheaper than using Modified Gram-Schmidt QR factorization, and hence preferred because it is also numerically more stable, as we will discuss later in the course. 6 Via the Singular Value Decomposition Given A C m n with linearly independent columns, let A = UΣV H be its SVD decomposition. Partition ΣT L U = U L U R and Σ =, where U L C m n and Σ T L R n n so that A = U L U R Σ T L V H = U L Σ T L V H. 4

5 We wish to compute the solution to the LLS problem: Find x C n such that 6.1 Simple derivation of the solution Ax y = min z C n Az y. Notice that we know that, if A has linearly independent columns, the solution is given by x = A H A 1 A H y the solution to the normal equations. Now, x = [A H A] 1 A H y Solution to the Normal Equations = [ U L Σ T L V H H U L Σ T L V H ] 1 UL Σ T L V H H y A = U L Σ T L V H = [ V Σ T L U H L U LΣ T L V H ] 1 V ΣT L U H L y BCDH = D H C H B H and Σ H T L = Σ T L = [ V Σ T L Σ T L V H] 1 V ΣT L U H L y U H L U L = I = V Σ 1 T L Σ 1 T L V H V Σ T L UL Hy = V Σ 1 T L U L Hy 6. Alternative derivation of the solution V 1 = V H and BCD 1 = D 1 C 1 B 1 V H V = I and Σ 1 T L Σ T L = I We now discuss a derivation of the result that does not depend on the Normal Equations, in preparation for the more general case discussed in the next section. min z C n Az y = min z C n UΣV H z y substitute A = UΣV H = min z C n UΣV H z U H y substitute UU H = I and factor out U = min z C n ΣV H z U H y multiplication by a unitary matrix preserves two-norm ΣT L U H = min z C n V H z L y UR Hy partition, partitioned matrix-matrix multiplication ΣT L V H z UL H = min z C n UR Hy partitioned matrix-matrix multiplication and addition = min z C n Σ T L V H z U H L y + U H R y v T v B = v T + v B The x that solves Σ T L V H x = UL H y minimizes the expression. That x is given by x = V Σ 1 T L U L Hy. This suggests the following approach: Compute the reduced SVD: A = U L Σ T L V H. Cost: Greater than computing the QR factorization! We will discuss this in a future note. Form ŷ = Σ 1 T L U H L y. Cost: approx. mn flops. Compute z = V ŷ. Cost: approx. mn flops. 5

6 7 What If A Does Not Have Linearly Independent Columns? In the above discussions we assume that A has linearly independent columns. Things get a bit trickier if A does not have linearly independent columns. There is a variant of the QR factorization known as the QR factorization with column pivoting that can be used to find the solution. We instead focus on using the SVD. Given A C m n with ranka = r < n, let A = UΣV H be its SVD decomposition. Partition U = U L U R, V = V L V R where U L C m r, V L C n r and Σ T L R r r so that Now, A = U L U R Σ T L and Σ = ΣT L V L V R H = UL Σ T LV H L. min z C n Az y = min z C n UΣV H z y substitute A = UΣV H = min z C n UΣV H z UU H y UU H = I = min { w C n } UΣV H V w UU H y choosing the max over w C n with is the same as choosing the max over z C n. = min { w C n } UΣw U H y factor out U and V H V = I = min { w C n } Σw U H y Uv = v = min { w C n = min { w C n } } ΣT L wt w B ΣT L w T UL H y UR Hy = min { w C n } Σ T L w T UL Hy + U H R y U H L U H R y partition Σ, w, and U partitioned matrix-matrix multiplication v T v B, = v T + v B Since Σ T L is a diagonal with no zeroes on its diagonal, we know that Σ 1 T L exists. Choosing w T = Σ 1 T L U L Hy means that { min } Σ T L w T UL H y w C n =, which obviously minimizes the entire expression. We conclude that Σ 1 T L x = V w = V L V U L Hy R = V L Σ 1 T L w U L H y + V R w B B characterizes all solutions to the linear least-squares problem, where w B can be chosen to be any vector of size n r. By choosing w B = and hence x = V L Σ 1 T L U L H y we choose the vector x that itself has minimal -norm. 6

7 The sequence of pictures on the following pages reasons through the insights that we gained so far in Notes on the Singular Value Decomposition and this note. These pictures can be downloaded as a PowerPoint presentation from 7

8 A CV L Row space Column space CU L dim = r ŷ dim = r Ax = ŷ x = x r + x n dim = n r dim = m r CV R Null space Le1 null space CU R If A C m n and A = U L U R Σ T L V L V R H = UL Σ T LV H L equals the SVD, where U L C m r, V L C n r, and Σ T L C r r, then The row space of A equals CV L, the column space of V L ; The null space of A equals CV R, the column space of V R ; The column space of A equals CU L ; and The left null space of A equals CU R. Also, given a vector x C n, the matrix A maps x to ŷ = Ax, which must be in CA = CU L. 8

9 A CV L Row space Column space CU L dim = r dim = r Ax = y?? x y dim = n r dim = m r CV R Null space Le1 null space CU R Given an arbitrary y C m not in CA = CU L, there cannot exist a vector x C n such that Ax = y. 9

10 CV L dim = r Row space x r = V L Σ 1 TL U H L y A Ax r =U L Σ TL V L H x r = ŷ Ax = ŷ x = x r + x n Column space ŷ =U L U L H y y CU L dim = r dim = n r Ax n = dim = m r CV R Null space x n = V R w B Le1 null space CU R The solution to the Linear Least-Squares problem, x, equals any vector that is mapped by A to the projection of y onto the column space of A: ŷ = AA H A 1 A H y = QQ H y. This solution can be written as the sum of a unique vector in the row space of A and any vector in the null space of A. The vector in the row space of A is given by x r = V L Σ 1 T L U H L y == V L Σ 1 T L U H L ŷ. The sequence of pictures, and their explanations, suggest a much simply path towards the formula for solving the LLS problem. We know that we are looking for the solution x to the equation Ax = U L U H L y. We know that there must be a solution x r in the row space of A. It suffices to find w T x r = V L w T. such that Hence we search for w T that satisfies AV L w T = U L U H L y. Since there is a one-to-one mapping by A from the row space of A to the column space of A, we know that w T is unique. Thus, if we find a solution to the above, we have found the solution. 1

11 Multiplying both sides of the equation by U H L yields U H L AV L w T = U H L y. Since A = U L Σ 1 T L V L H, we can rewrite the above equation as Σ T L w T = U H L y so that w T = Σ 1 T L U H L y. Hence x r = V L Σ 1 T L U H L y. Adding any vector in the null space of A to x r also yields a solution. Hence all solutions to the LLS problem can be characterized by x = V L Σ 1 T L U H L y + V R w R. Here is yet another important way of looking at the problem: We start by considering the LLS problem: Find x C n such that Ax y = max z C n Az y. We changed this into the problem of finding w L that satisfied where x = V L w L and ŷ = U L U H L y = U Lv T. Σ T L w L = v T Thus, by expressing x in the right basis the columns of V L and the projection of y in the right basis the columns of U L, the problem became trivial, since the matrix that related the solution to the right-hand side became diagonal. 8 Exercise: Using the the LQ factorization to solve underdetermined systems We next discuss another special case of the LLS problem: Let A C m n where m < n and A has linearly independent rows. A series of exercises will lead you to a practical algorithm for solving the problem of describing all solutions to the LLS problem Ax y = min z Az y. You may want to review Notes on the QR Factorization as you do this exercise. Exercise. Let A C m n with m < n have linearly independent rows. Show that there exist a lower triangular matrix L L C m m and a matrix Q T C m n with orthonormal rows such that A = L L Q T, noting that L L QT does not have any zeroes on the diagonal. Letting L = L L be C m n and unitary Q =, reason that A = LQ. Don t overthink the problem: use results you have seen before. Q B 11

12 Algorithm: [L, Q] := LQ CGS unba, L, Q AT LT L L T R Partition A, L, Q A B L BL L BR where A T has rows, L T L is, Q T has rows QT while ma T < ma do Repartition A L l 1 L AT a T LT L L T R 1, l1 T λ 11 l T 1, A B L BL L BR A L l 1 L where a 1 has 1 row, λ 11 is 1 1, q 1 has 1 row Q B QT Q B Q q T 1 Q Continue with AT endwhile A B A a T 1 A, LT L L T R L BL L BR L l 1 L l T 1 λ 11 l T 1 L l 1 L, QT Q B Q q T 1 Q Figure 1: Algorithm skeleton for CGS-like LQ factorization. Exercise 3. Let A C m n with m < n have linearly independent rows. Consider Ax y = min z Az y. Use the fact that A = L L Q T, where L L C m m is lower triangular and Q T has orthonormal rows, to argue that any vector of the form Q H T L 1 L y + QH B w B where w B is any vector in C n m is a solution to the LLS problem. Here Q = QT Q B. Exercise 4. Continuing Exercise, use Figure 1 to give a Classical Gram-Schmidt inspired algorithm for computing L L and Q T. The best way to check you got the algorithm right is to implement it! Exercise 5. Optional Continuing Exercise, use Figure to give a Householder QR factorization inspired algorithm for computing L and Q, leaving L in the lower triangular part of A and Q stored as Householder vectors above the diagonal of A. The best way to check you got the algorithm right is to implement it! 1

13 Algorithm: [A, t] := HLQ unba, t AT L A T R Partition A, t A BL A BR where A T L is, t T has rows while ma T L < ma do Repartition AT L A T R A BL A BR tt t B A a 1 A a T 1 α 11 a T 1 A a 1 A where α 11 is 1 1, τ 1 has 1 row, tt t B t τ 1 t Continue with AT L A T R endwhile A BL A BR A a 1 A a T 1 α 11 a T 1 A a 1 A, tt t B t τ 1 t Figure : Algorithm skeleton for Householder QR factorization inspired LQ factorization. 13