The Full-rank Linear Least Squares Problem

Transcription

1 Jim Lambers COS 7 Spring Semeseter 1-11 Lecture 3 Notes The Full-rank Linear Least Squares Problem Gien an m n matrix A, with m n, and an m-ector b, we consider the oerdetermined system of equations Ax = b, in the case where A has full column rank. If b is in the range of A, then there exists a unique solution x. For example, there exists a unique solution in the case of A = 1 1, b = but not if b = T. In such cases, when b is not in the range of A, then we seek to minimize Ax b p for some p. Different norms gie different solutions. If p = 1 or p =, then the function we seek to minimize, f(x) = Ax b p is not differentiable, so we cannot use standard minimization techniques. Howeer, if p =, f(x) is differentiable, and thus the problem is more tractable. We now consider two methods. The first approach is to take adantage of the fact that the -norm is inariant under orthogonal transformations, and seek an orthogonal matrix Q such that the transformed problem 1 1, min Ax b = min Q T (Ax b) is easy to sole. Let Then Q T 1 A = R and R A = Q = Q 1 Q R = Q 1 R. min Ax b = min Q T (Ax b) = min (Q T A)x Q T b = min R x Q T b If we partition Q T b = c d 1

2 then min Ax b = min R c x d = min Rx c + d. Therefore, the minimum is achieed by the ector x such that Rx = c and therefore min x Ax b = d ρ LS. The second method is to define φ(x) = 1 Ax b, which is a differentiable function of x. We can minimize φ(x) by noting that φ(x) = A T (Ax b), which means that φ(x) = if and only if A T Ax = A T b. This system of equations is called the normal equations, and were used by Gauss to sole the least squares problem. If m >> n then A T A is n n, which is a much smaller system to sole than Ax = b, and if κ(a T A) is not too large, we can use the LU factorization to sole for x. Which is the better method? This is not a simple question to answer. The normal equations produce an x whose relatie error depends on κ(a), whereas the QR factorization produces an x whose relatie error depends on u(κ (A) + ρ LS κ (A) ). The normal equations inole much less arithmetic when m >> n and they require less storage, but the QR factorization is often applicable if the normal equations break down. Gram-Schmidt Orthogonalization From the aboe discussion, it can be seen that to sole the least squares problem, it is not necessary to compute the full QR factorization R A = Q, where Q is an m m orthogonal matrix and R is an n n upper triangular matrix. Rather, it is sufficient to compute the economy-size QR factorization A = QR, where Q is an m n matrix with orthogonal columns. That is, A = a 1 a n = QR = q1 q n r 11 r 1n.... r nn where Q T Q = I. This is because the solution x can be obtained by soling the upper triangular system Rx = Q T 1 b. We therefore focus on computing this smaller factorization. From the aboe matrix product we can see that a 1 = r 11 q 1, from which it follows that r 11 = ± a 1, q 1 = 1 a 1 a 1.,

3 Next, from a = r 1 q 1 + r q we obtain In general, we use the relation to obtain Note that q k can be rewritten as r 1 = q T 1 a, r = ± a r 1 q 1, q = 1 r (a r 1 q 1 ). q k = 1 a k q k = 1 = 1 = 1 a k = k r jk q j r jk q j, r jk = q T j a k. a k (q T j a k )q j a k q j q T j a k I q j q T j a k. If we define P i = q i q T i, then P i is a symmetric projection that satisfies Pi = P i, and P i P j = δ ij. Thus we can write q k = 1 I P j a k = 1 k 1 (I P j )a k. Why doesn t Gram-Schmidt work? If a 1 and a are almost parallel, then a r 1 q 1 is almost zero and roundoff error becomes significant. Modified Gram-Schmidt Last time we indicated that the classical Gram-Schmidt process was numerically unstable. The Modified Gram-Schmidt method alleiates this difficulty. Recall A = QR = r 11 q 1 r 1 q 1 + r q 3

4 We define which means If we write then We then compute C (k) = q i r T i, r T i = r ii r i,i+1 r in i=1 A C (k) = A (k). A (k) = z B = z, q k = 1 z. rk,k+1 r k,n = q T k B which yields This process is numerically stable. We can show ˆQ T 1 ˆQ 1 = I + E MGS, A (k+1) = B q k rk,k+1 r kn E MGS uκ (A), and ˆQ 1 can be computed in approximately mn flops, whereas with Householder QR, ˆQ T 1 ˆQ 1 = I + E n, E n u, with ˆQ 1 being computed in approximately mn n /3 flops to factor A and an additional mn n /3 flops to obtain the n columns of Q. Using the Normal Equations We can sole the linear least squares problem using the normal equations A T Ax = A T b as follows: first, we sole the aboe system to obtain an approximate solution ˆx, and compute the residual ector r = b Aˆx. Now, because we obtain the system A T r = A T b A T Aˆx =, r + Aˆx = b A T r = 4

5 or, in matrix form, I A A T r ˆx This is a large system, but it preseres the sparsity of A. It can be used in connection with iteratie refinement, but unfortunately this procedure does not work well because it is ery sensitie to the residual. = Least Squares with Linear Constraints Suppose that we wish to fit data as in the least squares problem, except that we are using different functions to fit the data on different subinterals. A common example is the process of fitting data using cubic splines, with a different cubic polynomial approximating data on each subinteral. Typically, it is desired that the functions assigned to each piece form a function that is continuous on the entire interal within which the data lies. This requires that constraints be imposed on the functions themseles. It is also not uncommon to require that the function assembled from these pieces also has a continuous first or een second deriatie, resulting in additional constraints. The result is a least squares problem with linear constraints, as the constraints are applied to coefficients of predetermined functions chosen as a basis for some function space, such as the space of polynomials of a gien degree. The general form of a least squares problem with linear constraints is as follows: we wish to find an n-ector x that minimizes Ax b, subject to the constraint C T x = d, where C is a known n p matrix and d is a known p-ector. This problem is usually soled using Lagrange multipliers. We define Then To minimize f, we can sole the system A T A C C T b. f(x; λ) = b Ax + λ T (C T x d). f = (A T Ax A T b + Cλ). x λ = A T b d From A T Ax = A T b Cλ, we see that we can first compute x = ˆx (A T A) 1 Cλ where ˆx is the solution to the unconstrained least squares problem. Then, from the equation C T x = d we obtain the equation C T (A T A) 1 Cλ = C T ˆx d, which we can now sole for λ. The algorithm proceeds as follows: 1. Sole the unconstrained least squares problem Ax = b for ˆx.. 5

6 . Compute A = QR. 3. Form W = (R T ) 1 C. 4. Compute W = P U, the QR factorization of W. 5. Sole U T Uλ = η = C T ˆx d for λ. Note that 6. Set x = ˆx (A T A) 1 Cλ. U T U = (P T W ) T (P T W ) = W T P P T W = C T R 1 (R T ) 1 C = C T (R T R) 1 C = C T (R T Q T QR) 1 C = C T (A T A) 1 C This method is not the most practical since it has more unknowns than the unconstrained least squares problem, which is odd because the constraints should hae the effect of eliminating unknowns, not adding them. We now describe an alternate approach. Suppose that we compute the QR factorization of C to obtain Q T R C = where R is a p p upper triangular matrix. Then the constraint C T x = d takes the form R T u = d, Q T u x =. Then b Ax = b AQQ T x = b A u = b A1 Thus we can obtain x by the following procedure: 1. Compute the QR factorization of C, A = AQ A u = b A 1 u A 6

7 . Compute A = AQ 3. Sole R T u = d 4. Sole the new least squares problem of minimizing (b A 1 u) A 5. Compute u x = Q. This approach has the adantage that there are fewer unknowns in each system that needs to be soled, and also that κ( A ) κ(a). The drawback is that sparsity can be destroyed. Least Squares with Quadratic Constraints We wish to sole the problem b Ax = min, x = α, α A + b. This problem is known as least squares with quadratic constraints. To sole this problem, we define and seek to minimize φ. From we obtain the system If we denote the eigenalues of A T A by then If μ, then κ(a T A + μi) κ(a T A), because φ(x; μ) = b Ax + μ( x α ) φ = A T b A T Ax + μx (A T A + μi)x = A T b. λ i (A T A) = λ 1,..., λ n, λ 1 λ λ n λ i (A T A + μi) = λ 1 + μ,, λ n + μ. λ 1 + μ λ n + μ λ 1 λ n, so A T A + μi is better conditioned. Soling the least squares problem with quadratic constraints arises in many literatures, including 7

8 1. Statistics: Ridge Regression. Regularization: Tikhono 3. Generalized cross-alidation (GCV) where To sole this problem, we see that we need to compute If A = UΣV T is the SVD of A, then we hae x = (A T A + μi) 1 A T b x T x = b T A(A T A + μi) A T b = α. α = b T UΣV T (V Σ T ΣV T + μi) V Σ T U T b = c T Σ(Σ T Σ + μi) Σ T c, U T b = c = r c i σ i (σi + μ) i=1 = χ(μ) The function χ(μ) has poles at σ i for i = 1,..., r. Furthermore, χ(μ) as μ. We now hae the following procedure for soling this problem, gien A, b, and α : 1. Compute the SVD of A to obtain A = UΣV T.. Compute c = U T b. 3. Sole χ(μ ) = α where μ. Don t use Newton s method on this equation directly; soling 1/χ(μ) = 1/α is much better. 4. Use the SVD to compute Total Least Squares x = (A T A + μi) 1 A T b = V (Σ T Σ + μi) 1 Σ T U T b. In the ordinary least squares problem, we are soling Ax = b + r, r = min. In the total least squares problem, we wish to sole (A + E)x = b + r, E F + λ r = min. 8

9 or From Ax b + Ex r =, we obtain the system A b x 1 + E r x 1 (C + F )z =. =, We need the matrix C + F to hae rank < n + 1, and we want to minimize F F. To sole this problem, we compute the SVD of C = A b = UΣV T. Let ˆC = UΩ n V T. Then, if i is the ith column of V, we hae Our solution is ˆx 1 ˆC n+1 = UΩ n V T n+1 =. 1 = n+1 n+1,n+1 proided that n+1,n+1 =. Now, suppose that only some of the data is contaminated, i.e. E = E 1 where the first p columns of E are zero. Then, in soling (C + F )z =, we use Householder transformations to compute Q T (C +F ) where the first p columns are zero below the diagonal. Since F F = Q T F F, we then hae a block upper triangular system R11 R 1 + F 1 R + F We can find the total least squares solution of u z =, z = (R + F ) =,. and then set F 1 = and sole R 11 u + R 1 =. 9