Numerical Linear Algebra Chap. 2: Least Squares Problems

Transcription

1 Numerical Linear Algebra Chap. 2: Least Squares Problems Heinrich Voss Hamburg University of Technology Institute of Numerical Simulation TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

2 onto a line Problem: Given a point b R m and a line through the origin in the direction of a R m Find the point p on the line closest to the point b Observations: Hence p has a representattion αa for some α R The line connecting b to p is perpendicular to the vector a 0 = a T (b p) = a T (b αa) = a T b αa T a α = at b a T a. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

3 matrix p = αa = aα = a at b a T a = aat a T b =: Pb a The projection of some b onto the direction of some a is obtained by multiplying the vector b by the rank-one matrix P = aat a T a TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

4 onto a subspace Problem: Given a point b R m and n linearly independent vectors a 1,..., a n R m Find the linear combination p = n j=1 x ja j span{a 1,..., a n } which is closest to the point b p is called projection of b onto span{a 1,..., a n }. p is characterized by the condition that the error b p is perpendicular to the subspace, and this is equivalent to: b p is perpendicular to a 1,..., a n. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

5 onto a subspace ct. b p = b Ax is perpendicular to a 1,..., a n if and only if (a 1 ) T (b Ax) = 0 (a 2 ) T (b Ax) = 0. (a n ) T (b Ax) = 0 A T (b Ax) = 0 A T Ax = A T b. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

6 Theorem A T A is nonsingular if and only if the columns of A are linearly independent Let x be in the nullspace of A. Then it holds Ax = 0 A T Ax = 0. Hence, x is in the nullspace of A T A. If x is in the nullspace of A T A, then it holds A T Ax = 0 0 = x T A T Ax = (Ax) T (Ax) = Ax 2 Ax = 0. If the columns of A are linearly independent, then it follows x = 0. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

7 onto a subspace If a 1,..., a n are linearly independent, then the projection p of some vector b onto span{a1,..., a n } is given by p = Ax where x is the unique solution of A T Ax = A T b x = (A T A) 1 A T b The projector is given by P = A(A T A) 1 A T P is symmetric, and it holds P 2 = P The distance from b to the subspace is b Pb TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

8 Least squares problem TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

9 Least squares problem ct. Given are measurements (t j, b j ), j = 1,..., m. Find straight line f (t) = x 1 + x 2 t which matches the measurements best Try to solve the linear system x 1 + x 2 t j = b j, j = 1,..., m as good as possible. Solve the linear system 1 t 1 1 t 2. 1 t m ( x1 b 1 ) b 2 = x 2. b m as good as possible. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

10 Least squares problem ct. Replace problem Ax = b by: Find x such that the error Ax b is as small as possible. The (unique) solution is called least squares solution. Solution: The nearest point in the space {Ax : x R n } to the point b is the projection of b onto this space. Hence the solution of the least squares problem is x = (A T A) 1 Ab. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

11 Fitting a straight line Measurements t j b j A = 1 2, b = A T Ax = ( ) ( ) 7 14 x = A T 3.8 b = x = ( ) TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

12 Least squares problem TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

13 Fitting a straight line Given m points (t j, b j ), j = 1, dots, m in the plane. Find a straight line x 1 + x 2 t, t R fitting the data in the least squares sense. 1 t 1 b 1 1 t 2 ( ) A =., b 2 AT b = t 1 t 2... t m. = 1 t m b n ( m j=1 b ) j m j=1 t jb j 1 t 1 ( ) A T t 2 ( m m A = t 1 t 2... t. m. = j=1 t ) j m j=1 t m j j=1 t j 2 1 t m ( m m A T j=1 Ax = t ) j m j=1 t m j j=1 t j 2 x = ( m j=1 b j m j=1 t jb j ) = A T b TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

14 Trigonometric polynomial TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

15 Trigonometric polynomial ct. Given measurements (t j, b j ), j = 1,..., n. Find trigonometric polynomial f (t) = x 1 + x 2 sin(t) + x 3 cos(t) such that the measurements are closest to the graph of f 1 sin(t 1 ) cos(t 1 ) 1 sin(t 2 ) cos(t 2 ) A =. 1 sin(t n ) cos(t n ) Solve Ax b = min! A T Ax = A T b TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

16 Trigonometric polynomial ct TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

17 Orthonormal vectors q 1, q 2,..., q n R m are orthonormal if { (q j ) T q k 0 when j k = 1 when j = k Orthonormal vectors are linearly independent: n n α j q j = 0 0 = (q k ) T α j q j = j=1 j=1 n α j (q k ) T q j = α k, k = 1,..., n j=1 If n = m then q 1,..., q n is an orthonormal basis. If q 1,..., q n is an orthonormal basis of R n, then x R n has the representation x = n α j q j (q k ) T x = j=1 n α j (q k ) T q j = α k x = j=1 n x T q j q j. j=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

18 Example q 1 = ( ) cos θ, q 2 = sin θ ( ) sin θ, x = cos θ ( ) cos α sin α x = x T q 1 q 1 + x T q 2 q 2 = ((cos α cos θ + sin α sin θ)q 1 + ( cos α sin θ + sin α cos θ)q 2 ( ) ( ) cos θ sin θ = cos(α θ) + sin(α θ). sin θ cos θ TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

19 Orthogonal matrix An n n matrix with orthonormal columns is called orthogonal matrix. It is usually denoted by Q. Every orthogonal matrix Q is nonsingular. Q T Q = I Q 1 = Q T. If Q is orthogonal then Q T is orthogonal: (Q T ) T Q T = QQ T = QQ 1 = I TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

20 Rotation ( ) cos θ sin θ Q = sin θ cos θ rotates every vector in the plane through the angle θ ( ) ( ) Q T cos θ sin θ cos( θ) sin( θ) = = = Q 1 sin θ cos θ sin( θ) cos( θ) rotates every vector in the plane through the angle θ TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

21 Permutation P = obviously is an orthogonal matrix Px = x 1 x 2 x 3 x 4 = x 2 x 4 x 1 x P T = = P puts the components back into their original order x 2 x 4 x 1 x 3 = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51 x 1 x 2 x 3 x 4

22 Reflection If u is any unit vector, then Q = I 2uu T is orthogonal: since u T u = 1. Q T = I 2(uu T ) T = I 2(u T ) T u T = I 2uu T = Q, Q T Q = (I 2uu T )(I 2uu T ) = I 4uu T + 4uu T uu T = I Q defines a refection at the hyperplane u : Let x = u T x u + v, u T v = 0. Then it follows Qx = (u T x)(i 2uu T )u + (I 2uu T )v = (u T x)(u 2uu T u) + (v 2uu T v) = (u T x)u + v Now it is obvious, that Q 2 = Q: reflecting twice through a mirror brings back the original. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

23 Orthogonal matrices If Q is orthogonal, then it leaves lengths unchanged: Qx 2 = (Qx) T (Qx) = x T Q T Qx = x T x = x 2. It also leaves angles unchanged cos(x, y) = x T y x y = x T Q T Qy Qx Qy = (Qx)T (Qy) Qx Qy. Orthogonal matrices are excellent for computations: numbers never grow too large when lengths are fixed. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

24 Let q 1,..., q n R m orthonormal vectors (how do we know that m n?) Then with Q := [q 1,..., q n ] R m n the projector onto V := span{q 1,..., q n } is P = Q(Q T Q) 1 Q T = QQ T and the projection of x R m onto V is Px = QQ T x = n (x T q j )q j, j=1 the truncation of the Fourier development with respect to an orthonormal basis containing q 1,..., q n. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

25 Least squares problem Consider the least squares problem Qx b = min!, where the system matrix has orthonormal columns: Solution: x = Q T b TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

26 Example is an orthogonal matrix Q = Represent the vector x = (1, 1, 1) T as a linear combination of the columns q j of Q. x T q 1 = 19 13, x T q 2 = 11 13, x T q 3 = 5 13 x = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

27 Fitting a straight line Given measurements (t j, b j ), j = 1,..., m. Assume that the measurement times t j add to zero. Since the scalar product of (t j ) j=1,...,m and (1, 1,..., 1) T is zero the normal equations ( m m A T j=1 Ax = t ) j m j=1 t m j j=1 t j 2 x = obtain the form ( ) m 0 m 0 j=1 t j 2 x = ( m j=1 b j m j=1 t jb j ) = A T b ( m j=1 b ) j m j=1 t jb j and the solution is x = ( m j=1 b ) j/m m j=1 t jb j / m j=1 t j 2 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

28 Fitting a straight line Orthogonal columns are so helpful that it is worth moving the time origing to produce them. To this end subtract the average time t = 1 m m j=1 t j. Then the shifted measurement times t j := t j t add to zero. The solution of the least squares problem is x 1 + x 2 t = x 1 + x 2 (t t) = ( x 1 x 2 t) + x 2 t where x 1 = 1 m m j=1 b j and x 2 = b j t j m t. j=1 j 2 j=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

29 Example ct. Measurements t j b j The average time is t = 2, and t = ( 3, 2, 1, 0, 1, 2, 3) T, and x 1 = j=1 b j = 3.8 7, x j=1 2 = b j t j m t j=1 j 2 = Therefore, the solution of the least equares problem is ( x 1 x 2 t) + x 2 t = t = t. 28 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

30 Gram Schmidt process Assume that a 1,..., a n R m are given linearly independent vectors. Problem: Determine an orthogonal basis q 1,..., q n of V := span{a 1,..., a n } Advantage For Q := [q 1,..., q n ] it holds that Q T Q = diag{ q 1 2,..., q n 2 } is a diagonal matrix. Hence the orthogonal projection onto V decouples: Px := Q(Q T Q) 1 Q T x = Qdiag{ q 1 2,..., q n 2 } 1 ( x T q 1,..., x T q n) T = ( x T q 1 Q q 1 2,..., x T q n ) T n x T q j q n 2 = q j 2 qj. j=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

31 Example Project x = 2 3 to the subspace spanned by q1 = 2 3 and q2 = Since q 1 and q 2 are orthogonal, the projection is 1 p = x T q 1 q 1 2 q1 + x T q 2 q 2 2 q2 = Indeed, p x = 1 1 is orthogonal to q1 and q = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

32 Gram Schmidt process Assume that a 1,..., a n R m are given linearly independent vectors. Problem: Determine an orthogonal basis q 1,..., q n of V := span{a 1,..., a n } Idea: Set q 1 = a 1, and for j = 2, 3,..., n subtract the projection of a j to the subspace span{a 1,..., a j 1 } from a j. Then this difference q j is orthogonal to span{a 1,..., a j 1 }, and therefore to the previously determined vectors q i. Since q 1,..., q j 1 is an orthogonal basis of span{a 1,..., a j 1 }, the projections are decoupled. Hence, j 1 q j = a j k=1 (a j ) T q k q k 2 qk, j = 2,..., n. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

33 Example Apply the Gram Schmidt process to a 1 = 3 4, a 2 = 2 11, a 3 = q 1 = a 1, q 2 = a 2 (a2 ) T (q 1 ) q 1 2 q 1 = = q 3 = a 3 (a3 ) T (q 1 ) q 1 2 q 1 (a3 ) T (q 2 ) q 2 2 q 2 = = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

34 Factorization Remember j 1 q j = a j k=1 (a j ) T q k q k 2 qk, j = 2,..., n. It is more convenient to normalize the q vectors in the course of the algorithm and to get rid of the denominators. The Gram Schmidt algorithm then reads q 1 = q j = 1 a 1 a1 ( ) j 1 / j 1 a j (a j ) T q k q k a j (a j ) T q k q k, j = 2,..., n. k=1 k=1 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

35 QR Factorization The following form is more convenient: a 1 = a 1 q 1 j 1 j 1 a j = (a j ) T q k q k + aj (a j ) T q k q k q j, j = 2,..., n. k=1 k=1 Since at every step a j is a linear combination of q 1,..., q j, and later q i :s are not involved: r 11 r 12 r r 1n A = ( a 1,..., a n) = ( r 22 r r 2n q 1,..., q n)... = QR... where r ij = (a j ) T q i, i = 1,..., j 1 and r jj = a j j 1 k=1 (aj ) T q k q k. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51 r nn

36 Example Determine the QR-factorization of A = [a 1, a 2, a 3 ] = q 1 = 1 a 1 a1 = q 2 q 2 = a 2 (a 2 ) T q 1 q 1 = = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

37 Example ct. q 3 q 3 = a 3 (a 3 ) T q 1 q 1 (a 3 ) T q 2 q = = 1 1 q3 = Hence the QR factorization of A is / = 0 2/3 1/ 2 0 2/3 1/ TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

38 Least squares problems Any matrix A R m n with linearly independent columns can be factored into QR. Q R m n has orthonormal columns, and R R n n is upper triangular with positive diagonal. Given the QR factorization of A a least squares problem with system matrix A can be solved easily: A T Ax = A T b (QR) T (QR)x = (QR) T b R T Q T QRx = R T Q T b R T Rx = R T Q T b Rx = Q T b. Instead of solving the normal equations A T Ax = A T b by Gaussian elimination one just solves Rx = Q T b by back substitution. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

39 Example Solve the least squares problem Ax b where and b = Normal equations: A T Ax = x = A T b = 3 17 x = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

40 Example ct. Taking advantage of the known QR factorization / = 0 2/3 1/ 2 0 2/3 1/ we have to solve Rx = x = Q T b = 14/3 2 1/ x = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

41 Gram Schmidt algorithm 1: q 1 = a 1 / a 1 2: for j=2,...,n do 3: q j = a j 4: for k=1,...,j-1 do 5: r kj = (a j ) T q k ; 6: q j = q j r kj q k ; 7: end for 8: q j = q j / q j 9: end for In steps 5 and 6 the projection of a j onto spanq k is subtracted from q j for k = 1,..., j 1. At that time k 1 q j = a j (a j ) T q l q l l=1 and from (q l ) T q k = 0 for l = 1,..., k 1 it follows that we can replace step 5 by r kj = (q j ) T q k. This version is called modified Gram Schmidt method. It is numerically more stable than the original Gram Schmidt factorization. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

42 Householder transformation Although the modified Gram Schmidt method is knwon to be more stable than the original Gram Schmidt method, there is a better way to solve least square problems by Householder transformation. Problem: Given a vector a R m ; find a refection H = I 2uu T, u = 1 such that Ha is a multiple of the first unit vector e := (1, 0,..., 0) T. From the orthogonality of H it follows that Ha = a. Hence, Ha = a 2u T a u = ± a e, and therefore u and a ± a e are parallel. If a is not a multiple of e then there are two solutions of our problem: u = a + a e a + a e and u = a a e a a e, for a = a e the first solution is defined, for a = a e only the second one. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

43 Householder transformation ct. In any case the Householder transformation H = I 2 ww T w T w with w = a + sign(a 1) a e exists. For stability reasons (cancellation!) we always choose this one. Let a = (2, 2, 1) T. Then a = 3, and w = (5, 2, 1) T, and H = I 1 w 2 ww T = I ( ) Ha = A second solution is defined by w = ( 1, 2, 1) T : H = I 1 w 2 w w T = I ( ) Ha = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

44 Least equares problem Consider the least square problem Ax b = min! where A R m n, b R n. Since orthogonal matrices do not change the length of a vector, it follows Ax b = Q(Ax b) for every orthogonal matrix Q R m m. ( R Assume that QA = where Q O) T Q = I, R R n n is upper triangular and O R m n,n denotes the nullmatrix. Then we obtain ( Ax b 2 = QAx Qb 2 R = x Qb O) 2 = Rx b b 2 2 where b 1 R n contains the leading n components of Qb, and b 2 the remaining ones. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

45 Least equares problem ct. From Ax b 2 = Rx b b 2 2 it is obvious that x = R 1 b 1 is the solution of the least squares problem, and that b 2 is the residuum. Q is constructed step by step using Householder transformations. Let H 1 be a Householder transformation such that Ha 1 = ± a 1 e, where a 1 denotes the first column of A. Then r 11 r r 1n 0 A 1 := H 1 A =. Ã 1 0 TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

46 Least equares problem ct. Next we annihilate the subdiagonal elements of the second column. Let H 2 be the Householder matrix wich maps the first column of Ã1 to a multiple of the first unit vector (in R m 1 ), and let H 2 =. H 2 0 Then we obtain r 11 r 12 r r 1n 0 r 22 r r 2n H 2 H 1 A = H 2 A 1 =: A 2 = Ã TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

47 Least equares problem ct. Continuing that way we finally arrive at r 11 r r 1n 0 r r 2n ( H n H n 1 H 1 A =: A n = r nn R =: O) ) ( b1 With b := H n H n 1 H 1 b := b n, b 1 R n the least squares problem obtains the form with the unique solution Ax b = A n x b x = R 1 b1. TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

48 Example A = [a 1, a 2 ] = , b = From a 1 = 2 it follows that the Householder method mapping the first column a 1 of A to a multiple of the first unit vector is H 1 = I 2 ww T w T w where w = = A 1 = H 1 A = , H 1b = TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

49 Example ct. Obviously, this problem is equivalent to x = min! which is solved by ( ) x = ( ) x = ( ) TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

50 For didactic reasons we solve the transformed problem A 1 x H 1 b = x = min! using a Householder transformation. H 2 = I 2 vv T v T v, v = = maps the first column of Ã2 to a multiple of the first unit vector in R 3 Hence H ( ) = 0 1 0, H 2 A 1 = A 0 1 = 0 3 H TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51

51 H = ( ) = and we obtain the equivalent problem x = min! which is solved by ( ) 2 5 x = ( 6.75 ) 0 3 x = ( ) TUHH Heinrich Voss Numerical Linear Algebra Chap. 2: Least Squares Problems / 51