Lecture 13: Orthogonal projections and least squares (Section ) Thang Huynh, UC San Diego 2/9/2018

Transcription

1 Lecture 13: Orthogonal projections and least squares (Section ) Thang Huynh, UC San Diego 2/9/2018

2 Orthogonal projection onto subspaces Theorem. Let W be a subspace of R n. Then, each x in R n can be uniquely written as x = x + x. in W in W 1

3 Orthogonal projection onto subspaces Theorem. Let W be a subspace of R n. Then, each x in R n can be uniquely written as x = x + x. in W in W x is the orthogonal projection of x onto W. 1

4 Orthogonal projection onto subspaces Theorem. Let W be a subspace of R n. Then, each x in R n can be uniquely written as x = x + x. in W in W x is the orthogonal projection of x onto W. x is the point in W closest to x. 1

5 Orthogonal projection onto subspaces Theorem. Let W be a subspace of R n. Then, each x in R n can be uniquely written as x = x + x. in W in W x is the orthogonal projection of x onto W. x is the point in W closest to x. If v 1,, v m is an orthogonal basis of W, then x = ( x v 1 ) v v 1 v ( x v m ) v 1 v m v m. m 1

6 Orthogonal projection onto subspaces Theorem. Let W be a subspace of R n. Then, each x in R n can be uniquely written as x = x + x. in W in W x is the orthogonal projection of x onto W. x is the point in W closest to x. If v 1,, v m is an orthogonal basis of W, then Once x = ( x v 1 ) v v 1 v ( x v m ) v 1 v m v m. m x is determined, x = x x. 1

7 Orthogonal projection onto subspaces Example. Let W = span { { 3 0 1, } }, and x = Find the orthogonal projection of x onto W. Write x as a vector in W plus a vector orthogonal to W. 2

8 Orthogonal projection onto subspaces Definition. Let v 1,, v m be an orthogonal basis of W, a subspace of R n. The projection map π W R n R n, given by π W (x) = ( x v 1 v 1 v 1 ) v ( x v m v m v m ) v m is linear (why?). The matrix P representing π W with respect to the standard basis is the corresponding projection matrix. 3

9 Orthogonal projection onto subspaces Definition. Let v 1,, v m be an orthogonal basis of W, a subspace of R n. The projection map π W R n R n, given by π W (x) = ( x v 1 v 1 v 1 ) v ( x v m v m v m ) v m is linear (why?). The matrix P representing π W with respect to the standard basis is the corresponding projection matrix. Example. Find the projection matrix P which corresponds to { 3 orthogonal projection onto W = span 0, 0 } 1 in R 3. Then { 1 0 } find the orthogonal projection of x = 0 3 onto W. 10 3

10 Least squares In practice, Ax b. 4

11 Least squares In practice, Ax b. Definition. Ax = b if x is a least squares solution of the system x is such that Ax b is as small as possible. 4

12 Least squares In practice, Ax b. Definition. x is a least squares solution of the system Ax = b if x is such that Ax b is as small as possible. If Ax = b is consistent, i.e. b is in C(A), then a least squares solution x is just an ordinary solution. 4

13 Least squares In practice, Ax b. Definition. x is a least squares solution of the system Ax = b if x is such that Ax b is as small as possible. If Ax = b is consistent, i.e. b is in C(A), then a least squares solution x is just an ordinary solution. Interesting case: Ax = b is inconsistent, i.e. b is NOT in C(A). 4

14 Least squares In practice, Ax b. Definition. x is a least squares solution of the system Ax = b if x is such that Ax b is as small as possible. If Ax = b is consistent, i.e. b is in C(A), then a least squares solution x is just an ordinary solution. Interesting case: Ax = b is inconsistent, i.e. b is NOT in C(A). What should we do to find x? 4

15 Least squares In practice, Ax b. Definition. x is a least squares solution of the system Ax = b if x is such that Ax b is as small as possible. If Ax = b is consistent, i.e. b is in C(A), then a least squares solution x is just an ordinary solution. Interesting case: Ax = b is inconsistent, i.e. b is NOT in C(A). What should we do to find x? replace b with its projection and solve Ax = b. b onto C(A) 4

16 The normal equations Theorem x is a least squares solution of Ax = b if and only if A T Ax = A T b (the normal equation). Proof. 5

17 The normal equations Example. Find the least squares solution to Ax = b, where A = , b =

18 The normal equations Example. Find the least squares solution to Ax = b, where A = , b = Solution. A T A = = and A T b = =

19 The normal equations The normal equation A T Ax = A T b is 2 0 x = Solving it, we obtain x = 1/2. 3/2 7

20 The normal equations Example. Find the least squares solution to Ax = b, where A = , b = What is the projection of b onto C(A)? 8

21 The normal equations Example. Find the least squares solution to Ax = b, where A = , b = What is the projection of b onto C(A)? Solution. and A T A = = A T b = =

22 The normal equations The normal equation A T Ax = A T b is 17 1 x = Solving it, we obtain x =

23 The normal equations The normal equation A T Ax = A T b is 17 1 x = Solving it, we obtain x = 1. 2 The projection of b onto C(A) is Ax = =

24 Why is Ax the projection of b onto C(A)? The projection b of b onto C(A) is b = Ax, with x such that A T Ax = A T b. 10

25 Why is Ax the projection of b onto C(A)? The projection b of b onto C(A) is b = Ax, with x such that A T Ax = A T b. If A has full column rank (columns of A linearly independent), then b = A(A T A) 1 A T b. The projection matrix for projecting onto C(A) is P = A(A T A) 1 A T. 10

26 Application: least squares lines Example. Find β 1, β 2 such that the line y = β 1 + β 2 x best fits the data points (2, 1), (5, 2), (7, 3), (8, 3). 11

27 Application: least squares lines Example. Find β 1, β 2 such that the line y = β 1 + β 2 x best fits the data points (2, 1), (5, 2), (7, 3), (8, 3). Solution. The equations y i = β 1 + β 2 x i in matrix form 1 x 1 1 x 2 1 x 3 1 x 4 design matrix X β 1 = β 2 y 1 y 2 y 3 y 4 observation vector y. 11

28 Application: least squares lines We need to find a least squares solution to Then β = 2. β X T X = = , X T y = = β 1 = 2/7 β 2 5/14. 12