Recall: Dot product on R 2 : u v = (u 1, u 2 ) (v 1, v 2 ) = u 1 v 1 + u 2 v 2, u u = u u 2 2 = u 2. Geometric Meaning:

Transcription

1 Recall: Dot product on R 2 : u v = (u 1, u 2 ) (v 1, v 2 ) = u 1 v 1 + u 2 v 2, u u = u u 2 2 = u 2. Geometric Meaning: u v = u v cos θ. u θ v 1

2 Reason: The opposite side is given by u v. u v 2 = (u v) (u v) = u u v u u v + v v = u 2 + v 2 2u v. By Cosine law: c 2 = a 2 + b 2 2ab cos θ, i.e. u v 2 = u 2 + v 2 2 u v cos θ. Comparing two equalities, we get: u v = u v cos θ. 2

3 Inner Product Generalization of dot product. Direct generalization to R n : u v := u 1 v u n v n = n i=1 u i v i. Using matrix notation: n u i v i = [ u 1... u n ] v 1.. = u T v = v T u. i=1 v n This is called the (standard) inner product on R n. 3

4 Thm 1 (P.359): Let u, v, w R n and c R. Then: (i) u v = v u; (ii) (u + v) w = u w + v w; (iii) (cu) v = c(u v); (iv) u u 0, and u u = 0 iff u = 0. Note: (iv) is sometimes called: positive-definite property. A general inner product is defined using the above 4 properties. For complex inner product, need to add complex conjugate to (i). 4

5 Def: The length (or norm) of v is defined as: v := v v. v = 1 : called unit vectors. Def: The distance between u, v is defined as: dist(u, v) := u v. Def: The angle between u, v is defined as: u, v) := cos 1 u v u v. 5

6 Extra: The General Inner Product Space Let V be a vector space over R or C. Def: An inner product on V is a real/complex-valued function of two vector variables <u, v > such that: (a) <u, v >= <v, u>; (conjugate symmetric) (b) <u + v, w >=<u, w > + <v, w >; (c) <cu, v >= c <u, v >; (linear in the first vector variable) (d) <u, u> 0; and <u, u>= 0 iff u = 0. (positive-definite property) 6

7 Def: A real/complex vector space V equipped with an inner product is called an inner product space. Note: (i) An inner product is conjugate linear in the second vector variable: <u, cv 1 + dv 2 >= c <u, v 1 > + d <u, v 2 >. (ii) If we replace (a) by <u, v > = <v, u>, consider: <iu, iu> = i 2 <u, u>= <u, u>, it will be incompatible with (d). When working with complex inner product space, must take complex conjugate when interchanging u, v. 7

8 Examples of (general) inner product spaces: 1. The dot product on C n : ( : conjugate transpose) <u, v >:= u 1 v u n v n = v u. 2. A non-standard inner product on R 2 : [ <u, v >:= u 1 v 1 u 1 v 2 u 2 v 1 +2u 2 v 2 = v T An inner product on the matrix space M m n : m n <A, B >:= tr(b A) = a jk bjk. j=1 k=1 ] u. 8

9 4. Consider the vector space V of continuous real/complexvalued functions defined on the interval [a, b]. Then the following is an inner product on V : <f, g >:= 1 b a b a f(t)g(t)dt. [In real case, the norm f will give the root-meansquare (r.m.s.) of area bounded by the curve of f and the t-axis over the interval [a, b].] 9

10 Schwarz s inequality: (a 1 b a n b n ) 2 (a a 2 n)(b b 2 n). Pf: The following equation cannot have distinct solution: (a 1 x + b 1 ) (a n x + b n ) 2 = 0 (a a 2 n)x 2 + 2(a 1 b a n b n )x +(b b 2 n) = 0 So 0, and this gives the inequality. 10

11 The Cauchy-Schwarz Inequality: u v u v, and equality holds if, and only if, {u, v} is l.d. Proof: When u 0, set û = 1 u u. Consider w = v (v û)û. v w (v û)û u Obviously, w w = w 2 0 Cauchy-Schwarz Inequality. 11

12 Set k = v û: 0 (v kû) (v kû) = v v 2k(v û) + k 2 (û û) = v 2 k 2, Note that k = v u u, so: k 2 = (v u)2 u 2 v 2 (u v) 2 u 2 v 2. Taking positive square roots, we obtain the result. 12

13 Thm: (Triangle Inequality) For u, v R n : u + v u + v, and equality holds iff one of the vectors is a non-negative scalar multiple of the other. Proof: Consider u + v 2. (u + v) (u + v) = u 2 + 2(u v) + v 2 u u v + v 2 = ( u + v ) 2. Taking square root, we obtain the inequality. 13

14 Orthogonality: Pythagoras Theorem in vector form: u + v 2 = u 2 + v 2 u + v v But in general we have: u so we need u v = 0. u + v 2 = u 2 + 2(u v) + v 2, 14

15 Def: Let u, v be two vectors in R n. When u v = 0, we say that u is orthogonal to v, denoted by u v. This generalizes the concept of perpendicularity. 0 is the only vector that is orthogonal to every vector v in R n. Example: In R 2, we have: [ ] 3 4 [ ] 4. 3 Thm 2 (P.362): u and v are orthogonal iff u + v 2 = u 2 + v 2. 15

16 Common Orthogonality: Def: Let S be a set of vectors in R n. If u is orthogonal to every vector in S, we will say u is orthogonal to S, denoted by u S. i.e. We can regard u to be a common perpendicular to S. Examples: (i) 0 R n. (ii) In R 2, let S = x-axis. Then e 2 S. (iii) In R 3, let S = x-axis. Then both e 2 S and e 3 S. Exercise: Let u, v S. Show that: (i) (au + bv) S for any numbers a, b; (ii) u Span S. *** 16

17 Orthogonal Complement: Def: Let S be a set of vectors in R n. We define: S := {u R n u S}, called the orthogonal complement of S in R n. i.e. S collects all the common perpendiculars to S. Examples: (i) {0} = R n, (R n ) = {0}. (ii) In R 2, let S = x-axis. Then S = y-axis. (iii) In R 3, take S = {e 1 }. Then S = yz-plane. 17

18 Thm: S is always a subspace of R n. Checking: (i) 0 v for every v S. So 0 S. (ii) Pick any u 1, u 2 S. For any scalars a, b R, consider: (au 1 + bu 2 ) v = a(u 1 v) + b(u 2 v) = a 0 + b 0 = 0, whenever v S. So au 1 + bu 2 S (cf. previous exercise). Note: S itself need not be a subspace. Thm: (a) S = (Span S). (b) Span S (S ). Pf: (a) S (Span S) is easy to see, since any vector u Span S must also satisfy u S. 18

19 Now, pick any u S. For every v Span S, write: v = c 1 v c p v p, v i S, i = 1,..., p. Then since u S: u v = c 1 (u v 1 ) c p (u v p ) = 0, and hence u (Span S), so S (Span S) is proved. (b) Pick a vector w Span S, we have l.c.: w = c 1 v c p v p. For any u S : w u = c 1 (v 1 u) c p (v p u) = 0 w (S ). 19

20 Thm 3 (P.363): Let A be an m n matrix. Then: (Row A) = Nul A and (Col A) = Nul A T. Pf: The product Ax can be rewritten as: r 1 r T 1 x Ax =. x = r m. r T m x. So x Nul A x {r T 1,... r T m} x (Row A). Hence (Row A) = Nul A. Apply the result to A T, we obtain: (Col A) = (Row A T ) = Nul A T. 20

21 Orthogonal sets and Orthonormal sets Def: A set S is called orthogonal if any two vectors in S are always orthogonal to each other. Def: A set S is called orthonormal if (i) S is orthogonal, and (ii) each vector in S is of unit length. Example: Orthonormal set: 1 3 1, ,

22 Thm 4 (P.366): An orthogonal set S of non-zero vectors is always linearly independent. Pf: Let S = {u 1, u 2,..., u p } and consider the relation: c 1 u 1 + c 2 u c p u p = 0. Take inner product with u 1, then: c 1 (u 1 u 1 ) + c 2 (u 1 u 2 ) c p (u p u 1 ) = 0 u 1 c 1 u c c p 0 = 0. As u 1 = 0, we must have c 1 = 0. Similarly for other c i. So S must be l.i. 22

23 The method of proof of previous Thm 4 gives: Thm 5 (P.367): Let S = {u 1,..., u p } be an orthogonal set of non-zero vectors and let v Span S. Then: v = v u 1 u 1 2 u v u p u p 2 u p. Pf: Let c 1,..., c p be such that v = c 1 u c p u p. Take inner product with u 1, we have: v u 1 = c 1 (u 1 u 1 ) c p (u p u 1 ) = c 1 u 1 2. So c 1 = v u 1 u 1 2. Similarly for other c i. 23

24 Thm 5 : Let S = {û 1,... û p } be an orthonormal set. Then for any v Span S, we have: v = (v û 1 )û (v û p )û p. Remark: This generalizes our familiar expression in R 3 : v = (v i)i + (v j)j + (v k)k. Example: Express v as a l.c. of the vectors in S: v = 1 2, S = 3 1, 1 2,

25 New method: Compute c 1, c 2, c 3 directly: c 1 = c 2 = c 3 = c 1 = 8 11, c 2 = 6 6 = 1, c 3 = =

26 Exercise: Determine if v Span {u 1, u 2 }. v = 3 2 5, u 1 = 1 2 2, u 2 = *** 26

27 Orthogonal basis and Orthonormal basis Def: A basis for a subspace W is called an orthogonal basis if it is an orthogonal set. Def: A basis for a subspace W is called an orthonormal basis if it is an orthonormal set. Examples: (i) {e 1,..., e n } is an orthonormal basis for R n. [ ] [ ] 3 4 (ii) S = {, } is an orthogonal basis for R ] ] } is an orthonormal basis for R 2. S = {[ , [

28 (iii) The following set S: S = 3 1 1, 1 2 1, is an orthogonal basis for R 3. (iv) The columns of an n n orthogonal matrix A will form an orthonormal basis for R n. Orthogonal matrix: square matrix and A T A = I n. 28

29 Checking: Write A = [ v 1... v n ]. (i, j)-th entry of A T A = vi T v j = v i v j. { 1, i = j, (i, j)-th entry of I n = 0, i j. Above checking also works for non-square matrix: Thm 6 (P.371): The n columns of an m n matrix U are orthonormal iff U T U = I n. But for square matrices: AB = I BA = I. So: (iv) The rows of an n n orthogonal matrix A (written in column form) also form an orthonormal basis for R n. 29

30 Matrices having orthonormal columns are very special: Thm 7 (P.371): Let T : R n R m be a linear transformation given by an m n standard matrix U with orthonormal columns. Then for any x, y R n : a. U x = x (preserving length) b. (Ux) (Uy) = x y (preserving inner product) c. (Ux) (Uy) = 0 iff x y = 0 (preserving orthogonality) Pf: Direct verifications using U T U = I n. Results not true for just orthogonal columns. 30

31 Recall: Let S = {u 1,..., u p } be orthogonal. When v W = Span S, we have: v = v u 1 u 1 2 u v u p u p 2 u p. What happens if v W? LHS RHS, as RHS is always a vector in W. v = RHS is still computable. What is the relation between v and v? 31

32 p LHS = v, RHS = v = Take inner product of RHS with u j : i=1 v u i u i 2 u i. v u j = = ( p i=1 p i=1 ) v u i u i 2 u i u j v u i u i 2 (u i u j ) = v u j u j 2 (u j u j ) = v u j, which is the same as LHS u j. 32

33 In other words, (v v ) u j = 0 for j = 1,..., p. Thm: The vector z = v v is orthogonal to every vector in Span S, i.e. z (Span S). v z u i v W 33

34 Def: Let {u 1,..., u p } be an orthogonal basis for W. each v in R n, the following vector in W : For proj W v := v u 1 u 1 2 u v u p u p 2 u p, is called the orthogonal projection of v onto W. Remark: {u 1,..., u p } must be orthogonal, otherwise RHS will not give us the correct vector v. Note: v = proj W v v W. 34

35 Example: In R 3, consider S = {e 1, e 2 }. Then W = Span S is the xy-plane. For any vector v R 3 : z v v e 1 e 1 2 e 1 x proj W v v e 2 e 2 2 e 2 y proj W v = x y 0 35

36 Exercise: Consider in R 3 and W = Span {u 1, u 2 }. proj W v: Find v = 1 0 1, u 1 = 2 2 1, u 2 = *** 36

37 Def: The decomposition: v = proj W v + (v proj W v), (v proj W v) W, is called the orthogonal decomposition of v w.r.t. W. v z = v proj W v w = proj W v Thm 8 (P.376): Orthogonal decomposition w.r.t. W is the unique way to write v = w + z with w W and z W. 37 W

38 Exercise: Find the orthogonal projection of v onto W = Nul A. 1 A = [ ], 2 v =. 3 4 *** Thm 9 (P.378): Let v R n and let w W. Then we have: v proj W v v w, and equality holds only when w = proj W v. 38

39 Pf: We can rewrite v w as: v w = (v proj W v) + (proj W v w). v v proj W v W proj W v w v w proj W v w Can apply Pythagoras Theorem to the right-angle triangle. 39

40 v w 2 = v proj W v 2 + proj W v w 2 v proj W v 2, and equality holds iff proj W v w = 0 iff w = proj W v. Because of the inequality: v proj W v v w, proj W v sometimes is called the best approximation of v by vectors in W. 40

41 Def: The distance of v to W is defined as: dist(v, W ) := v proj W v. Obviously, v W iff dist(v, W ) = 0. Exercise: Let W = Span {u 1, u 2, u 3 }. Find dist(v, W ): u 1 = 1 1, u 1 2 = 1 1 1, u 1 3 = and v = 1 1 Sol: Remeber to check that {u 1, u 2, u 3 } is orthogonal. ***

42 Extension of Orthogonal Set Let S = {u 1,..., u p } be an orthogonal basis for W = Span S. When W R n, we can find a vector v W and: z = v proj W v 0. This vector z is in W, i.e. will satisfy: z w = 0 for every w W. Hence the following set will again be orthogonal: S {z} = {u 1,..., u p, z}. 42

43 Thm: Span(S {v}) = Span(S {z}). In other words, we can extend an orthogonal set S by adding the vector z. S 1 = {u 1 } orthogonal, v 2 Span S 1, then compute z 2. S 2 = {u 1, z 2 } is again orthogonal. and Span {u 1, v 2 } = Span {u 1, z 2 }. S 2 = {u 1, u 2 } orthogonal, v 3 Span S 2, compute z 3. S 3 = {u 1, u 2, z 3 } is again orthogonal. and Span {u 1, u 2, v 3 } = Span {u 1, u 2, z 3 }.. This is called the Gram-Schmidt orthogonalization process. 43

44 Thm 11 (P.383): Let {x 1,..., x p } l.i.. Define u 1 = x 1 and: u 2 = x 2 x 2 u 1 u 1 2 u 1 u 3 = x 3 x 3 u 2 u 2 2 u 2 x 3 u 1 u 1 2 u 1. u p = x p p 1 i=1 x p u i u i 2 u i. Then {u 1,..., u p } will be orthogonal and for 1 k p: Span {x 1,..., x k } = Span {u 1,..., u k }. 44

45 Notes: (i) Must use {u i } to compute proj Wk x k+1 since the formula: k x k+1 u i proj Wk x k+1 = u i 2 u i, i=1 is only valid for orthogonal set {u i }. (ii) If obtain u k = 0 for some k, i.e. x k = proj W x k, we have: x k Span {x 1,..., x k 1 }. so {x 1,..., x k } will be l.d. instead. (iii) All the u i will be non-zero vectors as {x i } is l.i. 45

46 Example: Apply Gram-Schmidt Process to {x 1, x 2, x 3 }: x 1 = 1 1 0, x 2 = Solution: Take u 1 = x 1. Then:, x 3 = u 2 = x 2 x 2 u 1 u 1 2 u 1 = x u 1 = u 3 = x u u 1 = , 1 46

47 Example: Apply Gram-Schmidt Process to {x 1, x 3, x 2 }: x 1 = 1 1 0, x 3 = Solution: Take u 1 = x 1. Then:, x 2 = u 2 = x 3 x 3 u 1 u 1 2 u 1 = x u 1 = u 3 = x u u 1 = , 47

48 Exercise: Find an orthogonal basis for Col A: A = Sol: First find a basis for Col A (e.g. pivot columns of A). Then apply Gram-Schmidt Process. *** 48

49 Approximation Problems: Solve Ax = b. Due to the presence of errors, a consistent system may appear as an inconsistent system: x 1 + x 2 = 1 x 1 x 2 = 0 2x 1 + 2x 2 = 2 x 1 + x 2 = 1.01 x 1 x 2 = x 1 + 2x 2 = 2.01 Also in practice, exact solutions are usually not necessary. How to obtain a good approximate solution for the above inconsistent system? 49

50 Least squares solution: How to measure the goodness of x 0 as an approximated solution to the system: Ax = b? Minimize the difference x x 0 Problem: But x is unknown... Another way of approximation: x 0 x Ax 0 Ax = b. 50

51 Analysis: Find x 0 such that: Ax 0 = b 0, and b 0 is as close to b as possible. b 0 must be in Col A. b b 0 2 is a sum of squares least squares solution. Best approximation property of orthogonal projection: b proj W b b w for every w in W = Col A. Should take b 0 = proj W b. 51

52 Example: Find the least squares solution of the inconsistent system: x 1 + x 2 = 1.01 x 1 x 2 = x 1 + 2x 2 = 2.01 To compute proj W b, we need an orthogonal basis for W = Col A first. a basis for Col A is: { 1 1 2, }. 52

53 Then by Gram-Schmidt Process, we get an orthogonal basis for W = Col A: { 1 1, Compute b 0 = proj W b: b 0 = } { , }

54 Hence: b 0 = Since b 0 Col A, the system Ax 0 = b 0 must be consistent. Solving Ax 0 = b 0 : Thus we have the following least squares solution: [ ] x 0 =

55 But we have the following result: (Col A) = Nul A T. Then, since we take b 0 = proj Col A b: (b b 0 ) (Col A) (b b 0 ) Nul A T A T (b b 0 ) = 0 A T b 0 = A T b. So, if x 0 is an approximate solution, we have: A T (Ax 0 ) = A T b. The above is usually called the normal equation of Ax = b. 55

56 Thm 13 (P.389): The least squares solutions of Ax = b are the solutions of the normal equation A T Ax = A T b. In the following case, the least square solution will be unique: Thm 14 (P.391): Let A be an m n matrix with rank A = n. Then the n n matrix A T A is invertible. Example: Find again the least squares solution: x 1 + x 2 = 1.01 x 1 x 2 = x 1 + 2x 2 =

57 Solution: Solve the normal equation. Compute: [ ] A T A = 1 1 [ ] =, A T b = [ ] = [ ] So the normal equation is: [ ] [ ] [ 6 4 x = x 2 ] [ x1 x 2 ] = [ ]

58 Least Squares Problems Linear Regression: Fitting data (x i, y i ) with straight line. y To minimize the differences indicated by the red intervals. x 58

59 When a straight line y = c + mx can pass through all the points, it will of course best fit the data. This requires: c + mx 1 = y 1. c + mx n = y n being consistent. 1 x 1 [ ].. c m 1 x n But in general the above system Ax = b is inconsistent. = y 1.. y n 59

60 Measurement of closeness: square sum of y-distances. y 1 (mx 1 + c) y n (mx n + c) 2. Note that this is expressed as b b 0 2, where: b = y 1. y n, b 0 = b 0 Col A since Ax = b 0 is consistent. Use normal equation! c + mx 1. c + mx n. 60

61 Example: Find a straight line that best fits the points: (2, 1), (5, 2), (7, 3), (8, 3), in the sense of minimizing the square-sum of y-distances. Sol: The (inconsistent) system is: [ ] c m = We are going the find its least squares solution. 61

62 Compute: A T A = A T b = [ [ ] ] = = [ ] [ ] 4 22,

63 So the normal equation is: [ ] [ ] 4 22 c m = [ ] 9, 57 which has a unique solution of ( 2 7, 5 14 ). The best fit straight line will be: y = x. 63

64 Polynomial Curve Fitting: Example: Find a polynomial curve of degree at most 2 which best fits the following data: in the sense of least squares. (2, 1), (5, 2), (7, 3), (8, 3), Sol: Consider the general form of the fitting curve: y = a a 1 x + a 2 x 2. 64

65 The curve cannot pass through all the 4 points as: a a a = 1 a a a = 2 a a a = 3 a a a = 3 is inconsistent. Again, use normal equation. 65

66 The corresponding normal equation A T Ax = A T b is: a 0 a a 2 = which has a unique solution of ( , 19 44, ). So the best fitting polynomial is: y = x x2., 66

67 General Curve Fitting: Example: Find a curve in the form c 0 + c 1 sin x + c 2 sin 2x which best fits the following data: ( π 6, 1), (π 4, 2), (π 3, 3), (π 2, 3), in the sense of least squares. Sol: Let y = c c 1 sin x + c 2 sin 2x. The system c c 1 sin π 6 + c 2 sin 2π 6 = 1 c c 1 sin π 4 + c 2 sin 2π 4 = 2 c c 1 sin π 3 + c 2 sin 2π 3 = 3 c c 1 sin π 2 + c 2 sin 2π 2 = 3 is inconsistent. 67

68 Solving A T Ax = A T b... c 0 = c 1 = c 2 = So the best fitting function is: ( ) + ( ) sin x + ( ) sin 2x. 68

69 Extra: Continuous Curve Fitting Find g(x) best fitting a given f(x). g(x) f(x) Try to minimize the difference (area) between two curves. 69

70 To minimize the root-mean-square (r.m.s.) between two curves: 1 b f(x) g(x) b a 2 dx. Given by the following inner product: a of area <f, g >= 1 b a b a f(x)g(x)dx. not in R n, not the standard inner product... No normal equation. But we can use orthogonal projection. 70

71 Recall: Formula of orthogonal projection in general: p <y, u i > proj W y = <u i, u i > u i, i=1 where {u 1,..., u p } is an orthogonal basis of W. Example: Fit f(x) = x over [0, 1] by l.c. of S = {1, sin 2πkx, cos 2πkx; k = 1, 2,..., n} Sol: S is orthogonal under the inner product: (direct checking) <f, g >= f(x)g(x)dx.

72 So compute those <y, u i > : <f(x), 1>= 1 2, <f(x), sin 2πkx>= 1, <f(x), cos 2πkx>= 0. 2πk We also need those <u i, u i > : <1, 1>= 1, <sin 2πkx, sin 2πkx>= 1 2, <cos 2πkx, cos 2πkx>=

73 So the best fitting curve is g(x) = proj W f(x): g(x) = ( ) sin 2πx sin 4πx sin 2nπx π 1 2 n When n = 5:

74 Example: Let f(x) = sgn(x), the sign of x: 1 for x < 0 sgn(x) = 0 for x = 0 1 for x > 0 Find the best r.m.s. approximation function over [ 1, 1] using l.c. of S = {1, sin kπx, cos kπx; k = 1, 2, 3,..., 2n + 1}. Sol: Interval changed. Use new inner product: <f, g >= f(x)g(x)dx. 74

75 Then S is orthogonal (needs another checking) and: <1, 1>= 1, <sin kπx, sin kπx>= 1 2 =<cos kπx, cos kπx>. So, we compute: <sgn(x), 1> = 0; { 0 if k is even, <sgn(x), sin kπx> = if k is odd; <sgn(x), cos kπx> = 0. 2 kπ 75

76 Hence the best r.m.s. approx. to sgn(x) over [ 1, 1] is: 4 π ( sin πx + sin 3πx 3 + sin 5πx sin(2n + 1)πx 2n + 1 ). When 2n + 1 = 9: