AMS526: Numerical Analysis I (Numerical Linear Algebra)

Transcription

1 AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 2: Orthogonal Vectors and Matrices; Vector Norms Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 11

2 Outline 1 Orthogonal Vectors and Matrices 2 Vector Norms Xiangmin Jiao Numerical Analysis I 2 / 11

3 Inner Product Inner product (dot product) of two column vectors u, v C m is u v = m i=1 ūiv i In contrast, outer product of u and v is uv Note that cross product is different Xiangmin Jiao Numerical Analysis I 3 / 11

4 Inner Product Inner product (dot product) of two column vectors u, v C m is u v = m i=1 ūiv i In contrast, outer product of u and v is uv Note that cross product is different Euclidean length of u is the square root of the inner product of u with itself, i.e., u u Xiangmin Jiao Numerical Analysis I 3 / 11

5 Inner Product Inner product (dot product) of two column vectors u, v C m is u v = m i=1 ūiv i In contrast, outer product of u and v is uv Note that cross product is different Euclidean length of u is the square root of the inner product of u with itself, i.e., u u Inner product of two unit vectors u and v is the cosine of the angle α between u and v, i.e., cos α = u v u v Xiangmin Jiao Numerical Analysis I 3 / 11

6 Inner Product Inner product (dot product) of two column vectors u, v C m is u v = m i=1 ūiv i In contrast, outer product of u and v is uv Note that cross product is different Euclidean length of u is the square root of the inner product of u with itself, i.e., u u Inner product of two unit vectors u and v is the cosine of the angle α between u and v, i.e., cos α = u v u v Inner product is bilinear, in the sense that it is linear in each vertex separately: (u 1 + u 2 ) v = u 1v + u 2v u (v 1 + v 2 ) = u v 1 + u v 2 (αu) (βv) = ᾱβu v Xiangmin Jiao Numerical Analysis I 3 / 11

7 Orthogonal Vectors Definition A pair of vectors are orthogonal if x y = 0. In other words, the angle between them is 90 degrees Xiangmin Jiao Numerical Analysis I 4 / 11

8 Orthogonal Vectors Definition A pair of vectors are orthogonal if x y = 0. In other words, the angle between them is 90 degrees Definition Two sets of vectors X and Y are orthogonal if every x X is orthogonal to every y Y. Xiangmin Jiao Numerical Analysis I 4 / 11

9 Orthogonal Vectors Definition A pair of vectors are orthogonal if x y = 0. In other words, the angle between them is 90 degrees Definition Two sets of vectors X and Y are orthogonal if every x X is orthogonal to every y Y. Definition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length. Xiangmin Jiao Numerical Analysis I 4 / 11

10 Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Xiangmin Jiao Numerical Analysis I 5 / 11

11 Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself. Question: If the column vectors of an m n matrix A are orthogonal, what is the rank of A? Xiangmin Jiao Numerical Analysis I 5 / 11

12 Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself. Question: If the column vectors of an m n matrix A are orthogonal, what is the rank of A? Answer: n = min{m, n}. In other words, A has full rank. Xiangmin Jiao Numerical Analysis I 5 / 11

13 Components of Vector Given an orthonormal set {q 1, q 2,..., q m } forming a basis of C m, vector v can be decomposed into orthogonal components as v = m i=1 (q i v)q i Xiangmin Jiao Numerical Analysis I 6 / 11

14 Components of Vector Given an orthonormal set {q 1, q 2,..., q m } forming a basis of C m, vector v can be decomposed into orthogonal components as v = m i=1 (q i v)q i Another way to express the condition is v = m i=1 (q iq i )v q i q i is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix. Xiangmin Jiao Numerical Analysis I 6 / 11

15 Components of Vector Given an orthonormal set {q 1, q 2,..., q m } forming a basis of C m, vector v can be decomposed into orthogonal components as v = m i=1 (q i v)q i Another way to express the condition is v = m i=1 (q iq i )v q i q i is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix. More generally, given an orthonormal set {q 1, q 2,..., q n } with n m, we have v = r + n (q i v)q i = r + i=1 n (q i q i )v and r q i = 0, 1 i n i=1 Let Q be composed of column vectors {q 1, q 2,..., q n }. QQ = n i=1 (q iq i ) is an orthogonal projection matrix. Xiangmin Jiao Numerical Analysis I 6 / 11

16 Unitary Matrices Definition A matrix is unitary if Q = Q 1, i.e., if Q Q = QQ = I. In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. In other words, q i q j = δ ij, the Kronecker delta. Xiangmin Jiao Numerical Analysis I 7 / 11

17 Unitary Matrices Definition A matrix is unitary if Q = Q 1, i.e., if Q Q = QQ = I. In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. In other words, q i q j = δ ij, the Kronecker delta. Question: What is the geometric meaning of multiplication by a unitary matrix? Xiangmin Jiao Numerical Analysis I 7 / 11

18 Unitary Matrices Definition A matrix is unitary if Q = Q 1, i.e., if Q Q = QQ = I. In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. In other words, q i q j = δ ij, the Kronecker delta. Question: What is the geometric meaning of multiplication by a unitary matrix? Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(q) = 1) or reflection (if det(q) = 1). Xiangmin Jiao Numerical Analysis I 7 / 11

19 Outline 1 Orthogonal Vectors and Matrices 2 Vector Norms Xiangmin Jiao Numerical Analysis I 8 / 11

20 Definition of Norms Norm captures size of vector or distance between vectors There are many different measures for sizes but a norm must satisfy some requirements: Xiangmin Jiao Numerical Analysis I 9 / 11

21 Definition of Norms Norm captures size of vector or distance between vectors There are many different measures for sizes but a norm must satisfy some requirements: Definition A norm is a function : C m R that assigns a real-valued length to each vector. It must satisfy the following conditions: (1) x 0, and x = 0 only if x = 0, (2) x + y x + y, (3) αx = α x. Xiangmin Jiao Numerical Analysis I 9 / 11

22 Definition of Norms Norm captures size of vector or distance between vectors There are many different measures for sizes but a norm must satisfy some requirements: Definition A norm is a function : C m R that assigns a real-valued length to each vector. It must satisfy the following conditions: (1) x 0, and x = 0 only if x = 0, (2) x + y x + y, (3) αx = α x. An example is Euclidean length (i.e, x = m i=1 x i 2 ) Xiangmin Jiao Numerical Analysis I 9 / 11

23 p-norms Euclidean length is a special case of p-norms, defined as x p = for 1 p ( m ) 1/p x i p i=1 Xiangmin Jiao Numerical Analysis I 10 / 11

24 p-norms Euclidean length is a special case of p-norms, defined as x p = for 1 p ( m ) 1/p x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) Xiangmin Jiao Numerical Analysis I 10 / 11

25 p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i Xiangmin Jiao Numerical Analysis I 10 / 11

26 p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i -norm: x. What is its value? Xiangmin Jiao Numerical Analysis I 10 / 11

27 p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i -norm: x. What is its value? Answer: x = max 1 i m x i Xiangmin Jiao Numerical Analysis I 10 / 11

28 p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i -norm: x. What is its value? Answer: x = max 1 i m x i Why we require p 1? What happens if 0 p < 1? Xiangmin Jiao Numerical Analysis I 10 / 11

29 Weighted p-norms A generalization of p-norm is weighted p-norm, which assigns different weights (priorities) to different components. It is anisotropic instead of isotropic Algebraically, x W = W x, where W is diagonal matrix with ith diagonal entry w i 0 being weight for ith component In other words, What happens if we allow w i = 0? ( m ) 1/p x W = w i x i p i=1 Xiangmin Jiao Numerical Analysis I 11 / 11

30 Weighted p-norms A generalization of p-norm is weighted p-norm, which assigns different weights (priorities) to different components. It is anisotropic instead of isotropic Algebraically, x W = W x, where W is diagonal matrix with ith diagonal entry w i 0 being weight for ith component In other words, What happens if we allow w i = 0? ( m ) 1/p x W = w i x i p Can we further generalize it to allow W being arbitrary matrix? No. But we can allow W to be arbitrary nonsingular matrix. i=1 Xiangmin Jiao Numerical Analysis I 11 / 11