AMS526: Numerical Analysis I (Numerical Linear Algebra)

AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 2: Orthogonal Vectors and Matrices; Vector Norms Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 11

Outline 1 Orthogonal Vectors and Matrices 2 Vector Norms Xiangmin Jiao Numerical Analysis I 2 / 11

Inner Product Inner product (dot product) of two column vectors u, v C m is u v = m i=1 ūiv i In contrast, outer product of u and v is uv Note that cross product is different Xiangmin Jiao Numerical Analysis I 3 / 11

Inner Product Inner product (dot product) of two column vectors u, v C m is u v = m i=1 ūiv i In contrast, outer product of u and v is uv Note that cross product is different Euclidean length of u is the square root of the inner product of u with itself, i.e., u u Inner product of two unit vectors u and v is the cosine of the angle α between u and v, i.e., cos α = u v u v Xiangmin Jiao Numerical Analysis I 3 / 11

Orthogonal Vectors Definition A pair of vectors are orthogonal if x y = 0. In other words, the angle between them is 90 degrees Xiangmin Jiao Numerical Analysis I 4 / 11

Orthogonal Vectors Definition A pair of vectors are orthogonal if x y = 0. In other words, the angle between them is 90 degrees Definition Two sets of vectors X and Y are orthogonal if every x X is orthogonal to every y Y. Definition A set of nonzero vectors S is orthogonal if they are pairwise orthogonal. They are orthonormal if it is orthogonal and in addition each vector has unit Euclidean length. Xiangmin Jiao Numerical Analysis I 4 / 11

Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Xiangmin Jiao Numerical Analysis I 5 / 11

Orthogonal Vectors Theorem The vectors in an orthogonal set S are linearly independent. Proof. Prove by contradiction. If a vector can be expressed as linear combination of the other vectors in the set, then it is orthogonal to itself. Question: If the column vectors of an m n matrix A are orthogonal, what is the rank of A? Xiangmin Jiao Numerical Analysis I 5 / 11

Components of Vector Given an orthonormal set {q 1, q 2,..., q m } forming a basis of C m, vector v can be decomposed into orthogonal components as v = m i=1 (q i v)q i Xiangmin Jiao Numerical Analysis I 6 / 11

Components of Vector Given an orthonormal set {q 1, q 2,..., q m } forming a basis of C m, vector v can be decomposed into orthogonal components as v = m i=1 (q i v)q i Another way to express the condition is v = m i=1 (q iq i )v q i q i is an orthogonal projection matrix. Note that it is NOT an orthogonal matrix. More generally, given an orthonormal set {q 1, q 2,..., q n } with n m, we have v = r + n (q i v)q i = r + i=1 n (q i q i )v and r q i = 0, 1 i n i=1 Let Q be composed of column vectors {q 1, q 2,..., q n }. QQ = n i=1 (q iq i ) is an orthogonal projection matrix. Xiangmin Jiao Numerical Analysis I 6 / 11

Unitary Matrices Definition A matrix is unitary if Q = Q 1, i.e., if Q Q = QQ = I. In the real case, we say the matrix is orthogonal. Its column vectors are orthonormal. In other words, q i q j = δ ij, the Kronecker delta. Question: What is the geometric meaning of multiplication by a unitary matrix? Answer: It preserves angles and Euclidean length. In the real case, multiplication by an orthogonal matrix Q is a rotation (if det(q) = 1) or reflection (if det(q) = 1). Xiangmin Jiao Numerical Analysis I 7 / 11

Outline 1 Orthogonal Vectors and Matrices 2 Vector Norms Xiangmin Jiao Numerical Analysis I 8 / 11

Definition of Norms Norm captures size of vector or distance between vectors There are many different measures for sizes but a norm must satisfy some requirements: Xiangmin Jiao Numerical Analysis I 9 / 11

Definition of Norms Norm captures size of vector or distance between vectors There are many different measures for sizes but a norm must satisfy some requirements: Definition A norm is a function : C m R that assigns a real-valued length to each vector. It must satisfy the following conditions: (1) x 0, and x = 0 only if x = 0, (2) x + y x + y, (3) αx = α x. Xiangmin Jiao Numerical Analysis I 9 / 11

p-norms Euclidean length is a special case of p-norms, defined as x p = for 1 p ( m ) 1/p x i p i=1 Xiangmin Jiao Numerical Analysis I 10 / 11

p-norms Euclidean length is a special case of p-norms, defined as x p = for 1 p ( m ) 1/p x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) Xiangmin Jiao Numerical Analysis I 10 / 11

p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i Xiangmin Jiao Numerical Analysis I 10 / 11

p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i -norm: x. What is its value? Xiangmin Jiao Numerical Analysis I 10 / 11

p-norms Euclidean length is a special case of p-norms, defined as for 1 p ( m ) 1/p x p = x i p i=1 Euclidean norm is 2-norm x 2 (i.e., p = 2) 1-norm: x 1 = m i=1 x i -norm: x. What is its value? Answer: x = max 1 i m x i Xiangmin Jiao Numerical Analysis I 10 / 11

Weighted p-norms A generalization of p-norm is weighted p-norm, which assigns different weights (priorities) to different components. It is anisotropic instead of isotropic Algebraically, x W = W x, where W is diagonal matrix with ith diagonal entry w i 0 being weight for ith component In other words, What happens if we allow w i = 0? ( m ) 1/p x W = w i x i p i=1 Xiangmin Jiao Numerical Analysis I 11 / 11