Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, 2019 1 / 22
Vector spaces A set of vectors X is called a vector space if for any x, y, z X and any scalars α, β C the following holds 1 x + y X 2 x + y = y + x 3 x + (y + z) = (x + y) + z 4 There exists a unique zero element 0 X, such that x + 0 = x 5 There exists a x X such x + ( x) = 0 6 αx X 7 1 x = x 8 α(x + y) = αx + αy 9 (α + β)x = αx + βx 10 α(βx) = (αβ)x The focus of this course is on the vector spaces R m and C m. Math 566 Linear Algebra Review: Part 2 January 16, 2019 2 / 22
Vector spaces A set of vectors X is called a vector space if for any x, y, z X and any scalars α, β C the following holds 1 x + y X 2 x + y = y + x 3 x + (y + z) = (x + y) + z 4 There exists a unique zero element 0 X, such that x + 0 = x 5 There exists a x X such x + ( x) = 0 6 αx X 7 1 x = x 8 α(x + y) = αx + αy 9 (α + β)x = αx + βx 10 α(βx) = (αβ)x The focus of this course is on the vector spaces R m and C m. Math 566 Linear Algebra Review: Part 2 January 16, 2019 2 / 22
Vector subspaces A subset of vectors Y of C m is called a subspace if for any two vectors x, y Y and any scalars α, β C, the following holds: αx + βy Y Example: The subset of vectors of R 3 given by t x = 2t, 3t where t R, is a subspace of R 3. Math 566 Linear Algebra Review: Part 2 January 16, 2019 3 / 22
Vector subspaces A subset of vectors Y of C m is called a subspace if for any two vectors x, y Y and any scalars α, β C, the following holds: αx + βy Y Example: The subset of vectors of R 3 given by t x = 2t, 3t where t R, is a subspace of R 3. Math 566 Linear Algebra Review: Part 2 January 16, 2019 3 / 22
Linear independence A set of vectors {x 1, x 2,..., x n } is linearly independent if α 1 x 1 + α 2 x 2 + + α n x n = 0 only holds when α 1 = α 2 = = α n = 0. Example: The set of vectors {[ ] [ ]} 1 1, 1 1 is linearly dependent, but the set of vectors {[ ] [ ]} 1 1, 1 1 is not. Math 566 Linear Algebra Review: Part 2 January 16, 2019 4 / 22
Linear independence A set of vectors {x 1, x 2,..., x n } is linearly independent if α 1 x 1 + α 2 x 2 + + α n x n = 0 only holds when α 1 = α 2 = = α n = 0. Example: The set of vectors {[ ] [ ]} 1 1, 1 1 is linearly dependent, but the set of vectors {[ ] [ ]} 1 1, 1 1 is not. Math 566 Linear Algebra Review: Part 2 January 16, 2019 4 / 22
Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, 2019 5 / 22
Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, 2019 5 / 22
Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, 2019 5 / 22
Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, 2019 5 / 22
Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, 2019 6 / 22
Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, 2019 6 / 22
Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, 2019 6 / 22
Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, 2019 6 / 22
Fundamental theorem of linear algebra Part 1: Let A C m n then the column space C(A) and row space C(A T ) have dimension r min(m, n) and the null space N (A) and N (A T ) have dimension n r and m r, respectively. The dimension r of C(A) and C(A T ) is called the rank of A. If m n and r = n then A is said to be of full column rank. If n m and r = m then A is said to be of full row rank. If r < min(m, n) then A is said to be rank deficient. Math 566 Linear Algebra Review: Part 2 January 16, 2019 7 / 22
Fundamental theorem of linear algebra Part 1: Let A C m n then the column space C(A) and row space C(A T ) have dimension r min(m, n) and the null space N (A) and N (A T ) have dimension n r and m r, respectively. The dimension r of C(A) and C(A T ) is called the rank of A. If m n and r = n then A is said to be of full column rank. If n m and r = m then A is said to be of full row rank. If r < min(m, n) then A is said to be rank deficient. Math 566 Linear Algebra Review: Part 2 January 16, 2019 7 / 22
Fundamental theorem of linear algebra Part 1: Let A C m n then the column space C(A) and row space C(A T ) have dimension r min(m, n) and the null space N (A) and N (A T ) have dimension n r and m r, respectively. The dimension r of C(A) and C(A T ) is called the rank of A. If m n and r = n then A is said to be of full column rank. If n m and r = m then A is said to be of full row rank. If r < min(m, n) then A is said to be rank deficient. Math 566 Linear Algebra Review: Part 2 January 16, 2019 7 / 22
Fundamental theorem of linear algebra Part 2: Let A C m n then the column space C(A) is the orthogonal complement of the left null space N (A T ) and the row space C(A T ) is the orthogonal complement of the null space N (A). This means: For any y N (A T ), y T z = 0 for all z C(A). For any x N (A), x T w = 0 for all w C(A T ). C m = C(A) N (A T ) C n = C(A T ) N (A) Math 566 Linear Algebra Review: Part 2 January 16, 2019 8 / 22
Fundamental theorem of linear algebra Part 2: Let A C m n then the column space C(A) is the orthogonal complement of the left null space N (A T ) and the row space C(A T ) is the orthogonal complement of the null space N (A). This means: For any y N (A T ), y T z = 0 for all z C(A). For any x N (A), x T w = 0 for all w C(A T ). C m = C(A) N (A T ) C n = C(A T ) N (A) Math 566 Linear Algebra Review: Part 2 January 16, 2019 8 / 22
Fundamental theorem of linear algebra Part 2: Let A C m n then the column space C(A) is the orthogonal complement of the left null space N (A T ) and the row space C(A T ) is the orthogonal complement of the null space N (A). This means: For any y N (A T ), y T z = 0 for all z C(A). For any x N (A), x T w = 0 for all w C(A T ). C m = C(A) N (A T ) C n = C(A T ) N (A) Math 566 Linear Algebra Review: Part 2 January 16, 2019 8 / 22
Fundamental theorem of linear algebra Part 3: This is about the Singular Value Decomposition (SVD) and we will cover this later. Math 566 Linear Algebra Review: Part 2 January 16, 2019 9 / 22
Invertibility of a matrix Theorem: For a square matrix A C n n the following are equivalent: A 1 exists rank(a) = n Columns of A form a basis for C n, i.e. C(A) = C n Rows of A form a basis for C n, i.e. C(A T ) = C n N (A) = {0} Math 566 Linear Algebra Review: Part 2 January 16, 2019 10 / 22
Eigenvalues and eigenvectors Let A C n n then we call λ C an eigenvalue of A if there exists a vector x C n with x 0 such that Ax = λx. x is called an eigenvector corresponding to the eigenvalue λ. Math 566 Linear Algebra Review: Part 2 January 16, 2019 11 / 22
Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, 2019 12 / 22
Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, 2019 12 / 22
Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, 2019 12 / 22
Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, 2019 12 / 22
Vector norms A vector norm is a scalar quantity that reflects the size of a vector x. The norm of a vector x is denoted as x. There are many ways to define the size of a vector. If x C n, the three most popular are one-norm : x 1 = n x k, k=1 two-norm : x 2 = n x k 2, -norm : k=1 x = max 1 k n x k Math 566 Linear Algebra Review: Part 2 January 16, 2019 13 / 22
Vector norms However a vector norm is defined, it must satisfy the following three properties to be called a norm: 1 x 0 and x = 0 if and only if x = 0 (i.e. x contains all zeros as its entries). 2 αx = α x, for any constant α. 3 x + y x + y, where y C n. This is called the triangle inequality. Math 566 Linear Algebra Review: Part 2 January 16, 2019 14 / 22
Unit vectors A vector x is called a unit vector if its norm is one, i.e. x = 1. Unit vectors will be different depending on the norm applied. Below are several unit vectors in the one, two, and norms for x R 2. 1 Unit vectors with respect to the one norm 1 Unit vectors with respect to the two norm 1 Unit vectors with respect to the norm 0.8 0.8 0.8 0.6 0.6 0.6 0.4 0.4 0.4 0.2 0.2 0.2 x 2 0 x 2 0 x 2 0 0.2 0.2 0.2 0.4 0.4 0.4 0.6 0.6 0.6 0.8 0.8 0.8 1 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 x 1 1 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 x 1 1 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 x 1 (a) One-norm (b) Two-norm (c) -norm Math 566 Linear Algebra Review: Part 2 January 16, 2019 15 / 22
Matrix norms A matrix norm is a scalar quantity that reflects the size of a matrix A C m n. The norm of A is denoted as A. Any matrix norm must satisfy the following four properties: 1 A 0 and A = 0 if and only if A = 0 (i.e. A contains all zeros as its entries). 2 αa = α A, for any constant α. 3 A + B A + B, where B C m n. 4 AB A B, where B C n p. This is called the submultiplicative inequality. Math 566 Linear Algebra Review: Part 2 January 16, 2019 16 / 22
Matrix norms Each vector norm induces a matrix norm according to the following definition: Ax p A p = max x p 0 x p where x C n and p = 1, 2,.... = max x p=1 Ax p, Induced norms describe how the matrix stretches unit vectors with respect to that norm. Math 566 Linear Algebra Review: Part 2 January 16, 2019 17 / 22
Induced matrix norms Two popular and easy to define induced matrix norms are One-norm : -norm : A 1 = max 1 k n A = max m a jk, j=1 1 j m k=1 n a jk. The one-norm corresponds to the maximum of the one norm of every column. The -norm corresponds to the maximum of the one norm of every row. Math 566 Linear Algebra Review: Part 2 January 16, 2019 18 / 22
Induced matrix norms The two-norm is also a popular induced matrix norm, but is not easily derived. Definition: Let B C n n then the spectral radius of B is ρ(b) = max 1 j n λ j, where λ j are the n eigenvalues of B. Theorem: Let A C m n then A 2 = ρ(a A). Math 566 Linear Algebra Review: Part 2 January 16, 2019 19 / 22
Geometric illustration of the two-norm [ ] 4 1 Let A = and consider applying A to all vectors x such that 1 3 x 2 = 1. 1 4 0.5 2 0 0-0.5-2 -4-1 -1-0.5 0 0.5 1-4 -2 0 2 4 (a) Unit vectors w.r.t the two-norm (b) Transformation of unit vectors after applying A The semi-major axis of the ellipse on the right (marked in black) is the two-norm of A. Math 566 Linear Algebra Review: Part 2 January 16, 2019 20 / 22
Non-induced matrix norms The most popular matrix norm that is not an induced norm is the Frobenius norm: m n A F = a jk 2. j=1 k=1 Math 566 Linear Algebra Review: Part 2 January 16, 2019 21 / 22
Important results on matrix norms The following are some useful inequalities involving matrix norms. Here A R m n : ρ(a) A for any matrix norm Ax A x 1 m A 1 A 2 n A 1 1 n A A 2 m A A 2 A F n A 2 A 2 A 1 A Math 566 Linear Algebra Review: Part 2 January 16, 2019 22 / 22