Review of Some Concepts from Linear Algebra: Part 2


 Monica Rodgers
 5 months ago
 Views:
Transcription
1 Review of Some Concepts from Linear Algebra: Part 2 Department of Mathematics Boise State University January 16, 2019 Math 566 Linear Algebra Review: Part 2 January 16, / 22
2 Vector spaces A set of vectors X is called a vector space if for any x, y, z X and any scalars α, β C the following holds 1 x + y X 2 x + y = y + x 3 x + (y + z) = (x + y) + z 4 There exists a unique zero element 0 X, such that x + 0 = x 5 There exists a x X such x + ( x) = 0 6 αx X 7 1 x = x 8 α(x + y) = αx + αy 9 (α + β)x = αx + βx 10 α(βx) = (αβ)x The focus of this course is on the vector spaces R m and C m. Math 566 Linear Algebra Review: Part 2 January 16, / 22
3 Vector spaces A set of vectors X is called a vector space if for any x, y, z X and any scalars α, β C the following holds 1 x + y X 2 x + y = y + x 3 x + (y + z) = (x + y) + z 4 There exists a unique zero element 0 X, such that x + 0 = x 5 There exists a x X such x + ( x) = 0 6 αx X 7 1 x = x 8 α(x + y) = αx + αy 9 (α + β)x = αx + βx 10 α(βx) = (αβ)x The focus of this course is on the vector spaces R m and C m. Math 566 Linear Algebra Review: Part 2 January 16, / 22
4 Vector subspaces A subset of vectors Y of C m is called a subspace if for any two vectors x, y Y and any scalars α, β C, the following holds: αx + βy Y Example: The subset of vectors of R 3 given by t x = 2t, 3t where t R, is a subspace of R 3. Math 566 Linear Algebra Review: Part 2 January 16, / 22
5 Vector subspaces A subset of vectors Y of C m is called a subspace if for any two vectors x, y Y and any scalars α, β C, the following holds: αx + βy Y Example: The subset of vectors of R 3 given by t x = 2t, 3t where t R, is a subspace of R 3. Math 566 Linear Algebra Review: Part 2 January 16, / 22
6 Linear independence A set of vectors {x 1, x 2,..., x n } is linearly independent if α 1 x 1 + α 2 x α n x n = 0 only holds when α 1 = α 2 = = α n = 0. Example: The set of vectors {[ ] [ ]} 1 1, 1 1 is linearly dependent, but the set of vectors {[ ] [ ]} 1 1, 1 1 is not. Math 566 Linear Algebra Review: Part 2 January 16, / 22
7 Linear independence A set of vectors {x 1, x 2,..., x n } is linearly independent if α 1 x 1 + α 2 x α n x n = 0 only holds when α 1 = α 2 = = α n = 0. Example: The set of vectors {[ ] [ ]} 1 1, 1 1 is linearly dependent, but the set of vectors {[ ] [ ]} 1 1, 1 1 is not. Math 566 Linear Algebra Review: Part 2 January 16, / 22
8 Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, / 22
9 Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, / 22
10 Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, / 22
11 Span, basis, dimension The span of a set of vectors is the subspace formed from all linear combinations of the vectors. A basis for a subspace X is a set of linearly independent vectors that span X. The dimension of a subspace X is the number of vectors in any basis for X. The standard basis for R n or C n is given by the {e 1, e 2,, e n }, where e R n and { 0 i j (e i ) j =, i, j = 1, 2,..., n, 1 i = j i.e. e i contains all zeros except for one 1 in the i th entry. Math 566 Linear Algebra Review: Part 2 January 16, / 22
12 Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, / 22
13 Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, / 22
14 Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, / 22
15 Fundamental subspaces of a matrix Let A C m n then the following are the four fundamental subspaces of A: Column space of A (or range of A): the subspace of C m formed by the span of the columns of A. Denoted as C(A). Row space of A (or range of A T ): the subspace of C n formed by the span of the rows of A. Denoted by C(A T ). Null space of A: The subspace of vectors x C n that satisfy Ax = 0. Denoted by N (A). Left null space of A: The subspace of vectors y C m that satisfy y T A = 0. Denoted by N (A T ). Math 566 Linear Algebra Review: Part 2 January 16, / 22
16 Fundamental theorem of linear algebra Part 1: Let A C m n then the column space C(A) and row space C(A T ) have dimension r min(m, n) and the null space N (A) and N (A T ) have dimension n r and m r, respectively. The dimension r of C(A) and C(A T ) is called the rank of A. If m n and r = n then A is said to be of full column rank. If n m and r = m then A is said to be of full row rank. If r < min(m, n) then A is said to be rank deficient. Math 566 Linear Algebra Review: Part 2 January 16, / 22
17 Fundamental theorem of linear algebra Part 1: Let A C m n then the column space C(A) and row space C(A T ) have dimension r min(m, n) and the null space N (A) and N (A T ) have dimension n r and m r, respectively. The dimension r of C(A) and C(A T ) is called the rank of A. If m n and r = n then A is said to be of full column rank. If n m and r = m then A is said to be of full row rank. If r < min(m, n) then A is said to be rank deficient. Math 566 Linear Algebra Review: Part 2 January 16, / 22
18 Fundamental theorem of linear algebra Part 1: Let A C m n then the column space C(A) and row space C(A T ) have dimension r min(m, n) and the null space N (A) and N (A T ) have dimension n r and m r, respectively. The dimension r of C(A) and C(A T ) is called the rank of A. If m n and r = n then A is said to be of full column rank. If n m and r = m then A is said to be of full row rank. If r < min(m, n) then A is said to be rank deficient. Math 566 Linear Algebra Review: Part 2 January 16, / 22
19 Fundamental theorem of linear algebra Part 2: Let A C m n then the column space C(A) is the orthogonal complement of the left null space N (A T ) and the row space C(A T ) is the orthogonal complement of the null space N (A). This means: For any y N (A T ), y T z = 0 for all z C(A). For any x N (A), x T w = 0 for all w C(A T ). C m = C(A) N (A T ) C n = C(A T ) N (A) Math 566 Linear Algebra Review: Part 2 January 16, / 22
20 Fundamental theorem of linear algebra Part 2: Let A C m n then the column space C(A) is the orthogonal complement of the left null space N (A T ) and the row space C(A T ) is the orthogonal complement of the null space N (A). This means: For any y N (A T ), y T z = 0 for all z C(A). For any x N (A), x T w = 0 for all w C(A T ). C m = C(A) N (A T ) C n = C(A T ) N (A) Math 566 Linear Algebra Review: Part 2 January 16, / 22
21 Fundamental theorem of linear algebra Part 2: Let A C m n then the column space C(A) is the orthogonal complement of the left null space N (A T ) and the row space C(A T ) is the orthogonal complement of the null space N (A). This means: For any y N (A T ), y T z = 0 for all z C(A). For any x N (A), x T w = 0 for all w C(A T ). C m = C(A) N (A T ) C n = C(A T ) N (A) Math 566 Linear Algebra Review: Part 2 January 16, / 22
22 Fundamental theorem of linear algebra Part 3: This is about the Singular Value Decomposition (SVD) and we will cover this later. Math 566 Linear Algebra Review: Part 2 January 16, / 22
23 Invertibility of a matrix Theorem: For a square matrix A C n n the following are equivalent: A 1 exists rank(a) = n Columns of A form a basis for C n, i.e. C(A) = C n Rows of A form a basis for C n, i.e. C(A T ) = C n N (A) = {0} Math 566 Linear Algebra Review: Part 2 January 16, / 22
24 Eigenvalues and eigenvectors Let A C n n then we call λ C an eigenvalue of A if there exists a vector x C n with x 0 such that Ax = λx. x is called an eigenvector corresponding to the eigenvalue λ. Math 566 Linear Algebra Review: Part 2 January 16, / 22
25 Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, / 22
26 Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, / 22
27 Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, / 22
28 Eigenvalues and eigenvectors The scalar λ C is an eigenvalue of A C n n if and only if det(a Iλ) = 0, where I is the identity matrix. p n (λ) = det(a Iλ) is a polynomial in λ of degree n. p n (λ) is called the characteristic polynomial of A. We never use p n (λ) to numerically compute the eigenvalues of A. We will discuss the correct way to compute the eigenvalues. Math 566 Linear Algebra Review: Part 2 January 16, / 22
29 Vector norms A vector norm is a scalar quantity that reflects the size of a vector x. The norm of a vector x is denoted as x. There are many ways to define the size of a vector. If x C n, the three most popular are onenorm : x 1 = n x k, k=1 twonorm : x 2 = n x k 2, norm : k=1 x = max 1 k n x k Math 566 Linear Algebra Review: Part 2 January 16, / 22
30 Vector norms However a vector norm is defined, it must satisfy the following three properties to be called a norm: 1 x 0 and x = 0 if and only if x = 0 (i.e. x contains all zeros as its entries). 2 αx = α x, for any constant α. 3 x + y x + y, where y C n. This is called the triangle inequality. Math 566 Linear Algebra Review: Part 2 January 16, / 22
31 Unit vectors A vector x is called a unit vector if its norm is one, i.e. x = 1. Unit vectors will be different depending on the norm applied. Below are several unit vectors in the one, two, and norms for x R 2. 1 Unit vectors with respect to the one norm 1 Unit vectors with respect to the two norm 1 Unit vectors with respect to the norm x 2 0 x 2 0 x x x x 1 (a) Onenorm (b) Twonorm (c) norm Math 566 Linear Algebra Review: Part 2 January 16, / 22
32 Matrix norms A matrix norm is a scalar quantity that reflects the size of a matrix A C m n. The norm of A is denoted as A. Any matrix norm must satisfy the following four properties: 1 A 0 and A = 0 if and only if A = 0 (i.e. A contains all zeros as its entries). 2 αa = α A, for any constant α. 3 A + B A + B, where B C m n. 4 AB A B, where B C n p. This is called the submultiplicative inequality. Math 566 Linear Algebra Review: Part 2 January 16, / 22
33 Matrix norms Each vector norm induces a matrix norm according to the following definition: Ax p A p = max x p 0 x p where x C n and p = 1, 2,.... = max x p=1 Ax p, Induced norms describe how the matrix stretches unit vectors with respect to that norm. Math 566 Linear Algebra Review: Part 2 January 16, / 22
34 Induced matrix norms Two popular and easy to define induced matrix norms are Onenorm : norm : A 1 = max 1 k n A = max m a jk, j=1 1 j m k=1 n a jk. The onenorm corresponds to the maximum of the one norm of every column. The norm corresponds to the maximum of the one norm of every row. Math 566 Linear Algebra Review: Part 2 January 16, / 22
35 Induced matrix norms The twonorm is also a popular induced matrix norm, but is not easily derived. Definition: Let B C n n then the spectral radius of B is ρ(b) = max 1 j n λ j, where λ j are the n eigenvalues of B. Theorem: Let A C m n then A 2 = ρ(a A). Math 566 Linear Algebra Review: Part 2 January 16, / 22
36 Geometric illustration of the twonorm [ ] 4 1 Let A = and consider applying A to all vectors x such that 1 3 x 2 = (a) Unit vectors w.r.t the twonorm (b) Transformation of unit vectors after applying A The semimajor axis of the ellipse on the right (marked in black) is the twonorm of A. Math 566 Linear Algebra Review: Part 2 January 16, / 22
37 Noninduced matrix norms The most popular matrix norm that is not an induced norm is the Frobenius norm: m n A F = a jk 2. j=1 k=1 Math 566 Linear Algebra Review: Part 2 January 16, / 22
38 Important results on matrix norms The following are some useful inequalities involving matrix norms. Here A R m n : ρ(a) A for any matrix norm Ax A x 1 m A 1 A 2 n A 1 1 n A A 2 m A A 2 A F n A 2 A 2 A 1 A Math 566 Linear Algebra Review: Part 2 January 16, / 22