Linear Algebra in A Nutshell

Transcription

1 Linear Algebra in A Nutshell Gilbert Strang Computational Science and Engineering Wellesley-Cambridge Press

2 Outline 1 Matrix Singularity 2 Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations 3 The Four Fundamental Subspaces 4 Dimension and Basis

3 Outline 1 Matrix Singularity 2 Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations 3 The Four Fundamental Subspaces 4 Dimension and Basis

4 Invertibility of an n-by-n matrix A is invertible The columns are independent The rows are independent The determinant is not zero Ax = 0 has one solution x = 0 Ax = b has one solution A 1 b A has n (nonzero) pivots A has full rank A is not invertible The columns are dependent The rows are dependent The determinant is zero Ax = 0 has infinitely many solutions Ax = b has no solution or infinitely many A has r < n pivots A has rank r < n

5 Invertibility of an n-by-n matrix (cont.) The reduced row echelon form is R = I The column space is all of R n The row space is all of R n All eigenvalues are nonzero A T A is symmetric positive definite A has n (positive) singular values R has at least one zero row The column space has dimension r < n The row space has dimension r < n Zero is an eigenvalue of A A T A is only semidefinite A has r < n nonzero (positive) singular values

6 Outline 1 Matrix Singularity 2 Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations 3 The Four Fundamental Subspaces 4 Dimension and Basis

7 Think of Ax a column at time Instead of thinking of Ax inner products, think of Ax a linear combination of columns of A: [ ] [ ] [ ] [ ] 1 2 C 1 2 = C + D 3 6 D 3 6

8 Think of Ax a column at time Instead of thinking of Ax inner products, think of Ax a linear combination of columns of A: [ ] [ ] [ ] [ ] 1 2 C 1 2 = C + D 3 6 D 3 6 In particular, [ ] [ ] = first column [ ] [ 0 1 ] = last column

9 In general matrix-vector multiplication: y = Ax

10 In general matrix-vector multiplication: y = Ax column version y = zeros(m,1); for j=1:n y = y + x(j)*a(:,j); endfor

11 In general matrix-vector multiplication: y = Ax column version y = zeros(m,1); for j=1:n y = y + x(j)*a(:,j); endfor matrix-matrix multiplication: C = AB

12 In general matrix-vector multiplication: y = Ax column version y = zeros(m,1); for j=1:n y = y + x(j)*a(:,j); endfor matrix-matrix multiplication: C = AB column version (Fortran, step 1) C(:,j) = A*B(:,j)

13 Row version vector-matrix multiplication: v T = u T A

14 Row version vector-matrix multiplication: v T = u T A row version v = zeros(1,n); for i=1:m v = v + u(i)*a(i,:); endfor

15 Row version vector-matrix multiplication: v T = u T A row version v = zeros(1,n); for i=1:m v = v + u(i)*a(i,:); endfor matrix-matrix multiplication: C = AB

16 Row version vector-matrix multiplication: v T = u T A row version v = zeros(1,n); for i=1:m v = v + u(i)*a(i,:); endfor matrix-matrix multiplication: C = AB row version (C) C(i,:) = A(i,:)*B

17 Rank and nullspace Suppose A is an m-by-n matrix, Ax = 0 has at least one (trivial) solution, namely x = 0. There are other (nontrivial) solutions in case n > m. Even if m = n, there might be nonzero solutions to Ax = 0 when A is not invertible. It is the number r of independent rows or columns that counts.

18 Rank and nullspace (cont.) Rank The number r of independent rows or columns is the rank of A (r m and r n, that is, r min(m, n)).

19 Rank and nullspace (cont.) Rank The number r of independent rows or columns is the rank of A (r m and r n, that is, r min(m, n)). Null space The null space of A is the set of all solutions x to Ax = 0. x in nullspace x 1 (column 1) + + x n (column n) = 0

20 Rank and nullspace (cont.) Rank The number r of independent rows or columns is the rank of A (r m and r n, that is, r min(m, n)). Null space The null space of A is the set of all solutions x to Ax = 0. x in nullspace x 1 (column 1) + + x n (column n) = 0 This nullspace N(A) contains only x = 0 when the columns of A are independent. In that case A is of full column rank r = n.

21 Rank and nullspace (cont.) Example. The nullspace of Question Find the line. [ ] is a line.

22 Rank and nullspace (cont.) Example. The nullspace of Question Find the line. [ ] is a line. We often require that A is of full column rank. In that case, A T A, n-by-n, is invertible, and symmetric and positive definite.

23 Column (range) space Column (range) space The column (range) space contains all combinations of the columns. Example. The column space of [ ] 1. 3 [ ] is always through

24 Column (range) space (cont.) In other words, the column space C(A) contains all possible products Ax, thus also called the range space R(A). For an m-by-n matrix, the column space is in m-dimensional space. The word space indicates: Any combination of vectors in the space stays in the space. The zero combination is allowed, so x = 0 is in every space.

25 Solution to linear equations A solution to Ax = b calls for a linear combination of the columns that equals b. Thus, if b is in R(A), there is a solution to Ax = b, otherwise, Ax = b has no solution.

26 Solution to linear equations A solution to Ax = b calls for a linear combination of the columns that equals b. Thus, if b is in R(A), there is a solution to Ax = b, otherwise, Ax = b has no solution. How do we write down all solutions, when b R(A)?

27 Solution to linear equations A solution to Ax = b calls for a linear combination of the columns that equals b. Thus, if b is in R(A), there is a solution to Ax = b, otherwise, Ax = b has no solution. How do we write down all solutions, when b R(A)? Suppose x p is a particular solution to Ax = b. Any vector x n in the nullspace solves Ax = 0. The complete solution to Ax = b has the form: x = (one x p ) + (all x n ).

28 Solution to linear equations (cont.) Questions Find the complete solution to [ Does the complete solution form a space? ] [ 5 x = 15 ].

29 Comments Suppose A is a square invertible matrix, then the nullspace only contains x n = 0. The complete solution x = A 1 b + 0 = A 1 b.

30 Comments Suppose A is a square invertible matrix, then the nullspace only contains x n = 0. The complete solution x = A 1 b + 0 = A 1 b. When Ax = b has infinitely many solutions, the shortest x always lies in the row space of A. A particular solution can be found by the pseudo-inverse pinv(a).

31 Comments Suppose A is a square invertible matrix, then the nullspace only contains x n = 0. The complete solution x = A 1 b + 0 = A 1 b. When Ax = b has infinitely many solutions, the shortest x always lies in the row space of A. A particular solution can be found by the pseudo-inverse pinv(a). Suppose A is tall and thin (m > n). The columns are likely to be independent. But if b is not in the column space, Ax = b has no solution. The least squares method minimizes Ax b 2 2 by solving AT A x = A T b.

32 Outline 1 Matrix Singularity 2 Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations 3 The Four Fundamental Subspaces 4 Dimension and Basis

33 Four spaces of A The column space R(A) of A is a subspace of R m. The nullspace N(A) of A is a subspace of R n. In addition, we consider N(A T ), a subspace of R m and R(A T ), a subspace of R n. Four spaces of A: R(A), N(A), N(A T ), R(A T )

34 Four spaces of A The column space R(A) of A is a subspace of R m. The nullspace N(A) of A is a subspace of R n. In addition, we consider N(A T ), a subspace of R m and R(A T ), a subspace of R n. Four spaces of A: R(A), N(A), N(A T ), R(A T ) Question Let A = [ ], Draw R(A) and N(A T ) in the same figure, and draw N(A) and R(A T ) in the another figure.

35 Four spaces of A (cont.) Four subspaces R(A) and N(A T ) are perpendicular (in R m ). N(A) and R(A T ) are perpendicular (in R n ).

36 Four spaces of A (cont.) Four subspaces R(A) and N(A T ) are perpendicular (in R m ). N(A) and R(A T ) are perpendicular (in R n ). Each subspace contains either infinitely many vectors or only the zero vector. If u is in a space, so are 10u and 100u (and most certainly 0u). We measure the dimension of a space not by the number of vector, but by the number of independent vectors. In the above example, a line has one independent vector but not two.

37 Outline 1 Matrix Singularity 2 Matrix Multiplication by Columns or Rows Rank and nullspace Column space and solutions to linear equations 3 The Four Fundamental Subspaces 4 Dimension and Basis

38 A basis for a space A full set of independent vectors is a basis for a space. Basis 1 The basis vectors are linearly independent. 2 Every vector in the space is a unique combination of those basis vectors.

39 A basis for a space A full set of independent vectors is a basis for a space. Basis 1 The basis vectors are linearly independent. 2 Every vector in the space is a unique combination of those basis vectors. Some particular bases for R n : standard basis = columns of the identity matrix general basis = columns of any invertible matrix orthogonal basis = columns of any orthogonal matrix

40 A basis for a space A full set of independent vectors is a basis for a space. Basis 1 The basis vectors are linearly independent. 2 Every vector in the space is a unique combination of those basis vectors. Some particular bases for R n : standard basis = columns of the identity matrix general basis = columns of any invertible matrix orthogonal basis = columns of any orthogonal matrix The dimension of a space is the number of vectors in a basis for the space.