A Review of Linear Algebra

Transcription

1 A Review of Linear Algebra Gerald Recktenwald Portland State University Mechanical Engineering Department These slides are a supplement to the book Numerical Methods with Matlab: Implementations and Applications, by Gerald W. Recktenwald, c, Prentice-Hall, Upper Saddle River, NJ. These slides are copyright c Gerald W. Recktenwald. The PDF version of these slides may be downloaded or stored or printed only for noncommercial, educational use. The repackaging or sale of these slides in any form, without written consent of the author, is prohibited. The latest version of this PDF file, along with other supplemental material for the book, can be found at or web.cecs.pdx.edu/~gerry/nmm/. Version.99 August, page Primary Topics Notation Vectors Matrices Mathematical Properties of Vectors and Matrices Special Matrices Variable type Typographical Convention Example scalar lower case Greek σ, α, β vector lower case Roman u, v, x, y, b matrix upper case Roman A, B, C NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

2 Defining Vectors in Matlab Vector Operations Assign any expression that evaluates to a vector >> v = [ 5 ] >> w = [; ; ; 8] >> x = linspace(,,5); >> y = ::8 >> z = sin(y*pi/8); Distinquish between row and column vectors >> r = [ ]; % row vector >> s = [ ] ; % column vector >> r - s??? Error using ==> - Matrix dimensions must agree. Although r and s have the same elements, they are not the same vector. Furthermore, operations involving r and s are bound by the rules of linear algebra. Addition and Subtraction Multiplication by a scalar Transpose Linear Combinations of Vectors Inner Product Outer Product Vector Norms NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5 Vector Addition and Subtraction Multiplication by a Scalar Addition and subtraction are element-by-element operations Multiplication by a scalar involves multiplying each element in the vector by the scalar: c = a + b c i = a i + b i i =,...,n b = σa b i = σa i i =,...,n Example: d = a b d i = a i b i i =,...,n a = 5 b = 5 a + b = 5 a b = 5 Example: a = 5 b = a = 5 8 NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

3 Vector Transpose Linear Combinations () The transpose of a row vector is a column vector: u = ˆ,, then u T = 5 Likewise if v is the column vector v = 55 then v T = ˆ, 5, Combine scalar multiplication with addition Example: u v αu + βv w α u. 5 + β v. 5 = αu + βv. 5 = w. 5 u m v m αu m + βv m w m r = 5 s = 5 t =r +s = = NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9 Linear Combinations () Linear Combinations () Any one vector can be created from an infinite combination of other suitable vectors. Example:»»» w = = + w =»» [,] [,] [,-] [,] [,] [-,] [,] w = w =»»»»» Graphical interpretation: Vector tails can be moved to convenient locations Magnitude and direction of vectors is preserved 5 NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

4 Vector Inner Product () Vector Inner Product () In physics, analytical geometry, and engineering, the dot product has a geometric interpretation nx σ = x y σ = x i y i i= x y = x y cos θ The rules of linear algebra impose compatibility requirements on the inner product. The inner product of x and y requires that x be a row vector y be a column vector y ˆx x x x y y 5 = x y + x y + x y + x y y NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Vector Inner Product () Computing the Inner Product in Matlab For two n-element column vectors, u and v, the inner product is σ = u T v σ = The inner product is commutative so that (for two column vectors) u T v = v T u nx u i v i i= The * operator performs the inner product if two vectors are compatible. >> u = (:) ; % u and v are >> v = (:-:) ; % column vectors >> s = u*v??? Error using ==> * Inner matrix dimensions must agree. >> s = u *v s = >> t = v *u t = NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5

5 Vector Outer Product Computing the Outer Product in Matlab The inner product results in a scalar. The outer product creates a rank-one matrix: A = uv T a i,j = u i v j Example: Outer product of two - element column vectors u uv T = u ˆv u 5 v v v u u v u v u v u v = u v u v u v u v u v u v u v u v 5 u v u v u v u v The * operator performs the outer product if two vectors are compatible. u = (:) ; v = (:-:) ; A = u*v A = NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Vector Norms () Vector Norms () Compare magnitude of scalars with the absolute value α > β Consider two element vectors, which lie in a plane b = (,) a = (,) c = (,) a = (,) Compare magnitude of vectors with norms x > y There are several ways to compute x. In other words the size of two vectors can be compared with different norms. Use geometric lengths to represent the magnitudes of the vectors l a = p + =, l b = p + =, l c = p + = 5 We conclude that l a = l b and l a >l c or a = b and a > c NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9

6 The L Norm p-norms The notion of a geometric length for D or D vectors can be extended vectors with arbitrary numbers of elements. The result is called the Euclidian or L norm: x = `x + x x n / = nx The L norm can also be expressed in terms of the inner product x = x x = x T x i= x i! / For any integer p The L norm is sum of absolute values The L norm or max norm is x p = ` x p + x p x n p /p x = x + x x n = nx x i i= x = max ( x, x,..., x n ) = max i ( x i ) Although p can be any positive number, p =,, are most commonly used. NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Application of Norms () Application of Norms () Are two vectors (nearly) equal? Floating point comparison of two scalars with absolute value: α β <δ α where δ is a small tolerance. Comparison of two vectors with norms: y z <δ z Notice that is not equivalent to y z <δ z y z <δ. z This comparison is important in convergence tests for sequences of vectors. See Example. in the textbook. NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

7 Application of Norms () Orthogonal Vectors Creating a Unit Vector Given u =[u,u,...,u m ] T, the unit vector in the direction of u is Proof: The following are not unit vectors û = u u û = u u u = u u = u From geometric interpretation of the inner product cos θ = u v = u v cos θ u v u v = u T v u v Two vectors are orthogonal when θ = π/ or u v =. In other words u T v = if and only if u and v are orthogonal. u u NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5 Orthonormal Vectors Matrices Orthonormal vectors are unit vectors that are orthogonal. A unit vector has an L norm of one. The unit vector in the direction of u is û = u u Since u = u u it follows that u u =if u is a unit vector. Columns and Rows of a Matrix are Vectors Addition and Subtraction Multiplication by a scalar Transpose Linear Combinations of Vectors Matrix Vector Product Matrix Matrix Product Matrix Norms NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

8 Notation Matrices Consist of Row and Column Vectors The matrix A with m rows and n columns looks like: a a a n A = a a a n.. 5 a m a mn a ij = element in row i, and column j In Matlab we can define a matrix with As a collection of column vectors A = a () a () a (n) 5 As a collection of row vectors A = a () a (). a (m) 5 >> A = [... ;... ;... ] where semicolons separate lists of row elements. The a, element of the Matlab matrix A is A(,). A prime is used to designate a row vector on this and the following pages. NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9 Preview of the Row and Column View Matrix Operations Matrix and vector operations Row and column Element-by-element operations operations Addition and subtraction Multiplication by a Scalar Matrix Transpose Matrix Vector Multiplication Vector Matrix Multiplication Matrix Matrix Multiplication NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

9 Matrix Operations Addition and subtraction C = A + B or c i,j = a i,j + b i,j i =,...,m; j =,...,n or Matrix Transpose B = A T b i,j = a j,i i =,...,m; j =,...,n Multiplication by a Scalar or B = σa b i,j = σa i,j i =,...,m; j =,...,n Note: Commas in subscripts are necessary when the subscripts are assigned numerical values. For example, a, is the row, column element of matrix A, whereas a is the rd element of vector a. When variables appear in indices, such as a ij or a i,j, the comma is optional In Matlab >> A = [ ; ; ; ] A = >> B = A B = NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page The Column View gives mathematical insight The Row View easy to do by hand Matrix Vector Product The Vector View A square matrice rotates and stretches a vector Column View of Matrix Vector Product () Consider a linear combination of a set of column vectors {a (),a (),...,a (n) }. Each a (j) has m elements Let x i be a set (a vector) of scalar multipliers or Expand the (hidden) row index x x a () + x a () x n a (n) = b nx a (j) x j = b j= a a a n b a. 5 + x a x a n n. 5 = b. 5 a m a m a mn b m NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5

10 Column View of Matrix Vector Product () Form a matrix with the a (j) as columns x a () a () a (n) x. 5 = b 5 5 x n Or, writing out the elements a a a n b a a a n x b x =. 5 x n 5 a m a m a mn b m Column View of Matrix Vector Product () Thus, the matrix-vector product is a a a n b a a a n x b x =. 5 x n 5 a m a m a mn b m Save space with matrix notation Ax = b NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Column View of Matrix Vector Product () Column View of Matrix Vector Product (5) where x and b are column vectors The matrix vector product b = Ax produces a vector b from a linear combination of the columns in A. b = Ax b i = nx a ij x j j= Algorithm. initialize: b = zeros(m, ) for j =,...,n for i =,...,m b(i) =A(i, j)x(j) +b(i) end end NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9

11 Compatibility Requirement Row View of Matrix Vector Product () Inner dimensions must agree A x = b [m n] [n ] = [m ] Consider the following matrix vector product written out as a linear combination of matrix columns = This is the column view. NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Row View of Matrix Vector Product () Row View of Matrix Vector Product () Now, group the multiplication and addition operations by row: (5)() + ()() + ()( ) + ( )( ) = ( )() + ()() + ( )( ) + ()( ) 5 = ()() + ()() + ()( ) + ()( ) 5 Product of a matrix, A, witha vector, x, looks like a () x a () a x () x 5 = x b a () x 5 = b 5 5 x a () x b a () where a (), a (), and a (),aretherow vectors constituting the A matrix. Final result is identical to that obtained with the column view. The matrix vector product b = Ax produces elements in b by forming inner products of the rows of A with x. NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

12 Row View of Matrix Vector Product () Vector View of Matrix Vector Product If A is square, the product Ax has the effect of stretching and rotating x. i = i Pure stretching of the column vector 5 5 = 5 a' (i) x y i Pure rotation of the column vector 5 5 = 5 NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5 Vector Matrix Product Vector Matrix Product Matrix vector product Vector Matrix product = m n n m Compatibility Requirement: Inner dimensions must agree u A = v [ m] [m n] = [ n] = m m n n NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

13 Matrix Matrix Product Column View of Matrix Matrix Product Computations can be organized in six different ways We ll focus on just two Column View extension of column view of matrix vector product Row View inner product algorithm, extension of column view of matrix vector product The product AB produces a matrix C. The columns of C are linear combinations of the columns of A. AB = C c (j) = Ab (j) c (j) and b (j) are column vectors. i j = j r A b ( j ) c ( j ) The column view of the matrix matrix product AB = C is helpful because it shows the relationship between the columns of A and the columns of C. NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9 Inner Product (Row) View of Matrix Matrix Product Matrix Matrix Product Summary () The product AB produces a matrix C. Thec ij element is the inner product of row i of A and column j of B. a (i) is a row vector, b (j) is a column vector. AB = C c ij = a (i) b (j) The Matrix vector product looks like: 5 5 = 5 i r a' (i ) j b ( j ) = j i c ij c ij The vector Matrix product looks like: ˆ 5 = ˆ The inner product view of the matrix matrix product is easier to use for hand calculations. NMM: A Review of Linear Algebra page 5 NMM: A Review of Linear Algebra page 5

14 Matrix Matrix Product Summary () Matrix Matrix Product Summary () The Matrix Matrix product looks like: 5 5 = 5 Compatibility Requirement Inner dimensions must agree Also, in general A B = C [m r] [r n] = [m n] AB BA NMM: A Review of Linear Algebra page 5 NMM: A Review of Linear Algebra page 5 Matrix Norms The Frobenius norm treats a matrix like a vector: just add up the sum of squares of the matrix elements.» X m nx / A F = a ij i= j= More useful norms account for the affect that the matrix has on a vector. A = max x = Ax L or spectral norm Mathematical Properties of Vectors and Matrices Linear Independence Vector Spaces Subspaces associated with matrices Matrix Rank Matrix Determinant A = max j n A = max mx a ij column sum norm i= i m j= nx a ij row sum norm NMM: A Review of Linear Algebra page 5 NMM: A Review of Linear Algebra page 55

15 Linear Independence () Two vectors lying along the same line are not independent u = 5 and v = u = 5 Any two independent vectors, for example, v = 5 and w = 5 define a plane. Any other vector in this plane of v and w can be represented by Linear Independence () A set of vectors is linearly independent if it is impossible to use a linear combination of vectors in the set to create another vector in the set. Linear independence is easy to see for vectors that are orthogonal, for example, are linearly independent. 5, 5, 5 x = αv + βw x is linearly dependent on v and w because it can be formed by a linear combination of v and w. NMM: A Review of Linear Algebra page 5 NMM: A Review of Linear Algebra page 5 Linear Independence () Consider two linearly independent vectors, u and v. If a third vector, w, cannot be expressed as a linear combination of u and v, then the set {u, v, w} is linearly independent. In other words, if {u, v, w} is linearly independent then can be true only if α = β = δ =. More generally, if the only solution to αu + βv = δw α v () + α v () + + α n v (n) = () is α = α =... = α n =, then the set {v (),v (),...,v (n) } is linearly independent. Conversely, if equation () is satisfied by at least one nonzero α i,thenthe set of vectors is linearly dependent. Linear Independence () Let the set of vectors {v (),v (),...,v (n) } be organized as the columns of a matrix. Then the condition of linear independence is α v () v () v (n) α. 5 =. 5 () 5 α n The columns of the m n matrix, A, are linearly independent if and only if x = (,,...,) T is the only n element column vector that satisfies Ax =. NMM: A Review of Linear Algebra page 58 NMM: A Review of Linear Algebra page 59

16 Vector Spaces Spaces and Subspaces Spaces and Subspaces Span of a Subspace Basis of a Subspace Subspaces associated with Matrices Group vectors according to number of elements they have. Vectors from these different groups cannot be mixed. R = Space of all vectors with one element. These vectors define the points along a line. R = Space of all vectors with two elements. These vectors define the points in a plane. R n = Space of all vectors with n elements. These vectors define the points in an n- dimensional space (hyperplane). NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Subspaces Span of a Subspace The three vectors u = 5, v = 5, w = 5, lie in the same plane. The vectors have three elements each, so they belong to R, but they span a subspace of R. 5-5 [,,] T [-,,] T If w can be created by the linear combination β v () + β v () + + β n v (n) = w where β i are scalars, then w is said to be in the subspace that is spanned by {v (),v (),...,v (n) }. If the v i have m elements, then the subspace spanned by the v (i) is a subspace of R m.if n m it is possible, though not guaranteed, that the v (i) could span R m. - - y axis [,,-] T - x axis - NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

17 Basis and Dimension of a Subspace Subspaces Associated with Matrices A basis for a subspace is a set of linearly independent vectors that span the subspace. Since a basis set must be linearly independent, it also must have the smallest number of vectors necessary to span the space. (Each vector makes a unique contribution to spanning some other direction in the space.) The number of vectors in a basis set is equal to the dimension of the subspace that these vectors span. Mutually orthogonal vectors (an orthogonal set) form convenient basis sets, but basis sets need not be orthogonal. The matrix vector product y = Ax creates y from a linear combination of the columns of A The column vectors of A form a basis for the column space or range of A. NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5 Matrix Rank Matrix Rank () The rank of a matrix, A, is the number of linearly independent columns in A. rank(a) is the dimension of the column space of A. Numerical computation of rank(a) is tricky due to roundoff. Consider u = 5 v = 5 w = 5. Do these vectors span R? What if u = ε m? We can use Matlab s built-in rank function for exploratory calculations on (relatively) small matrices Example: >> A = [ ; ; e-5] % A(,) is small A =... >> rank(a) ans = NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

18 Matrix Rank () Matrix Determinant () Repeat numerical calculation of rank with smaller diagonal entry >> A(,) = eps/ % A(,) is even smaller A =... >> rank(a) ans = Even though A(,) is not identically zero, it is small enough that the matrix is numerically rank-deficient Only square matrices have determinants. The determinant of a (square) matrix is a scalar. If det(a) =,thena is singular, and A does not exist. det(i) =for any identity matrix I. det(ab) = det(a) det(b). det(a T )=det(a). Cramer s rule uses (many!) determinants to express the the solution to Ax = b. The matrix determinant has a number of useful properties: NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9 Matrix Determinant () Special Matrices det(a) is not useful for numerical computation Computation of det(a) is expensive Computation of det(a) can cause overflow For diagonal and triangular matrices, det(a) is the product of diagonal elements The built in det computes the determinant of a matrix by first factoring it into A = LU, and then computing det(a) = det(l) det(u) = `l l...l nn `u u...u nn Diagonal Matrices Tridiagonal Matrices The Identity Matrix The Matrix Inverse Symmetric Matrices Positive Definite Matrices Orthogonal Matrices Permutation Matrices NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page

19 Diagonal Matrices () Diagonal Matrices () Diagonal matrices have non-zero elements only on the main diagonal. c C = diag (c,c,...,c n )= c c n The diag function is used to either create a diagonal matrix from a vector, or and extract the diagonal entries of a matrix. >> x = [ -5 ]; >> A = diag(x) A = -5 The diag function can also be used to create a matrix with elements only on a specified super-diagonal or sub-diagonal. Doing so requires using the two-parameter form of diag: >> diag([ ],) ans = >> diag([ 5 ],-) ans = 5 NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Identity Matrices () Identity Matrices () An identity matrix is a square matrix with ones on the main diagonal. Example: The identity matrix An identity matrix is special because I = 5 AI = A and IA = A for any compatible matrix A. This is like multiplying by one in scalar arithmetic. Identity matrices can be created with the built-in eye function. >> I = eye() I = Sometimes I n is used to designate an identity matrix with n rows and n columns. For example, I = 5 NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page 5

20 Identity Matrices () Matrix Inverse () A non-square, identity-like matrix can be created with the two-parameter form of the eye function: >> J = eye(,5) J = >> K = eye(,) K = J and K are not identity matrices! Let A be a square (i.e. n n) with real elements. The inverse of A is designated A, and has the property that A A = I and AA = I The formal solution to Ax = b is x = A b. Ax = b A Ax = A b Ix = A b x = A b NMM: A Review of Linear Algebra page NMM: A Review of Linear Algebra page Matrix Inverse () Functions to Create Special Matrices Although the formal solution to Ax = b is x = A b, it is considered bad practice to evaluate x this way. The recommended procedure for solving Ax = b is Gaussian elimination (or one of its variants) with backward substitution. This procedure is described in detail in Chapter 8. Solving Ax = b by computing x = A b requires more work (more floating point operations) than Gaussian elimination. Even if the extra work does not cause a problem with execution speed, the extra computations increase the roundoff errors in the result. If A is small (say 5 5 or less) and well conditioned, the penalty for computing A b will probably not be significant. Nonetheless, Gaussian elimination is preferred. Matrix Diagonal Tridiagonal Identity Inverse Matlab function diag tridiags (NMM Toolbox) eye inv NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 9

21 Symmetric Matrices Tridiagonal Matrices If A = A T,thenA is called a symmetric matrix. Example: 5 5 Note: B = A T A is symmetric for any (real) matrix A. Example: 5. The diagonal elements need not be equal. The general form of a tridiagonal matrix is a b c a b c a b A = c n a n b n 5 c n a n NMM: A Review of Linear Algebra page 8 NMM: A Review of Linear Algebra page 8 To Do Add slides on: Tridiagonal Matrices Positive Definite Matrices Orthogonal Matrices Permutation Matrices NMM: A Review of Linear Algebra page 8