Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39
Numerical Linear Algebra Linear systems of equations occur in almost every area of the applied science, engineering, and mathematics Hence, numerical linear algebra is one of the pillars of computational mathematics Grady B Wright Linear Algebra Basics February 2, 2015 2 / 39
Linear systems of equations A linear system of m equations and n unknowns can be expressed in the following general form: a 11 x 1 + a 12 x 2 + a 13 x 3 + + a 1n x n = b 1, a 21 x 1 + a 22 x 2 + a 23 x 3 + + a 2n x n = b 2, a 31 x 1 + a 32 x 2 + a 33 x 3 + + a 3n x n = b 3, a m1 x 1 + a m2 x 2 + a m3 x 3 + + a mn x n = b m (1) Here a ij are the coefficients of the systems, b i are the right hand sides (RHS), and x j are the unknown values that must be determined a ij and b i will be given by the problem Grady B Wright Linear Algebra Basics February 2, 2015 3 / 39
Linear systems of equations Linear systems can be classified into the following three types: 1 Square linear system: If the number of equations equals the number of unknowns (ie m = n) 2 Overdetermined system: If the number of equations is greater than the number of unknowns (ie m > n) 3 Underdetermined system: If the number equations is less than the number of unknowns (ie m < n) Grady B Wright Linear Algebra Basics February 2, 2015 4 / 39
Matrices and vectors A convenient notation to describe a linear system of equations is in terms of matrices and vectors Grady B Wright Linear Algebra Basics February 2, 2015 5 / 39
Matrices A matrix is just a table of numbers containing m rows and n columns and can be expressed as: a 11 a 12 a 13 a 1n a 21 a 22 a 23 a 2n A = a 31 a 32 a 33 a 3n a m1 a m2 a m3 a mn We typically use capital letters to denote matrices We write A R m n to denote a matrix with m rows and n columns A common shorthand notation for a matrix is A = { a ij }, where the values for i and j are understood from the problem Grady B Wright Linear Algebra Basics February 2, 2015 6 / 39
Matrices A matrix is just a table of numbers containing m rows and n columns and can be expressed as: a 11 a 12 a 13 a 1n a 21 a 22 a 23 a 2n A = a 31 a 32 a 33 a 3n a m1 a m2 a m3 a mn We typically use capital letters to denote matrices We write A R m n to denote a matrix with m rows and n columns A common shorthand notation for a matrix is A = { a ij }, where the values for i and j are understood from the problem Grady B Wright Linear Algebra Basics February 2, 2015 6 / 39
Matrices A matrix is just a table of numbers containing m rows and n columns and can be expressed as: a 11 a 12 a 13 a 1n a 21 a 22 a 23 a 2n A = a 31 a 32 a 33 a 3n a m1 a m2 a m3 a mn We typically use capital letters to denote matrices We write A R m n to denote a matrix with m rows and n columns A common shorthand notation for a matrix is A = { a ij }, where the values for i and j are understood from the problem Grady B Wright Linear Algebra Basics February 2, 2015 6 / 39
Matrices A matrix is just a table of numbers containing m rows and n columns and can be expressed as: a 11 a 12 a 13 a 1n a 21 a 22 a 23 a 2n A = a 31 a 32 a 33 a 3n a m1 a m2 a m3 a mn We typically use capital letters to denote matrices We write A R m n to denote a matrix with m rows and n columns A common shorthand notation for a matrix is A = { a ij }, where the values for i and j are understood from the problem Grady B Wright Linear Algebra Basics February 2, 2015 6 / 39
Vectors If the matrix only has one row or column then it is called a vector A column vector with n entries can be expressed as x 1 x 2 x = x 3 x n A row vector and can be expressed as x = [ x 1 x 2 x 3 x n ] We typically use bold lower-case letters to denote vectors A column vector with n real entries is denoted by x R n, while a row vector is denoted by x R 1 n Grady B Wright Linear Algebra Basics February 2, 2015 7 / 39
Vectors If the matrix only has one row or column then it is called a vector A column vector with n entries can be expressed as x 1 x 2 x = x 3 x n A row vector and can be expressed as x = [ x 1 x 2 x 3 x n ] We typically use bold lower-case letters to denote vectors A column vector with n real entries is denoted by x R n, while a row vector is denoted by x R 1 n Grady B Wright Linear Algebra Basics February 2, 2015 7 / 39
Vectors If the matrix only has one row or column then it is called a vector A column vector with n entries can be expressed as x 1 x 2 x = x 3 x n A row vector and can be expressed as x = [ x 1 x 2 x 3 x n ] We typically use bold lower-case letters to denote vectors A column vector with n real entries is denoted by x R n, while a row vector is denoted by x R 1 n Grady B Wright Linear Algebra Basics February 2, 2015 7 / 39
Vectors If the matrix only has one row or column then it is called a vector A column vector with n entries can be expressed as x 1 x 2 x = x 3 x n A row vector and can be expressed as x = [ x 1 x 2 x 3 x n ] We typically use bold lower-case letters to denote vectors A column vector with n real entries is denoted by x R n, while a row vector is denoted by x R 1 n Grady B Wright Linear Algebra Basics February 2, 2015 7 / 39
Vectors If the matrix only has one row or column then it is called a vector A column vector with n entries can be expressed as x 1 x 2 x = x 3 x n A row vector and can be expressed as x = [ x 1 x 2 x 3 x n ] We typically use bold lower-case letters to denote vectors A column vector with n real entries is denoted by x R n, while a row vector is denoted by x R 1 n Grady B Wright Linear Algebra Basics February 2, 2015 7 / 39
Matrix & vector operations Grady B Wright Linear Algebra Basics February 2, 2015 8 / 39
Matrix & vector operations Grady B Wright Linear Algebra Basics February 2, 2015 8 / 39
Matrix & vector operations: Transpose Let A R m n with entries a 11 a 12 a 13 a 1n a 21 a 22 a 23 a 2n A = a 31 a 32 a 33 a 3n, a m1 a m2 a m3 a mn then the transpose of A switches the columns of A with the rows, ie a 11 a 21 a 31 a m1 a 12 a 22 a 32 a m2 A T = a 13 a 23 a 33 a m3 a 1n a 2n a 3n a mn Note that A T R n m and that (A T ) T = A Grady B Wright Linear Algebra Basics February 2, 2015 9 / 39
Matrix & vector operations: Transpose The transpose can also be applied to vectors In this case if x is a (column) vector then x T is a row vector: x 1 x 2 if x = x 3 x n then x T = [ x 1 x 2 x 3 x n ] Similarly if x is row vector then x T is a column vector Grady B Wright Linear Algebra Basics February 2, 2015 10 / 39
Matrix & vector operations: Matrix addition Let A R m n and B R m n then the sum of A and B is given by } A + B = {a ij + b ij This is just the sum of the corresponding entries of the elements of A and B For this sum to make sense A and B must be the same size Grady B Wright Linear Algebra Basics February 2, 2015 11 / 39
Matrix & vector operations: scalar multiplication Let α be a real number and A R m n then the product of α and A is given by } αa = {α a ij Note that this is just α times each entry of A Grady B Wright Linear Algebra Basics February 2, 2015 12 / 39
Matrix & vector operations: Vector-vector products There are two types of vector-vector products that arise quite frequently These can be derived from the definition for matrix-matrix products (discussed later), but it is worth stating them separately Let x, y R n then the inner product or dot product of x and y is y 1 x T y = [ ] y 2 x 1 x 2 x n = x 1y 1 + x 2 y 2 + + x n y n = y n n x j y j Note that the inner product is a single number The inner product is sometimes denoted by x y j=1 Grady B Wright Linear Algebra Basics February 2, 2015 13 / 39
Matrix & vector operations: Vector-vector products There are two types of vector-vector products that arise quite frequently These can be derived from the definition for matrix-matrix products (discussed later), but it is worth stating them separately Let x R m and y R n then the outer product of x with y is x 1 x 1 y 1 x 1 y 2 x 1 y n xy T x 2 [ ] x 2 y 1 x 2 y 2 x 2 y n = y1 y 2 y n = x m x m y 1 x m y 2 x m y n Note that the outer product is a matrix of size m-by-m Grady B Wright Linear Algebra Basics February 2, 2015 14 / 39
Matrix & vector operations: Matrix-vector products Let A R m n and x R n then the product of A and x is given by Ax =x 1 a 11 a 21 a 31 + x 2 a 12 a 22 a 32 + x 3 a 13 a 23 a 33 + + x n a 1n a 2n a 3n a m1 a m2 a m3 a mn (2) Thus, the product Ax is a linear combination of the columns of A Grady B Wright Linear Algebra Basics February 2, 2015 15 / 39
Matrix & vector operations: Matrix-vector products Important observations regarding the matrix-vector product Ax: The only way for this product to make sense is if A has the same number of columns as x does rows Ax R m, ie the product is a column vector containing m entries If we let b = Ax then we can alternatively express the ith entry of b as b i = n a ij x j, i = 1,, m j=1 This illustrates that b i is just the inner product of the ith row of A with the vector x In general, computing Ax using the above formulas requires mn multiplications and m(n 1) additions Grady B Wright Linear Algebra Basics February 2, 2015 16 / 39
Matrix & vector operations: Matrix-matrix products Let A R m n and B R n p, and let B have columns B = b 1 b 2 b n The matrix-matrix product C = AB is given as C = Ab 1 Ab 2 Ab n This shows the kth column of the product AB is a linear combination of the columns of A with the coefficients in the linear combinations being determined by entries in the kth column of B Grady B Wright Linear Algebra Basics February 2, 2015 17 / 39
Matrix & vector operations: Matrix-matrix products Important observations regarding the matrix-matrix product AB, A R m n, B R n p : Number rows of A must equal number columns B AB R m p, ie the product is a matrix containing m rows and p columns In general, AB BA, ie the product does not commute Grady B Wright Linear Algebra Basics February 2, 2015 18 / 39
Matrix & vector operations: Matrix-matrix products Important observations regarding the matrix-matrix product AB, A R m n, B R n p : We can express each entry of C as c ik = n a ij b jk, i = 1,, m, k = 1,, p j=1 So c ik is just the inner product of the ith row of A with the kth column of B Computing AB using the above formulas requires mnp multiplications and m(n 1)p additions The transpose of the product AB satisfies: (AB) T = B T A T Grady B Wright Linear Algebra Basics February 2, 2015 19 / 39
Linear systems in matrix-vector notation Recall that we can express a linear system of equations with m equations and n unknowns as a 11 x 1 + a 12 x 2 + a 13 x 3 + + a 1n x n = b 1, a 21 x 1 + a 22 x 2 + a 23 x 3 + + a 2n x n = b 2, a 31 x 1 + a 32 x 2 + a 33 x 3 + + a 3n x n = b 3, a m1 x 1 + a m2 x 2 + a m3 x 3 + + a mn x n = b m (3) We can express this linear system in matrix-vector notation using the previous definitions Grady B Wright Linear Algebra Basics February 2, 2015 20 / 39
Linear systems in matrix-vector notation Let x R n, b R m, and A R m n, then the linear system is given as Ax = b, or a 11 a 12 a 13 a 1n x 1 b 1 a 21 a 22 a 23 a 2n x 2 b 2 a 31 a 32 a 33 a 3n x 3 = b 3 a m1 a m2 a m3 a mn x n b m }{{}}{{}}{{} A x b Grady B Wright Linear Algebra Basics February 2, 2015 21 / 39
Linear systems: solvability Recall that Ax is a linear combination of the columns of A: Ax =x 1 a 11 a 21 a 31 a m1 + x 2 a 12 a 22 a 32 a m2 + x 3 a 13 a 23 a 33 a m3 + + x n a 1n a 2n a 3n a mn Thus, the only way there will be a solution to Ax = b is if b can be written as a linear combination of the columns of A Grady B Wright Linear Algebra Basics February 2, 2015 22 / 39
Linear systems: solvability There are three possibilities for the linear system Ax = b: 1 There are an infinite number of solutions that satisfy Ax = b An infinite number of ways to linearly combine the columns of A to equal b 2 There is one unique solution to the linear system Only one way to linearly combine the columns of A to equal b 3 There is no solution to the linear system There is no way to linearly combine the columns of A to equal b Grady B Wright Linear Algebra Basics February 2, 2015 23 / 39
Special types of matrices: Diagonal matrix A diagonal matrix is an n-by-n square matrix with zeros on in every entry except possibly the main diagonal: d 1 0 0 0 0 d 2 0 0 D = 0 0 d 3 0, 0 0 0 d n where d j, j = 1,, n are some real numbers Grady B Wright Linear Algebra Basics February 2, 2015 24 / 39
Special types of matrices: Identity matrix The identity matrix is a diagonal matrix with every diagonal entry equal to 1: 1 0 0 0 0 1 0 0 I = 0 0 1 0 0 0 0 1 It has the property that for any matrix A R n n, IA = AI = A Grady B Wright Linear Algebra Basics February 2, 2015 25 / 39
Special types of matrices: Lower triangular matrix A matrix L R m n is lower triangular if all the entries above its main diagonal are zero Square n-by-n lower triangular matrices take the form l 11 0 0 0 l 21 l 22 0 0 L = l 31 l 32 l 33 0, l n1 l n2 l n3 l nn where l i,j, i = 1,, n, j = i,, n are some real numbers Grady B Wright Linear Algebra Basics February 2, 2015 26 / 39
Special types of matrices: Upper triangular matrix A matrix U R m n is upper triangular if all the entries below its main diagonal are zero Square n-by-n upper triangular matrices take the form u 11 u 12 u 13 u 1n 0 u 22 u 23 u 2n U = 0 0 u 33 u 3n, 0 0 0 u nn where u i,j, i = 1,, n, j = i,, n are some real numbers Grady B Wright Linear Algebra Basics February 2, 2015 27 / 39
Special types of matrices: Symmetric matrix A matrix A is symmetric if A = A T Note that only square matrices can be symmetric Grady B Wright Linear Algebra Basics February 2, 2015 28 / 39
Inverse of a matrix Let A be an n-by-n square matrix (ie A R n n ) If there exists a square matrix B R n n such that BA = AB = I, where I is the n-by-n identity matrix, then B is called the inverse of A The inverse of A is denoted by A 1 If A 1 exists then A is called nonsingular, otherwise it is singular Grady B Wright Linear Algebra Basics February 2, 2015 29 / 39
Matrix inverses and linear systems If A is a square, nonsingular matrix, then the solution to the linear system Ax = b is given formally as x = A 1 b Important: When solving a linear system, one should never first compute A 1 and then compute the product A 1 b There are much better ways to solve the system (for example using Gaussian elimination when n is not too large) Grady B Wright Linear Algebra Basics February 2, 2015 30 / 39
Properties of matrix inverses Suppose A is nonsingular then the following statements are true A 1 is unique A 1 is nonsingular and its inverse is A A T is nonsingular If B R n n is nonsingular then AB is nonsingular and (AB) 1 = B 1 A 1 The linear system Ax = b has a unique solution Grady B Wright Linear Algebra Basics February 2, 2015 31 / 39
Vector norms A vector norm is a scalar quantity that reflects the size of a vector x The norm of a vector x is denoted as x There are many ways to define the size of a vector If x R n, the three most popular are one-norm : x 1 = n x k, k=1 two-norm : x 2 = n x k 2, -norm : k=1 x = max 1 k n x k Grady B Wright Linear Algebra Basics February 2, 2015 32 / 39
Vector norms However, a vector norm is defined, it must satisfy the following three properties to be called a norm: 1 x 0 and x = 0 if and only if x = 0 (ie x contains all zeros as its entries) 2 αx = α x, for any constant α 3 x + y x + y, where y R n This is called the triangle inequality Grady B Wright Linear Algebra Basics February 2, 2015 33 / 39
Unit vectors A vector x is called a unit vector if its norm is one, ie x = 1 Unit vectors will be different depending on the norm applied Below are several unit vectors in the one, two, and norms for x R 2 1 Unit vectors with respect to the one norm 1 Unit vectors with respect to the two norm 1 Unit vectors with respect to the norm 08 08 08 06 06 06 04 04 04 02 02 02 x 2 0 x 2 0 x 2 0 02 02 02 04 04 04 06 06 06 08 08 08 1 1 08 06 04 02 0 02 04 06 08 1 x 1 1 1 08 06 04 02 0 02 04 06 08 1 x 1 1 1 08 06 04 02 0 02 04 06 08 1 x 1 (a) One-norm (b) Two-norm (c) -norm Grady B Wright Linear Algebra Basics February 2, 2015 34 / 39
Matrix norms A matrix norm is a scalar quantity that reflects the size of a matrix A R m n The norm of A is denoted as A Any matrix norm must satisfy the following four properties: 1 A 0 and A = 0 if and only if A = 0 (ie A contains all zeros as its entries) 2 αa = α A, for any constant α 3 A + B A + B, where B R m n 4 AB A B, where B R n p This is called the submultiplicative inequality Grady B Wright Linear Algebra Basics February 2, 2015 35 / 39
Matrix norms Each vector norm induces a matrix norm according to the following definition: Ax p A p = max x p 0 x p where x R n and p = 1, 2, = max Ax p, x p=1 Induced norms describe how the matrix stretches unit vectors with respect to that norm Grady B Wright Linear Algebra Basics February 2, 2015 36 / 39
Induced matrix norms Two popular and easy to define induced matrix norms are One-norm : -norm : A 1 = max 1 k n A = max m a jk, j=1 1 j m k=1 n a jk The one-norm corresponds to the maximum of the one norm of every column The -norm corresponds to the maximum of the one norm of every row The two-norm of A is defined as the largest eigenvalue of the matrix A T A This is computationally expensive to compute Grady B Wright Linear Algebra Basics February 2, 2015 37 / 39
Non-induced matrix norms The most popular matrix norm that is not an induced norm is the Frobenius norm: m n A F = a jk 2 j=1 k=1 Grady B Wright Linear Algebra Basics February 2, 2015 38 / 39
Important results on matrix norms The following are some useful inequalities involving matrix norms Here A R m n : Ax A x 1 m A 1 A 2 n A 1 1 n A A 2 m A A 2 A 1 A Grady B Wright Linear Algebra Basics February 2, 2015 39 / 39