Linear Algebra Tutorial for Math3315/CSE3365 Daniel R. Reynolds These notes are meant to provide a brief introduction to the topics from Linear Algebra that will be useful in Math3315/CSE3365, Introduction to Scientific Computing. They are meant to be complete enough for students who have not taken a course in Linear Algebra, yet short enough to provide a refresher for those students who took Linear Algebra but do not remember everything from the course. 1 Systems of Linear Equations A linear equation in the variables x 1, x,..., x n is an equation that can be written in the form a 1 x 1 + a x +... + a n x n = b, where n is any positive integer, and b and {a i } n i=1 are coefficients that may be either real or complex numbers. We will restrict our attention to the case of real numbers (e.g., π, 1/3). Example 1. Both x 1 x = 5 and 3x 1 = 4x 6x 1 are linear equations because they can be re-written in the appropriate form, ()x 1 + ( 1)x = 5, (3 + 6)x 1 + ( 4)x = 0. Example. Neither 3e x 1 x = 4 or cos(x 1 ) x 1/ = 7 are linear equations. A system of linear equations is a collection of multiple linear equations involving the same variables. Example 3. The equations x 1 3x + 4x 3 = 7, x 1 + x 3 = 1, form a system of linear equations in the variables x 1, x and x 3. A solution to a linear system is a set of numbers x 1, x,..., x n such that when they are inserted into the system, all of the equations are true. Some linear systems have no solution, others have a single solution, whereas others have multiple solutions. In this class, we ll primarily consider linear systems with a single (unique) solution. Matrices and Vectors A convenient notation for linear systems is to group them into a matrix. The system has the coefficient matrix solution vector x 1 x x 3 x 1 x = 4 x 1 + x x 3 = 6 3x + 4x 3 = 0 1 0 1 1 1 0 3 4, the right-hand side (RHS) vector 4 6 0, and the. These are often written in a short-hand augmented matrix format,
1 0 4 1 1 1 6 0 3 4 0. The size of a matrix A is given by a pair of numbers, m n (pronounced m by n ). The first number corresponds to the number of rows in the coefficient matrix, and the second number corresponds to the number of columns, i.e. m is the number of linear equations, and n is the number of variables. If a m n matrix A contains only real numbers, we denote it as A m n, where corresponds to the set of real numbers. If instead a m n matrix B contains some complex numbers (e.g. 3 4i), we denote it as B m n. A vector may be thought of as a matrix with one dimension of size 1. y = [ [1 4 is a row vector, and may be denoted y 1 3 or as y 3 for short. Similarly, z = is a column 6 vector and may be denoted as z 1 or as z. Naturally, the short-hand notation 3 and above may be confusing, since it loses any information on the row/column orientation of the vector. Given a matrix A m n, we typically refer to the entries of the matrix using a -index subscript, where the first index corresponds to the [ row and the second to the column in the matrix 1 3 where the entry is found. For example, if D =, then D 1,3 = 3 and D,1 = 4. Similarly, given a vector r n we refer to the vector entry using a single index, corresponding to the location in the non-trivial dimension. For example in the preceding vectors, y 3 = 4 and z 1 =. 3 Special Matrices We have some special matrices that arise in Linear Algebra and Scientific Computing. We ll define them here, and discuss their properties in subsequent sections. Definition 1. The identity matrix, denoted I n n is a square matrix (i.e. the number of rows equals the number of columns) having the entries { 1, i = j, I i,j = 0, i j. Definition. The transpose of the matrix A m n, denoted A T n m, is a matrix having the entries A T i,j = A j,i. Definition 3. The inverse of a square matrix A n n, denoted A 1 n n, is a matrix with the property that AA 1 = A 1 A = I, where we will define matrix multiplication the next section. Notes: not all matrices have an inverse, if an inverse of a matrix exists, it is unique.
4 Matrix Arithmetic Like with regular numbers, there are rules for performing arithmetic with matrices and vectors. We may multiply a matrix or a vector by a scalar, by multiplying the scalar by every matrix/vector entry, e.g. [ [ [ [ 7 8 1 3 3 6 9 4 =, 3 =. 8 1 15 18 We can also add or subtract matrices and vectors, so long as the objects being combined are the same size, e.g. 1 0 1 4 9 6 5 14 4 + 8 = 1, 0 4 3 = 4 1. 0 1 1 3 1 Addition and subtraction of objects with different shapes is undefined. Multiplication is more complicated, so we ll define this precisely. Definition 4. Given two matrices A m p and B p n, we define the matrix C m n according to the formula p C i,j = A i,k B k,j. Notes: k=1 The number of columns in A and the number of rows in B must be identical; here they are both p. C will have the same number of rows as A and columns as B. The C i,j entry may be thought of as the product of the ith row of A with the jth column of B, where all the separate products are are then added together. Example 4. Example 5. [ 1 3 1 3 4 5 6 [ = = 1 4 + 5 + 3 6 = 4 + 10 + 18 = 3. 1 4 1 5 1 6 4 5 6 3 4 3 5 3 6 = 8 10 1 1 15 18. Example 6. 1 3 7 8 9 10 11 1 = 1 10 + 11 + 3 1 4 10 + 5 11 + 6 1 7 10 + 8 11 + 9 1 = 10 + + 36 40 + 55 + 7 70 + 88 + 108 = 68 167 66.
Example 7. [ 1 3 Example 8. 7 8 9 10 11 1 = [ 1 4 + 7 + 3 10 1 5 + 8 + 3 11 1 6 + 9 + 3 1 = [ 4 + 14 + 30 5 + 16 + 33 6 + 18 + 36 = [ 48 54 60. [ 1 3 7 8 9 10 11 1 = = [ 1 7 + 9 + 3 11 1 8 + 10 + 3 1 4 7 + 5 9 + 6 11 4 8 + 5 10 + 6 1 [ 7 + 18 + 33 8 + 0 + 36 8 + 45 + 66 3 + 50 + 7 5 Properties of Matrix Arithmetic = [ 58 64 139 154 In a standard Linear Algebra class, students are often required to prove the following matrix arithmetic identites. Here, we ll just state them for the record. In all of the following identities, A, B and C are appropriately-shaped matrices so that each operation is defined, and d is a scalar (i.e. a regular number) that is assumed to be nonzero. (A + B) + C = A + (B + C) (AB)C = A(BC) A + B = B + A A(B + C) = AB + AC (A + B)C = AC + BC (AB) T = B T A T (A + B) T = A T + B T (da) 1 = 1 d A 1 (AB) 1 = B 1 A 1 While all of the above identities make sense in relation to what you already know about arithmetic with scalar quantities, there are a some properties of scalar arithmetic that do not hold for matrix arithmetic. Specifically, matrix multiplication is not commutative, i.e. AB is not generally the same as BA. Other common mistakes that students make include: (A + B) 1 A 1 + B 1, which also pertains to scalar arithmetic. is undefined. Never attempt to put a matrix in the denominator of a fraction. Use A 1 instead. 1 A.
6 Solving Linear Systems (Gaussian Elimination) We may solve a linear system using a straightforward approach, called Gaussian elimination, that will always work for finding a solution to a linear system if one exists. Later in the semester, we will consider algorithms to do this process on a computer, but for now we ll just consider how to do this on paper. Gaussian elmination consists of three simple operations: (a) replace an equation/row by the sum of itself and a multiple of another equation/row, (b) swap two equations/rows, (c) scale an equation/row by a nonzero number. These operations are typically performed on the augmented matrix system (to reduce the amount of writing), and have a simple goal: convert the matrix to the identity (if possible), the resulting RHS vector is the solution. Specific techniques for achieving this goal include: proceed from the first row downward, if on row k, the entry in column k (called the diagonal) is zero, swap row k with a row below to put a nonzero number on the diagonal, when on row k, add multiples of row k to the following rows to force the entries below the diagonal to be 0, once we make it to the bottom, proceed upwards forcing the entries above the diagonal to be 0. It may be easiest to learn this through an example. Example 9. Consider the linear system x + 5x 3 = 4 x 1 + 4x 4x 3 = 4 x 1 + x 3 = 1 Converting this to augmented matrix form, we have 0 1 5 4 4 4 4 1 0 1 1. We start with row 1. Noticing that the (1, 1) entry is zero, we must swap rows. We could swap row 1 with either 4 4 4 row or 3; let s swap with row, 0 1 5 4. Before we eliminate values below 1 0 1 1 the current diagonal (the (1, 1) entry) let s scale the first row by 1 to simplify our calculations, 1 0 1 5 4 1 0 1 1. We can now replace row 3 with the sum of row 3 and row 1, to achieve the result that all entries below the (1, 1) entry are zero, 1 0 1 5 4 0 1 3. Having achieved our goal for row 1, we move on to row. The diagonal entry on this row is in the (, ) position, and is already nonzero (with value 1), so we do not need to swap or rescale rows. To eliminate
the below the diagonal, we can replace row 3 with the sum of row 3 and (-) times row, 1 0 1 5 4. Moving on to row 3, we have no entries below the diagonal entry, and 0 0 11 11 1 can just rescale by 1 11 to put a 1 in the (3, 3) position, 0 1 5 4. We now proceed 0 0 1 1 back upwards, elminating the entries above the diagonal. We therefore can replace row by the sum of itself and (-5) times row 3, and can replace row 1 by the sum of itself and times row 1 0 0 3, 0 1 0 1. Moving upwards to row, we eliminate the entries above the diagonal by 0 0 1 1 1 0 0 replacing row 1 with the sum of itself and (-) times row, 0 1 0 1. The matrix on 0 0 1 1 the left is now the identity, and hence the solution to our system is x = 1. 1 It is always a good idea to check your answer, which can be done easily by multiplying our original matrix by x, 0 1 5 4 4 1 0 1 1 1 = 0 1 + 5 4 4 4 + 0 + 1 = which equals the RHS to our original linear system of equations, so x is correct. During the Gaussian elminiation process, it is possible that a diagonal entry and all the entries in the column below it are 0, making it impossible to swap rows to put a nonzero on the diagonal. If that occurs, the linear system has either zero or infinitely many solutions (but not a unique solution). We will leave that scenario for a Linear Algebra course, and assume that this will not occur in our class. 4 4 1,