Linear Algebra Basics For the next chapter, understanding matrices and how to do computations with them will be crucial. So, a good first place to start is perhaps What is a matrix? A matrix A is an array of real numbers (they can be complex, but real will be enough for the purposes of this course). We say that A is an m n matrix, meaning that the matrix A consists of m rows and n columns; m and n are often referred to as the dimensions of the matrix. If m n, then the matrix is called square. The operations that we will discuss first apply to any arbitrary m n matrices, but we will eventually switch and consider only n n (square) matrices. A matrix is also often written as A pa ij q, where a ij denotes the number in the ith row and jth column. If a matrix is square, we refer to the entries starting in the top left corner moving straight down to the bottom right corner as the diagonal or diagonal entries. We can think of a vector with n components as an n matrix, which is often useful. Examples: 7 4 5 4 5 4 5 4 Now that we know what a matrix is, what can we do with them? How do basic mathematical operations work? Addition Given two matrices A and B with the same dimensions, i.e. both are m n, then the sum A B can be found by simply adding the corresponding components. This is why they need to have the same dimensions! Formally, if A pa ij q and B pb ij q, then A B pa ij b ij q. Example: If A and B 4, then A B 6 p q 4 6 We often denote by the zero matrix. is the zero matrix, but it can be defined for any m n. The main advantage of the zero matrix is the following property: A A A. Scalar Multiplication We can multiply any m n matrix A by a scalar (constant) c, which is defined by ca cpa ij q pca ij q. This works regardless of the dimensions of the matrix. In particular, p qa A is called the additive inverse of A since A p Aq p Aq A. Subtraction Using the definitions of addition and scalar multiplication, subtraction now makes sense. We can find A B by taking A p Bq, assuming that A and B have the same dimensions of course. Example: If A, B 4, then A B 6 4 6 4 6 8 8 9
Matrix Multiplication Matrix multiplication can be pretty confusing at first. Before discussing how to multiply matrices, it is important to know when you can even multiply them. If A is m n and B is n r, then their product AB is defined and will be m r. Notice that the inner dimensions need to match for multiplication to make sense! Why will become more clear when we see how to do it. Matrix multiplication is done by taking each row of A times each column of B (remember: row times column, row times column). Times is used in the sense of the dot product of two vectors, where the vectors are all of the rows of A and columns of B. Writing the definition formally is difficult, so here is an example to illustrate the idea: If A, B 4 ñ ñ, then AB 6 4 6 4 6 However, we soon encounter an interesting problem! BA 6 4 4-6 6 4 4 6 4 6 6 4 4 7 4 7 Notice that the products AB and BA are not equal! In fact, for any arbitrary m n matrix A and n r matrix B, it is generally true that AB BA, though it is possible for certain matrices. A fancy way to say this is that matrix multiplication is not commutative, unlike multiplication of numbers. Moreover, even if the product AB makes sense, BA may not, since we would need r m for that to be true. The Identity Matrix One of the most useful matrices is the identity matrix, often denoted by I or I n, which is the n n matrix where all of the entries on the diagonal are and all others are. Important: all identity matrices are square! I, I, etc. The identity matrix has several nice properties, among them the fact that for any n n matrix A, we have AI n I n A A. Determinant and Trace a b Let s assume that we have an arbitrary matrix A. We define the determinant and trace c d of A as follows: detpaq ad bc TrpAq a d Note that the determinant is the product down the diagonal minus the product down the cross diagonal and that the trace is simply the sum down the diagonal. Trace is defined the same way for any n n matrix, but computing the determinant changes for matrices larger than. The interpretation of these numbers will not be important in this course, but they are helpful quantities to know and recognize.
Systems of Equations So, where are matrices useful? The first place many people encounter a matrix is as an alternate solution method for systems of equations. Let s first recall what a system of equations in two variables looks like. Here s an example: x y x y 9 If you think back to your algebra classes, there were two main methods of solution: substitution and elimination. It takes a very particular system for substitution to be useful, so let s focus on elimination. How does it work? The idea is to multiply each equation by some constant in order to add them together and eliminate a variable. We ll see shortly that these ideas translate to matrices as well. We can use matrices to rewrite the above system in the following form, bearing in mind that two matrices are equal only if all of their entries are equal: x y Note that the entries in the first matrix are the coefficients of x and y, respectively, and that the matrix on the right side consists of the numbers found on the right side of each equation. Our next step is to form an augmented matrix and use row reduction techniques to find a solution. 9 This is the augmented matrix for the given system. The vertical bar is intended to make sure that we keep track of the coefficients of the variables on the left and the numbers from the right. Notice that you don t see the variables x or y, but you should keep in mind that numbers in the first column correspond to x and the second column to y. Row Reduction The ideas and steps here are a simplification of the method of elimination. When using elimination, you can multiply any equation by a nonzero constant, switch the order of the equations, and add equations together without changing the equations or the solution(s). The same is true for the method of row reduction. The allowable operations are as follows: 9 ) You can multiply any row by a nonzero constant. ) You can swap any two rows. ) You can add two rows together, replacing one with their sum. The goal is to *ideally* arrive at the identity matrix on the left side of the bar, but this isn t always possible. A general process that works well though is to begin by making the top left entry into a, whether that takes a row swap, multiplying by a constant, or both. Then, we use that to create a in the space below it by adding the correct multiple of the first row to the second row. We continue this pattern, working down the diagonal to make all diagonal entries and all entries below them. We then use the bottom right to work back up and achieve only a above the as well, moving to the left as much as possible. This process terminates when we either arrive at the identity matrix or can no longer zero out any remaining entries. We will do a few examples below to help illustrate the process. For simplicity in describing the row operations, we use R to represent row, to show which row is replacing another, and Ø to denote row swaps. Let s begin by row reducing the augmented matrix from before. 9
The top left entry is already a, so we begin by eliminating the in the second row. This can be done by adding - times row to row. 9 R RR ÝÝÝ 9 We now have a in the top left position and a below it, so we move on to creating another in the bottom right corner (working our way down the diagonal). We can do this by multiplying row by -/, or equivalently, dividing by -. {RR ÝÝÝ We now have a in the top left, below it, and a in the bottom right. All that is left to do to arrive at the identity matrix is to zero out the entry in the top right. We can do this by subtracting row from row. R RR ÝÝÝ p q 4 This augmented matrix is now said to be in reduced row echelon form. Again, arriving at the identity matrix on the left is the ideal goal, but it is not always possible. Now, remembering that the first column corresponds to x and the second to y, what does this say? The first row of the augmented matrix says that x y 4, or simply x 4. Therefore, the second row tells us similarly that y. Hence the solution to our original system is x 4, y. For a small example (say one involving only a matrix), this may seem inefficient, but it usually takes just as much if not less work than elimination. Also, this method is invaluable and crucial when you have a system of, say, equations in unknowns. Elimination would be incredibly tedious, and that is precisely where row reduction shines. Let s do another example. See if you can identify the row operations that were used in each step. You can write them in above the arrows. 4 5.5.5.5 7 5.5 5 Now write out the system that this augmented matrix came from, as well as the solution. There are two other possibilities for the final, row-reduced matrix. First, let s say you arrive at a matrix like the following: 7 The second row is equivalent to the equation x y, or simply, which is not a true statement. Therefore, this system is called inconsistent and has no solution. Another common row-reduced matrix is something similar to the following: 7 This matrix is nearly identical to the previous one, except that the second row now says x y, which is always a true statement for any values of x and y. Therefore, this row provides no information, and we only know that x y 7. This corresponds to an infinite number of solutions, namely any pair px, yq that lies on the line x y 7. For example, we could have x, y or x 7, y. 4
Eigenvalues and Eigenvectors Now that we have discussed some basics, we can tackle a major question that will be crucial in this course. Given an n n matrix A, is it possible to find a constant λ so that A v λ v for some vector v? For example, say we let A. Can we find such a λ if v? A v v ñ λ What if we take v instead? A v 4 λ v for any constant λ It is always possible to find such a value λ for any matrix A, but only certain vectors v will work. Such value(s) of λ are called eigenvalues of A, and the corresponding nonzero vectors v for each eigenvalue are called its eigenvectors. An arbitrary n n matrix A could have as few as eigenvalue or as many as n, and each eigenvalue will have an infinite number of corresponding eigenvectors. We require eigenvectors to be nonzero because A λ for any constant λ, so the zero vector gives no information about the matrix A. How do we find the eigenvalues and eigenvectors for a given matrix A? Why exactly this works is beyond the scope of this course, but to find the eigenvalues, we need to solve the equation detpa λiq. The quantity on the left is a polynomial in the variable λ, denoted ppλq, and it is called the characteristic polynomial for the matrix A. Therefore, we are essentially finding the roots of a polynomial to find the eigenvalues, and if A is, then this will always be a quadratic polynomial. Then, once we know the eigenvalues of A, we substitute each value independently into the matrix equation pa λiq v and solve this equation for v, which is really a system of equations. Any nonzero solution will work, and any constant multiple of an eigenvector will again be an eigenvector. Example: Find all eigenvalues and eigenvectors for the matrix A. 4 We begin by finding the characteristic polynomial. ppλq detpa λiq det λ 4 det 4 p λqp4 λq p λqp4 λq ñ λ, 4 λ λ det λ 4 λ So, the eigenvalues of the given matrix are and 4. We now find the corresponding eigenvectors for each. Let s start with λ. We want to solve the matrix equation pa Iq v ñ v ñ v, which is equivalent to row reducing the 4 augmented matrix. The matrix on the left is simply A with λ subtracted down the diagonal. λ : ñ x y ñ x y. So, we can take v for example, but any constant multiple of this vector will also be an eigenvector, so ALL eigenvectors for λ are c. 5
λ 4: ñ x, but y is a free variable and can take any value. So v easy choice, and therefore ALL eigenvectors for λ 4 are c. is an An interesting consequence to observe is that the only possible eigenvectors for the matrix A have either the form c or c. No other vectors can possibly work! So, we have used the determinant, but where is the trace of a matrix helpful? Well, here is a shortcut that you may feel free to use from this point forward: a b Given a matrix A, then its characteristic polynomial will always have the form c d ppλq λ TrpAqλ detpaq. Example: Find all eigenvalues and eigenvectors for the matrix A 4. 6 detpaq and TrpAq 6 8, so ppλq λ 8λ. Setting ppλq, we have λpλ 8q, so the eigenvalues are λ and λ 8. λ : 4 6 ñ x y ñ x y ñ v λ 8: 6 4 ñ x y ñ x y ñ v (Note: we didn t need to fully row reduce this matrix to get the eigenvector. We just needed to recognize that rows and were multiples of each other in order to zero one of them out.) Therefore, the matrix A has eigenvalues λ and λ 8 with eigenvectors c and c, respectively. Notice that 4 c 6 4 6 c p c q 4 c c p c q 6 c which was what it meant for c to be an eigenvector of A with eigenvalue. 4 What is c 6? c, 6