Basic Linear Algebra in MATLAB 9.29 Optional Lecture 2 In the last optional lecture we learned the the basic type in MATLAB is a matrix of double precision floating point numbers. You learned a number of different tools for initializing matrices and some basic functions that used them. This time, we ll make sure that we understand the basic algebraic operations that can be performed on matrices, and how we can use them to solve a set of linear equations. A Note on Notation The convention used in this lecture and in most linear algebra books is that an italics lower case letter (k) denotes a scalar, a bold lower case letter (x) denotes a vector, and a capital letter (A) denotes a matrix. Typically we name our MATLAB variables with a capital letter if they will be used as matrices, and lower case for scalars and vectors. Vector Algebra Remember that in MATLAB, a vector is simply a matrix with the size of one dimension equal to. We should distinguish between a row vector (a xn matrix) and a column vector (an nx matrix). Recall that we change a row vector x into a column vector using the transpose operator (x in MATLAB). The same trick works for changing a column vector into a row vector. We can add two vectors, x and y, together if they have the same dimensions. The resulting vector z = x + y is simply an element by element addition of the components of x and y: z i = x i + y i. From this is follows that vector addition is both commutative and associative, just like regular addition. MATLAB also allows you to add a scalar k (a x matrix) to a vector. The result of z = x + k is the element by element addition z i = k + x i. Vector multiplication can take a few different forms. First of all, if we multiply a scalar k times a vector x, the result is a vector with the same dimension as x: z = kx implies z i = kx i. There are two standard ways to multiply two vectors together: the inner product and the outer product. The inner product, sometimes called the dot product, is the result of multiplying a row vector times a column vector. The result is a scalar z = xy = i x iy i. To take the inner product of two column vectors, use z = x y. As we ll see, the orientation of the vectors matters because MATLAB treats vectors as matrices.
Unlike the inner product, the result of the outer product of two vectors is a matrix. In MATLAB, you get the outer product my multiplying a column vector times a row vector: Z = xy. The components of Z are Z ij = x i y j. To take the outer product of two column vectors, use Z = xy. Occassionally, what we really want to do is to multiply two vectors together element by element: z i = x i y i. MATLAB provides the.* command for this operation: z = x. y. To test our understanding, let s try some basic matlab commands: x = :5 y = 6:0 x+y x+5 5*x x*y x *y x.*y How would you initialize the following matrix in MATLAB using outer products? 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 3 4 5 2 Matrix Algebra The matrix operations are simply generalizations of the vector operations when the matrix has multiple rows and columns. You can think of a matrix as a set of row vectors or as a set of column vectors. Matrix addition works element by element, just like vector addition. It is defined for any two matrices of the same size. C = A + B implies that C ij = A ij + B ij. Once again, it is both commutative and associative. Scalar multiplication of matrices is also defined as it was with vectors. The result is a matrix: C = ka implies C ij = ka ij. If you multiply a matrix times a column vector, then the result is another column vector - the column of inner products: b = Ax implies b i = j A ij x j. Similarly, you can multiply a row vector times a matrix to get a row of inner products: b = xa implies b i = j A jix j. Notice that in both cases, the definitions require that the first variable must have the same number of columns as the the second variable has rows. This idea generalizes to multiplying two matrices together. For the multiplication C = AB, the matrix C is simply a collection of inner products: C ik = j A ij B jk. In this case, A must have the same number of columns as B has rows. Like ordinary multiplication, matrix multiplication is associative and distributive, but unlike ordinary multiplication, it is not commutative. In general, AB BA. Now we are in a position to better understand the matrix transpose. If B = A, then B ij = A ji. Think of this as flipping the matrix along the diagonal. This explains why 2
the transpose operator changes a row vector into a column vector and vice versa. The following identity holds for the definitions of multiplication and transpose: (AB) = B A. This help us to understand the difference between x A and Ax. Notice that for column vector x, (Ax) = x A. There are a few more matrix terms we should know. A square matrix is an nxn matrix (it has the same number of rows and columns). A diagonal matrix A has non-zero elements only along the diagonal (A ii ), and zeros everywhere else. You can initialize a diagonal matrix in MATLAB by passing a vector to the diag command. The identity matrix is a special diagonal matrix with all diagonal elements set to. You can initialize an idenitity matrix using the eye command. Try the following matlab commands: diag(:5) 3 Solving Linear Equations Let s take a step back for a moment, and try to solve the following set of linear equations: x + 3x 2 = 4 2x + 2x 2 = 9 With a little manipulation, we find that x = 4.75 and x 2 = 0.25. We could solve this set of equations because we had 2 equations and 2 unknowns. How should we solve a set of equations with 50 equations and 50 unknowns? Let s rewrite the previous expression in matrix form: [ ] [ ] [ ] 3 x 4 = 2 2 x 2 9 Notice that we could use the same form, Ax = b, for our set of 50 equations with 50 unknowns. As expected, MATLAB provides all of the tools that we need to solve this matrix formula, and it uses the idea of a matrix inverse. The inverse of a square matrix A, which is A in the textbooks and inv(a) in MATLAB, has the property that A A = AA = I. Using this, let s manipulate our previous equation: Ax = b A Ax = A b x = A b Now solve the original equations in MATLAB using inv(a) b. You should get the vector containing 4.75 and -0.25. There are a few things to remember about matrix inverses. First of all, they are only defined for square matrices. It works with the transpose and multiplication operations with the following identities: (A ) = (A ) and (AB) = B A. You should 3
be able to verify these properties on your own using the ideas we ve developed. But the most important thing to know about matrix inverses is that they don t always exist, even for square matrices. Using MATLAB, try taking the inverse of the following matrix: [ ] 2 A = 2 4 Now try inserting this A into the system of equations at the beginning of this section, and solving it using good old fashioned algebra. Why doesn t the inverse exist? 4 Quadratic Optimization We would like to solve the equation Ax = b even if A is not square. Let s seperate the problem into a few cases where the matrix A is an mxn matrix: If m < n, then we have more unknowns than equations. In general, this system will have infinitely many solutions. If m > n, then we have more equations than unknowns. In general, this system doesn t have any solution. What if we don t want to have matlab always return no solution, but we actually want the closest solution in the least squares sense? This is equivalent to minimizing the following quantity: E = ( A ij x j b i ) 2 2 i j MATLAB provides the backslash operator to accomplish the least squares fit for a matrix equation: x = A \ b. Type help slash to appreciate the power of this command. You ll see that we could have used this command to solve the square matrix equations, too. 5 Eigenmannia How does this relate to the fish data that we used in problem set? Recall that we were given a set of points (x i, y i ), and we were asked to find the coefficients a and b to fit the following linear model: y i a + bx i You can think of each point as an equation, and write the entire data set in matrix form: x y x 2 [ ] y 2 a =.. b. x m y m If you call the left matrix A and the right side b, then calling A \ b will ask MATLAB to solve for the values of a and b that minimize the least-squared error of the model. It will return exactly the same values for a and b that the polyfit command returns. 4
For this simple example, the polyfit command and the backslash command accomplished the same task. But what if you were given a set of points (x i, y i, z i ) and you were asked to fit the following linear model: z i a + bx i + cy i The matrix notation easily scales to this problem, but the polyfit function does not. 5