IT 131 MATHEMATCS FOR SCIENCE LECTURE NOTE 6 LEAST SQUARES REGRESSION ANALYSIS and DETERMINANT OF A MATRIX Source: Larson, Edwards, Falvo (2009): Elementary Linear Algebra, Sixth Edition You will now look at a procedure that is used in statistics to develop linear models. The next example demonstrates a visual method for approximating a line of best fit for a given set of data points. Example 9: A Visual Straight-Line Approximation Determine the straight line that best fits the points. 1 Solution Plot the points, as shown in Figure 2.2. It appears that a good choice would be the line whose slope is 1 and whose -intercept is 0.5. The equation of this line is y = 0.5 + x. One way of measuring how well a function y = f(x)fits a set of points is to compute the differences between the values from the function and the actual values, as shown in Figure 2.4. By squaring these differences and summing the results, you obtain a measure of error that is called the sum of squared error. The sums of squared errors for our two linear models are shown in Table 2.1 below.
2 The sums of squared errors confirm that the second model fits the given points better than the first. Of all possible linear models for a given set of points, the model that has the best fit is defined to be the one that minimizes the sum of squared error. This model is called the least squares regression line, and the procedure for finding it is called the method of least squares. Definition of Least Squares Regression Line To find the least squares regression line for a set of points, begin by forming the system of linear equations where the right-hand term, [y i f(x i )], of each equation is thought of as the error in the approximation y i by f(x i ) of by Then write this error as So that the system of equations takes the form
3 Now, if you define Y, X, A, and E as The n linear equations may be replaces by the matrix equation Y = XA + E. Note that the matrix X has two columns, a column of 1 s (corresponding to a 0 ) and a column containing the x i s. This matrix equation can be used to determine the coefficients of the least squares regression line, as follows. Matrix Form for Linear Regression Example 10 demonstrates the use of this procedure to find the least squares regression line for the set of points from Example 9. Example 10: Finding the Least Squares Regression Line Find the least squares regression line for the points (1,1)(2,2)(3,4)(4,4), and (5,6) (see Figure 2.5). Then find the sum of squared error for this regression line.
4 Solution Using the five points below, the matrices X and Y are This means that And Now, using (X T X) 1 to find the coefficient matrix A, you have The least squares regression line is (See Figure 2.5.) The sum of squared error for this line can be shown to be 0.8, which means that this line fits the data better than either of the two experimental linear models determined earlier.
5
6
DETERMINANTS 3.1: The Determinant of a Matrix Every square matrix can be associated with a real number called its determinant. Definition of the Determinant of a 2 x 2 Matrix 7 A convenient method for remembering the formula for the determinant of a 2 2matrix is shown in the diagram below. The determinant is the difference of the products of the two diagonals of the matrix. Note that the order is important, as demonstrated above. Example 1: The Determinant of a Matrix of Order 2 Find the determinant of a Matrix of Order 2 Solution R E M A R K : The determinant of a matrix can be positive, zero, or negative. The determinant of a matrix of order 1 is defined simply as the entry of the matrix. For instance, if A = [ 2]then det(a) = 2. To define the determinant of a matrix of order higher than 2, it is convenient to use the notions of minors and cofactors. Definitions of Minors and Cofactors of a Matrix
For example, if A is a 3 3 matrix, then the minors and cofactors of a 21 and are as shown in the diagram below. 8 As you can see, the minors and cofactors of a matrix can differ only in sign. To obtain the cofactors of a matrix, first find the minors and then apply the checkerboard pattern of + s and s shown below. Note that odd positions (where i + j is odd) have negative signs, and even positions (where i + j is even) have positive signs. Example 2: Find the Minors and Cofactors of a Matrix Find all the minors and cofactors of Solution To find the minor M 11, delete the first row and first column of A and evaluate the determinant of the resulting matrix.
Similarly, to find M 12, delete the first row and second column. 9 Continuing this pattern, you obtain Now to find the cofactors, combine the checkerboard pattern of signs with these minors to obtain Now that the minors and cofactors of a matrix have been defined, you are ready for a general definition of the determinant of a matrix. The next definition is called inductive because it uses determinants of matrices of order n 1 to define the determinant of a matrix of order n. Definition of the Determinant of a Matrix When you use this definition to evaluate a determinant, you are expanding by cofactors in the first row. This procedure is demonstrated in Example 3. Example 3: The Determinant of a Matrix of Order 3 Find the determinant of Solution This matrix is the same as the one in Example 2. There you found the cofactors of the entries in the first row to be By the definition of a determinant, you have
Although the determinant is defined as an expansion by the cofactors in the first row, it can be shown that the determinant can be evaluated by expanding by any row or column. For instance, you could expand the 3 3 matrix in Example 3 by the second row to obtain 10 Or by the first column to obtain Try other possibilities to confirm that the determinant of can be evaluated by expanding by any row or column. Theorem 3.1: Expansion by Cofactors When expanding by cofactors, you do not need to evaluate the cofactors of zero entries, because a zero entry times its cofactor is zero. The row (or column) containing the most zeros is usually the best choice for expansion by cofactors. This is demonstrated in the next example. Example 4: The Determinant of a Matrix of Order 4 Find the determinant of Solution By inspecting this matrix, you can see that three of the entries in the third column are zeros. You can eliminate some of the work in the expansion by using the third column. Because C 23, C 33, and C 43 have zero coefficients, you need only find the cofactor C 13 To do this, delete the first row and third column of A and evaluate the determinant of the resulting matrix.
11 Expanding by cofactors in the second row yields There is an alternative method commonly used for evaluating the determinant of a 3 x 3 matrix A. To apply this method, copy the first and second columns of A to form fourth and fifth columns. The determinant of A is then obtained by adding (or subtracting) the products of the six diagonals, as shown in the following diagram. Try confirming that the determinant of A is Example 5: The Determinant of a Matrix of Order 3 Find the determinant of Solution Begin by recopying the first two columns and then computing the six diagonal products as follows. Now, by adding the lower three products and subtracting the upper three products, you can find the determinant of A to be
12
13
14