VII Selected Topics. 28 Matrix Operations

Transcription

1 VII Selected Topics Matrix Operations Linear Programming Number Theoretic Algorithms Polynomials and the FFT Approximation Algorithms 28 Matrix Operations We focus on how to multiply matrices and solve sets of simultaneous linear equations First we show how to solve a set of linear equations using LUP decompositions Then, we explore the close relationship between multiplying and inverting matrices Finally, we discus the class of symmetric positive-denite matrices and show how we can use them to nd a least-squares solution to an overdetermined set of linear equations MAT AA+DS, Fall Nov

2 Matrix inverses and ranks If a matrix has an inverse, it is called invertible, or nonsingular The vectors,, are linearly dependent if there exist coefcients,,, not all of which are zero, such that =0 E.g., the row vectors = (13), = (24), and = (4119) are linearly dependent since + 3 =0 The column rank of a nonzero matrix is the size of the largest set of linearly independent columns of MAT AA+DS, Fall Nov For any matrix its row rank = its column rank, so that we can simply refer to the rank of Square matrix has full rank if its rank is An matrix has full column rank if its rank is An matrix is positive-denite if >0for all -vectors 0 E.g., the identity matrix is positive-denite, since for any nonzero vector =, = >0 MAT AA+DS, Fall Nov

3 Matrices that arise in applications are often positive-denite due to the following theorem Theorem D.6 For any matrix with full column rank, the matrix is positive-denite. MAT AA+DS, Fall Nov Solving systems of linear equations Numerous applications need to solve sets of simultaneous linear equations We can formulate a linear system as a matrix equation in which each matrix or vector element belongs to a eld, typically the real numbers How to solve a system of linear equations using a method called LUP decomposition? We start with a set of linear equations in unknowns,, MAT AA+DS, Fall Nov

4 ,, A solution to these equations is a set of values for,, that satisfy all of the equations simultaneously We treat only the case in which there are exactly equations in unknowns MAT AA+DS, Fall Nov We can rewrite the equations as the matrixvector equation = Equivalently, letting = =,and = If is nonsingular (= invertible), it possesses an inverse, and is the solution vector MAT AA+DS, Fall Nov

5 is the unique solution to the equation Let there be two solutions, and Then and, letting denote an identity matrix, = ) = = MAT AA+DS, Fall Nov We are predominantly concerned with the case in which is nonsingular Equivalently, the rank of is equal to the number of unknowns If the number of equations is or, more generally, if the rank of is then the system is underdetermined Such a system typically has innitely many solutions, although it may have no solutions at all if the equations are inconsistent If the number of equations exceeds, the system is overdetermined, and there may not exist any solutions MAT AA+DS, Fall Nov

6 Later we address the problem of nding good approximate solutions to overdetermined systems of linear equations Returning to the problem of solving the system of equations in unknowns We could compute and then, multiply by, yielding This approach suffers in practice from numerical instability Fortunately, another approach LUP decomposition is numerically stable and has the further advantage of being faster in practice MAT AA+DS, Fall Nov Overview of LUP decomposition The idea behind LUP decomposition is to nd three matrices,, and such that is a unit lower-triangular matrix, is an upper-triangular matrix, and is a permutation matrix (has exactly one 1 in each row and column) Call matrices,, and satisfying the equation an LUP decomposition of the matrix Every nonsingular matrix has such a decomposition MAT AA+DS, Fall Nov

7 Computing an LUP decomposition for has the advantage that we can more easily solve linear systems when they are triangular, as is the case for both matrices and Once we have found an LUP decomposition for, we can solve equation by solving only triangular linear systems, as follows Multiplying both sides of by yields the equivalent equation, which, by Exercise D.1-4, amounts to permuting the original linear equations Using our decomposition, we obtain = MAT AA+DS, Fall Nov We can now solve this equation by solving two triangular linear systems Let us dene, where is the desired solution vector First, we solve the lower-triangular system for the unknown vector by a method called forward substitution Having solved for, we then solve the uppertriangular system for the unknown by back substitution MAT AA+DS, Fall Nov

8 Because the permutation matrix is invertible (Exercise D.2-3), multiplying both sides of by gives, so that Hence, the vector is our solution to : We still need to show how forward and back substitution work MAT AA+DS, Fall Nov Forward and back substitution Forward substitution can solve the lowertriangular system in time, given,, and For convenience, we represent the permutation compactly by an array [1.. ] The entry ] indicates that ] =1and =0for ] Thus, has in row and column, and has as its th element MAT AA+DS, Fall Nov

9 Since is unit lower-triangular, we can rewrite as ] ] ] ] The rst equation tells us that ] Knowing the value of, we can substitute it into the second equation, yielding ] And so forth MAT AA+DS, Fall Nov In general, we substitute,,, forward into the th equation to solve for : Having solved for, we solve for in using back substitution, which is similar to forward substitution Here, we solve the th equation rst and work backward to the rst equation Like forward substitution, this process runs in time MAT AA+DS, Fall Nov

10 Given,,, and, LUP-SOLVE solves for by combining forward and back substitution Assume that the dimension appears in the attribute and that the permutation matrix is represented by the array LUP-SOLVE ) let be a new vector of length 3. for =1to for downto 1 6. )/ 7. return MAT AA+DS, Fall Nov Consider the system of linear equations = = We wish to solve for the unknown The LUP decomposition is = = = MAT AA+DS, Fall Nov

11 Let us verify that = = = = MAT AA+DS, Fall Nov By forward substitution, we solve = for : = obtaining = by computing rst, then, and nally ] =8 ] = = 1.4 ] = = = 1.5 MAT AA+DS, Fall Nov

12 Using back substitution, we solve for : = thereby obtaining the desired answer 1.4 = by computing rst, then, and nally MAT AA+DS, Fall Nov Inverting matrices We do not generally use matrix inverses to solve systems of linear equations, rather use more stable techniques such as LUP decomposition We show how to use LUP decomposition to compute a matrix inverse Matrix multiplication and computing the inverse are equivalently hard problems (subject to technical conditions): we can use an algorithm for one to solve the other in the same asymptotic running time MAT AA+DS, Fall Nov

13 Computing a matrix inverse from an LUP decomposition Suppose that we have an LUP decomposition of a matrix :,, and such that Using LUP-SOLVE, we can solve an equation of the form in time ) Since the LUP decomposition depends on but not, we can run LUP-SOLVE on a second set of equations in additional time ) Once we have the LUP decomposition of, we can solve, in time, versions of the equation = that differ only in MAT AA+DS, Fall Nov Think of the equation which denes, the inverse of, as a set of distinct equations of the form Let denote the th column of, and recall that the unit vector is the th column of We can then solve the equation for by using the LUP decomposition for to solve each equation separately for Once we have the LUP decomposition of, we can compute from it in time Hence we can compute the inverse of a matrix in time ( ) MAT AA+DS, Fall Nov

14 Matrix multiplication and matrix inversion The speedups obtained for matrix multiplication translate to speedups for matrix inversion In fact, matrix inversion is equivalent to matrix multiplication, in the following sense If ) denotes the time to multiply two matrices, then we can invert a nonsingular matrix in time O( ) Moreover, if ) denotes the time to invert a nonsingular matrix, then we can multiply two matrices in time ( ) MAT AA+DS, Fall Nov Theorem 28.1 (Multiplication is no harder than inversion) If we can invert an matrix in time ), where and satises the regularity condition ), then we can multiply two matrices in time ).. Proof Let and be matrices whose matrix product we wish to compute. We dene the 3 matrix by 0 = 0 0 MAT AA+DS, Fall Nov

15 The inverse of is =, 0 0 and thus we can compute the product by taking the upper right submatrix of. We can construct matrix in ) time, which is )because we assume that, and we can invert in )time, by the regularity condition on ). We thus have = ( ). MAT AA+DS, Fall Nov () satises the regularity condition whenever ) for any constants >0and 0. The following proof relies on some properties of symmetric positive-denite matrices proved later. Theorem 28.2 (Inversion is no harder than multiplication) Suppose we can multiply two real matrices in time ), where and ) satises the two regularity conditions ) = ) for any in the range and 2 ) for some constant < 1. 2Then we can compute the inverse of any real nonsingular matrix in time ( ). MAT AA+DS, Fall Nov

16 28.3 Symmetric positive-denite matrices and least-squares approximation Lemma 28.3 Any positive-denite matrix is nonsingular. Proof Suppose that a matrix is singular. Then by Corollary D.3, there exists a nonzero vector such that =0. Hence, =0, and cannot be positive-denite. MAT AA+DS, Fall Nov Least-squares approximation One application of symmetric pos-def matrices arises in tting curves to given data points We are given a set of data points,,,, where we know that the are subject to measurement errors We would like to determine a function ) such that the approximation errors are small for = 1,2,, MAT AA+DS, Fall Nov

17 The form of depends on the problem at hand Here, we assume that it has the form of a linearly weighted sum, = The number of summands and the specic basis functions are chosen based on knowledge of the problem at hand A common choice is, meaning that is a polynomial of degree 1 in MAT AA+DS, Fall Nov Thus, given data points,,,,, we wish to calculate coefcients,, that minimize the approximation errors,, Choosing, we can calculate each exactly Such a high-degree ts the noise as well as the data, however, and gives poor results when used to predict for unseen values of We choose and hope that by choosing the coefcients well, we can obtain that nds the signicant patterns in the data points without paying undue attention to the noise MAT AA+DS, Fall Nov

18 Once we choose a value of, we end up with an overdetermined set of equations whose solution we wish to approximate Let = denote the matrix of values of the basis functions at the given points; that is, The desired -vector of coefcients is = MAT AA+DS, Fall Nov Then = = is the -vector of predicted values for Thus, ) ) ) is the -vector of approximation errors To minimize approximation errors, we choose to minimize the norm of the error vector MAT AA+DS, Fall Nov

19 This gives us a least-squares solution, since Because = = = we can minimize by differentiating with respect to each and then setting the result to 0: MAT AA+DS, Fall Nov = 2 =0 The equations for = 1,2,, are equivalent to the single matrix equation =0 or, equivalently (using Exercise D.1-2), to =0 which implies In statistics, this is called the normal equation MAT AA+DS, Fall Nov

20 The matrix is symmetric by Exercise D.1-2, and if has full column rank, then by Theorem D.6, is positive-denite as well Hence, exists, and the solution to the previous equation is = where the matrix = is the pseudoinverse of the matrix The pseudoinverse naturally generalizes the notion of a matrix inverse to the case in which is not square MAT AA+DS, Fall Nov Equation is the approximate solution to Whereas the solution is the exact solution to = MAT AA+DS, Fall Nov

21 As an example of producing a least-squares t, suppose that we have ve data points = (1,2) = (1,1) = (2,1) = (3,0) = (5,3) shown as black dots in the next figure We wish to t these points with a quadratic polynomial = + + MAT AA+DS, Fall Nov MAT AA+DS, Fall Nov

22 We start with the matrix of basis-function values = 1 = whose pseudoinverse is MAT AA+DS, Fall Nov Multiplying by, we obtain the coefcient vector = , which corresponds to the quadratic polynomial = as the closest-tting quadratic to the given data, in a least-squares sense MAT AA+DS, Fall Nov