Numerical Linear Algebra

Transcription

1 Chapter 3 Numerical Linear Algebra We review some techniques used to solve Ax = b where A is an n n matrix, and x and b are n 1 vectors (column vectors). We then review eigenvalues and eigenvectors and look at some possible applications. 3.1 Direct Solution Methods for Solving Linear Systems Introduction The solution of linear systems is one of the most basic aspect of computational science. The solution techniques devised have to be as effi cient as possible as the systems used with Big Data will be extremely large. In this section we will review some standard direct techniques used to solve linear systems of equations. More specifically, we look at solving a system of n equations in n unknowns which can be written in matrix equation as Ax = b where A = (a ij ) is an n n matrix and b = (b i ) and x = (x i ) are two n 1 vectors also known as column vectors. To solve such a system with MATLAB, one would use x = A\b. We will explain what MATLAB actually does when it solves such a system. But first, we review some of the standard techniques used to solve such a system. We begin by looking at special matrices which play an important role in solving such a system Triangular Matrices Recall that: Definition Let A be an m n matrix. 1. A is said to be upper triangular if all its entries below the main diagonal are 0. 15

2 16 CHAPTER 3. NUMERICAL LINEAR ALGEBRA 2. A is said to be lower triangular if all its entries above the main diagonal are A is said to be triangular if it is either upper triangular or lower triangular. 4. If the entries on the main diagonal of a (upper or lower) triangular matrix are all 1, the matrix is called (upper or lower) unitriangular. 5. If the entries on the main diagonal of a (upper or lower) triangular matrix are all 0, the matrix is called strictly (upper or lower) triangular. Many operations, when defined, on upper triangular matrices preserve the shape: The sum of two upper triangular matrices is upper triangular. The product of two upper triangular matrices is upper triangular. The inverse of an invertible upper triangular matrix is upper triangular. The product of an upper triangular matrix by a constant is an upper triangular matrix. The above properties remain true if the word "upper" is replaced by the word "lower". If A is a triangular matrix, then solving this system is very easy. 1. If A is lower triangular, we use forward substitution. 2. If A is upper triangular, we use backward substitution. Both techniques are O ( n 2) meaning that the number of operations required to solve the system has the magnitude of n Gaussian Elimination In the most general case, we can solve this system using Gaussian elimination. We proceed as follows: 1. We form the augmented matrix denoted [A : b] which consists of the matrix A and the column vector b. 2. Using elementary row operations, we transform this augmented matrix into a lower triangular matrix. The elementary row operations are: (a) Replace row i by itself plus a multiple of row j, denoted E i + λe j E i (b) Replace row i by a multiple of row i, denoted λe i E i (c) Switch rows i and j, denoted E i E j

3 3.1. DIRECT SOLUTION METHODS FOR SOLVING LINEAR SYSTEMS17 3. We finish solving the system using backward substitution. Remark Important facts about Gaussian elimination: The number of operations required is O ( n 3), which is prohibitive for large matrices. This procedure can be carried out if A is nonsingular, that is if the determinant of A is not 0 or det (A) 0. Even if A is singular, it is possible to encounter a zero pivot. In this case, two rows must be switched. Often, a system with the same matrix A but different matrix b has to be solved. We look at ways to improve Gaussian elimination LU Decomposition The LU decomposition of a matrix A writes A as A = LU where L is lower triangular and U is upper triangular. The command to do this in MATLAB is [L U] = lu(a). The matrix L obtained will have one s on its diagonal. Example Try the lu command with A = A = If A = and U =. What do you notice? then [L U] = lu(a) results in L = then with which is as expected and one can verify that LU = A If A = then [L U] = lu(a) results in L = and U = which is not as expected

4 18 CHAPTER 3. NUMERICAL LINEAR ALGEBRA The second example illustrates the fact that the algorithm to find the LU factorization involves Gaussian elimination. It is possible to encounter a zero pivot. In which case, rows of A have to be switched. In this case, the lu command can return a third argument which is a permutation matrix recording all the row switching that were necessary for the algorithm to complete. In other words, if [L U P ] = lu(a) then P A = LU. This has important consequences when solving Ax = b (see questions at the end of the slides). Now, we can redo the above example. Example Try the lu command with A = A = What do you notice? then with For the first matrix, since it worked, there is nothing else to do. Note that if we had typed [L U P ] = lu(a) we would have gotten the same L and U as above and P would have been the identity matrix. This is to be expected since there were no zero pivots, there is no permutation to perform. If A = U = that P A = LU. then [L U P ] = lu(a) results in L =, and P = and we can verify We can use the LU decomposition of a matrix A to solve Ax = b as follows: To solve Ax = b (assuming no permutation was needed for the decomposition), that is LUx = b, one writes Ux = y hence, one has to solve: Ly = b which is O ( n 2) Ux = y which is also O ( n 2). Of course, the LU factorization is O ( n 3). But if Ax = b has to be solved many time for the same A and different b s, then the LU decomposition only needs to be done once Cholesky Decomposition Definition An n n matrix A is symmetric if it is equal to its transpose that is A = (a ij ) is symmetric if a ij = a ji for every i, j between 1 and n.,

5 3.1. DIRECT SOLUTION METHODS FOR SOLVING LINEAR SYSTEMS19 Definition An n n symmetric matrix A is positive definite if x T Ax > 0, for every vector x in R n where x T is the transpose of x. Remark These two concepts can be checked easily in MATLAB. One can check to see if a matrix is symmetric by defining the function issym=@(x) isequal(x,x. ). This function will return 1 if the matrix is symmetric, 0 otherwise. The MATLAB functions isequal checks if two arrays are equal. One can check to see if a matrix is positive definite by checking if all the eigenvalues of the matrix are positive (see next slide). In MATLAB, this can be done with posdef=@(x) all(eig(x)>0). The MATLAB function all determined if all array elements are nonzero or true. We now look at the Cholesky decomposition and when it exists. Theorem If an n n symmetric matrix A is positive definite then the following is true: 1. Every square submatrix of A is also positive definite. 2. All the eigenvalues of A are positive. Conversely, if all the eigenvalues of A are positive, then A is positive definite. 3. There exists a unique decomposition of A = LL T where L is lower triangular. 4. There exists a unique decomposition of A = U T U where U is upper triangular. 5. For any real nonsingular matrix, the product A T A is a positive definite matrix. is called Cholesky decompo- Definition The decomposition A = LL T sition. Given a square matrix A, the following can be done with MATLAB: The function to perform Cholesky decomposition is chol. U = chol(a) will return the upper triangular matrix U such that A = U T U. Given A, L = chol(a, lower ) will return the lower triangular matrix L such that A = LL T. What is the relationship between L and U (extremely easy!)? Example Consider the matrix A =

6 20 CHAPTER 3. NUMERICAL LINEAR ALGEBRA 1. Using MATLAB, verify the matrix is symmetric. If we define the function issym as explained above, it will return 1 when its input is A. 2. Using MATLAB, verify the matrix is positive definite. If we define the function posdef as explained above, it will return 1 when its input is A. 3. Find an upper triangular matrix U such that A = U T U The command U = chol(a) returns U = Find a lower triangular matrix L such that A = LL T The command L = chol(a, lower ) returns L = Given a positive definite matrix A = U T U, one can solve the system Ax = b in two steps as follows: 1. Let Ux = y and solve U T y = b (how?) 2. Solve Ux = y (how?) Each system takes O ( n 2) operations, and Cholesky decomposition, when possible, is done in half the time it takes to do an LU decomposition Conclusion There are other matrix decompositions such as QR decomposition, SVD decomposition. We will study the latter one. Which decomposition to use to solve Ax = b depends on the matrix A. The solution for Ax = b in MATLAB is given by x = A\b. When using \, MATLAB attempts to use the most effi cient algorithm. Here is an outline of what MATLAB does: 1. See if A is triangular then use backward or forward substitution. 2. See if A is positive definite, then use Cholesky factorization. 3. If the above fails, perform LU decomposition, then forward and backward substitution. Remark Note that x = A 1 b is never used, unless we already have A 1 as it is the slowest technique.

7 3.1. DIRECT SOLUTION METHODS FOR SOLVING LINEAR SYSTEMS21 1. Research the LU algorithm and describe it thoroughly. 2. Make sure your description handles the case with zero pivots, described in the slides, that is when a permutation matrix P is also needed. 3. The slides describe how Ax = b is solved when A = LU in the case no permutation is needed. Update this description to include the case when a permutation is needed Exercises 1. Research the LU algorithm and describe it thoroughly. Make sure your description handles the case with zero pivots, described in the slides, that is when a permutation matrix P is also needed. 2. The slides describe how Ax = b is solved when A = LU in the case no permutation is needed. Update this description to include the case when a permutation is needed. 3. In the notes, we outlined how to solve Ax = b when A = U T U we left some questions unanswered (see where "how?" is written. Answer these questions. 4. Do the same if we decompose A as A = LL T. 5. We saw above that we can write A as A = U T U and also as A = LL T. What is the relationship between L and U?

8 Bibliography [1] M. C, S. M, Y. Z, V. C. L, Big data: related technologies, challenges and future prospects, Springer, [2] J. D, Big data, data mining, and machine learning: value creation for business leaders and practitioners, John Wiley & Sons, [3] L. G, What s this all about?, Time, 186 (2015), pp [4] H. A. K, Big Data: techniques and technologies in geoinformatics, CRC Press, [5] J. N. K, Data-Driven Modeling & Scientific Computation: Methods for Complex Systems and Big Data, Oxford University Press, [6] F. L R. I, Google le nouvel einstein, Science & Vie, 1138 (2012), pp [7] K.-C. L, H. J, L. T. Y, A. C, Big Data: Algorithms, Analytics, and Applications, CRC Press, [8] T. MP, Will our data drown us, IEEE Spectrum, (2015), pp [9] T. P, Giving your body a "check engine" light, IEEE Spectrum, (2015), pp [10] L. S, Should you get paid for your data, IEEE Spectrum, (2015), pp [11] E. S, Their prescription: Big data, IEEE Spectrum, (2015), pp [12] S. Q. Y, Big data analysis for bioinformatics and biomedical discoveries, CRC Press,