Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14
Introduction The solution of linear systems is one of the most basic aspect of computational science. The solution techniques devised have to be as effi cient as possible as the systems used with Big Data will be extremely large. In these slides we will review some standard direct techniques used to solve linear systems of equations. More specifically, we look at solving a system of n equations in n unknowns which can be written in matrix equation as Ax = b where A = (a ij ) is an n n matrix and b = (b i ) and x = (x i ) are two n 1 vectors. To solve such a system with MATLAB, one would use x = A\b. We will explain what MATLAB actually does when it solves such a system. But first, we review some of the standard techniques used to solve such a system. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 2 / 14
Triangular matrices First, we look at special matrices which play an important role in solving Ax = b. They are called triangular matrices. What is a triangular matrix? See: Upper or lower triangular Strictly upper or strictly lower triangular Upper or lower unitriangular If A is a triangular matrix, then solving this system is very easy. 1 If A is lower triangular, we use forward substitution. 2 If A is upper triangular, we use backward substitution. Both techniques are O ( n 2) meaning that the number of operations required to solve the system has the magnitude of n 2. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 3 / 14
Gaussian Elimination In the most general case, we can solve this system using Gaussian elimination. We proceed as follows: 1 We form the augmented matrix denoted [A : b] which consists of the matrix A and the column vector b. 2 Using elementary row operations, we transform this augmented matrix into a matrix which is upper triangular. The elementary row operations are: 1 Replace row i by itself plus a multiple of row j, denoted E i + λe j E i 2 Replace row i by a multiple of row i, denoted λe i E i 3 Switch rows i and j, denoted E i E j 3 We finish solving the system using backward substitution. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 4 / 14
Direct Methods: Gaussian Elimination The number of operations required is O ( n 3), which is prohibitive for large matrices. This procedure can be carried out if A is nonsingular, that is if the determinant of A is not 0 or det (A) 0. Even if A is singular, it is possible to encounter a zero pivot. In this case, two rows must be switched. Often, a system with the same matrix A but different matrix b has to be solved. We look at ways to improve Gaussian elimination. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 5 / 14
LU Decomposition The LU decomposition of a matrix A writes A as A = LU where L is lower triangular and U is upper triangular. The command to do this in MATLAB is [L U] = lu(a). The matrix L obtained will have one s on its diagonal. Example Try the lu command with A = A = 1 2 3 4 5 6 10 21 47 4 3 1 2 4 5 1 2 6. What do you notice? then with Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 6 / 14
LU Decomposition The algorithm to find the LU factorization involves Gaussian elimination. It is possible to encounter a zero pivot. In which case, rows of A have to be switched. In this case, the lu command can return a 3rd argument which is a permutation matrix recording all the row switching that were necessary for the algorithm to complete. In other words, if [L U P] = lu(a) then PA = LU. This has important consequences when solving Ax = b (see questions at the end of the slides). Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 7 / 14
LU Decomposition To solve Ax = b (assuming no permutation was needed for the decomposition), that is LUx = b, one writes Ux = y hence, one has to solve: Ly = b which is O ( n 2) Ux = y which is also O ( n 2). Of course, the LU factorization is O ( n 3). But if Ax = b has to be solved many time for the same A and different b s, then the LU decomposition only needs to be done once. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 8 / 14
Cholesky Decomposition Definition An n n matrix A is symmetric if it is equal to its transpose that is A = (a ij ) is symmetric if a ij = a ji for every i, j between 1 and n. Definition An n n symmetric matrix A is positive definite if x T Ax > 0, for every vector x in R n where x T is the transpose of x. In MATLAB, to see if a matrix is symmetric we can define the function issym=@(x) isequal(x,x. ). This function will return 1 if the matrix is symmetric, 0 otherwise. The MATLAB function isequal checks if two arrays are equal. To see if a matrix is positive definite, check if all the eigenvalues of the matrix are positive (see next slide). In MATLAB, this can be done with posdef=@(x) all(eig(x)>0). The MATLAB function all determined if all array elements are nonzero or true. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 9 / 14
Cholesky Decomposition Theorem If an n n symmetric matrix A is positive definite then the following is true: 1 Every square submatrix of A is also positive definite. 2 All the eigenvalues of A are positive. Conversely, if all the eigenvalues of A are positive, then A is positive definite. 3 There exists a unique decomposition of A = LL T where L is lower triangular. 4 There exists a unique decomposition of A = U T U where U is upper triangular. 5 For any real nonsingular matrix, the product A T A is a positive definite matrix. Definition The decomposition A = LL T is called Cholesky decomposition. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 10 / 14
Cholesky Decomposition IN MATLAB, the function to perform Cholesky decomposition is chol. Given A, U = chol(a) will return the upper triangular matrix U such that A = U T U. Given A, L = chol(a, lower ) will return the lower triangular matrix L such that A = LL T. Example Consider the matrix A = 2 1 1 1 2 1 1 1 4 1 Using MATLAB, verify the matrix is symmetric. 2 Using MATLAB, verify the matrix is positive definite. 3 Find an upper triangular matrix U such that A = U T U. 4 Find a lower triangular matrix L such that A = LL T. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 11 / 14
Cholesky Decomposition Given a positive definite matrix A = U T U, one can solve the system Ax = b in two steps as follows: 1 Let Ux = y and solve U T y = b (how?) 2 Solve Ux = y (how?) Each system takes O ( n 2) operations, and Cholesky decomposition, when possible, is done in half the time it takes to do an LU decomposition. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 12 / 14
Conclusion There are other matrix decompositions such as QR decomposition, SVD decomposition. We will study the latter one. Which decomposition to use to solve Ax = b depends on the matrix A. The solution for Ax = b in MATLAB is given by x = A\b. When using \, MATLAB attempts to use the most effi cient algorithm. Here is an outline of what MATLAB does: 1 See if A is triangular then use backward or forward substitution. 2 See if A is positive definite, then use Cholesky factorization. 3 If the above fails, perform LU decomposition, then forward and backward substitution. Note that x = A 1 b is never used, unless we already have A 1 as it is the slowest technique. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 13 / 14
Assignment See the problems at the end of my notes on Direct Solution Methods for Solving Linear Systems. Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 14 / 14