Note Set 3 Numerical Linear Algebra

Transcription

1 Note Set 3 Numerical Linear Algebra 3.1 Overview Numerical linear algebra includes a class of problems which involves multiplying vectors and matrices, solving linear equations, computing the inverse of a matrix, and computing various matrix decompositions. It is one of the areas where knowledge of appropriate numerical methods can lead to rather substantial speed improvements. 3. ypes of Matrices In numerical linear algebra, efficient algorithms will take advantage of the special structure of a matrix. In addition to the rectangular matrix, special matrix types include symmetric matrices, symmetric positive definite matrices, band-diagonal matrices, blockdiagonal matrices, lower and upper triangular matrices, lower and upper band diagonal matrices, oeplitz matrices, and sparse matrices. 1

2 Each of these matrices is stored in a different way, generally in an array in memory. A general matrix is usually stored as one row after another in memory. A symmetric matrix can be stored in either packed or unpacked storage, and we may store either the lower triangle, the upper triangle, or both. Suppose that n is the dimension of a symmetric matrix. Unpacked storage requires n locations in memory, while packed storage requires nn+ ( 1)/ units. In both cases, we typically store symmetric matrices row after row. Lower and upper triangular matrices may also be stored in both packed and unpacked form. Figures 3.1 and 3. illustrate the storage of lower, upper, and symmetric matrices in packed and unpacked storage. Bold number illustrate a location in memory that contains an element of the matrix and non-bold elements contain a location in memory which contains garbage. he advantage of packed storage is, of course, efficient use of memory. he advantage of unpacked storage is that this storage usually allows for faster computation. his occurs for a number of reasons, but one important reasons is that block versions of common algorithms can be applied only if matrices are stored in unpacked form. Band diagonal matrices are stored in a different fashion. Let k l denote the number of lower bands, let k u denote the number of upper bands, and let d = kl + ku + 1. A band diagonal matrix is stored in an nd element arrays. Each d units in the array correspond to the nonzero elements of a row of A. For example, consider the following example of a 5x5 band diagonal matrix with k l = and k u = 1. Figure 3.3 illustrates the storage of such a matrix. he advantage of storing the matrix this way is both efficient use of memory and computational speed.

3 Figure 3.1: Storage of Lower, Upper, and Symmetric Matrices (Unpacked Storage) Figure 3.: Storage of Lower, Upper, and Symmetric Matrices (Packed Storage) Figure 3.3: Storage of Band Matrix 3

4 A sparse matrix is a large matrix that contains few non-zero elements. Sparse matrices are most often stored in compressed row format. Compressed row format requires 3 arrays, which we label r, c, and v. he array v contains the values of all of the nonzero elements in the matrix. he array c contains column in which each corresponding nonzero element is located. he array r points to the location in c which contains the elements that correspond to a row. For example, consider the following sparse matrix, We could store this matrix as, r = (1,3,3,6,8) c = (1,3,1,,5,1,5) v = ( 1,,5,1,3,, ) he final type of matrix (which is often useful in time-series econometrics applications) is a oeplitz matrix. A oeplitz matrix has the form, a1 a a3 a4 a5 a5 a1 a a3 a 4 a4 a5 a1 a a 3 a3 a4 a5 a1 a a a3 a4 a5 a 1 A oeplitz matrix can be stored as, ( a1, a, a3, a4, a 5). 3.3 Elementary Linear Algebra and the BLAS 4

5 Elementary linear algebra operations include operations such as y Ax, C A B, and y 1 L x. Often, these operations can be implemented in a few lines of code. here are, however, more efficient ways to implement these types of algorithms. his differences most important in matrix-matrix computation (e.g. C A B ), but can occasionally be useful in matrix-vector computations (e.g. y Ax, y 1 L x). he Basic Linear Algebra Subroutines (BLAS) contains efficient implementations of these and many other algorithms. he BLAS are a set of FORRAN subroutines, but CBLAS ports these subroutines to C using an automatic converter called FortranC. In order to apply the CBLAS, you must be relatively familiar with pointers. Consider GEMM (which is the routine for multiplying two rectangular matrices). 1 he routine computes C αop( A) op( B) + βc, where op( A) = A or op( A) = A. int dgemm_(char *transa, char *transb, integer *m, integer * n, integer *k, doublereal *alpha, doublereal *a, integer *lda, doublereal *b, integer *ldb, doublereal *beta, doublereal *c, integer *ldc) Notice that every variable here is a pointer. he variable transa determines where op( A) = A or op( A) = A. he variables m and n are the dimensions of C. he variable k is the number of columns or A, which is also equal to the number of rows in B. he variables lda, ldb, and ldc, store the leading dimension of A, B, and C. his option allows the algorithm to refer to the following types of matrices, 1 A quick reference for the BLAS can be found at, 5

6 In this case, we have a 4 by 5 matrix, with a leading dimension of 8, starting at location 3. Notice that this allows us the call the BLAS algorithms on sub-matrices without reallocating memory. BLAS is divided in to level-1, level-, and level-3 algorithms. Level-1 algorithms perform operations on vectors (and are On ( ) ), level- algorithms operate on a vector and a matrix (and are On ( ) ), and level-3 algorithms operate on two matrices (and are 3 On ( )). 3.4 Matrix Decompositions Any matrix A can be composed into the product of an upper and unit lower triangular matrix, A = LU. his decomposition has a number of advantages. First, we can solve the linear system, Lx = a and Uy = b in On ( ) operations using a relatively simple algorithm. Second, we can compute the inverse A = U L (in those cases where A in invertible). hird, we can compute the condition number and the determinant of the matrix. First, let us consider the problem of computing the LU decomposition. Notice that, A Unit Lower riangular matrix is a lower triangular matrix that has ones along the diagonal. 6

7 U11 U1 U13 U14 L U U U L L U U L41 L4 L U 44 U11 U1 U13 U14 LU LU + U LU + U LU + U = LU LU LU 3 LU LU U33 LU LU U34 LU LU + LU LU + LU + LU LU + LU + LU + U We have, U U U U A A A A A A A A = A31 A3 A33 A A A A A L = A / U, L = A / U, = A = A = A = A U = A L U U = A L U U = A L U,,, L = ( A L U )/ U, U = A L U L U U = A L U L U L = A / U, L = ( A L U )/ U, L,,,, = ( A L U L U )/ U, U = A L U L U L U Clearly, we have a relatively simple algorithm for computing the LU decomposition, 7

8 for(int i=0; i<n; i++) { for(int j=0; j<n-i+1; j++) { U(i,j) = A(i,j); for(int k=0; k<i-1; k++) U(i,j) -= L(i,k) * U(k,j); } for(int k=i; k<n; k++) { L(i,j) = A(i,j); for(int k=0; k<i-1; k++) L(i,j) -= L(i,k) * U(k,i); L(i,j) = L(i,j) / A(j,j); } } We can rewrite this algorithm in a way that factorizes A in place. We have, for(int i=0; i<n; i++) { for(int j=0; j<n; j++) { for(int k=0; k<i-1; k++) A(i,j) -= A(i,k) * A(k,j); } for(int k=i; k<n; k++) A(i,j) = A(i,j) / A(j,j); } Notice that this algorithm is 3 On ( ). In general, this algorithm is not numerically stable. o make this algorithm more effective, we can employ pivoting. We consider the decomposition, A= PLU, where P is a permutation matrix. Special types of matrices admit special types of decompositions. Suppose that A is symmetric and positive definite. hen there exists a lower triangular matrix L such that A = LL. he algorithm for computing L is called the Cholesky decomposition. he Cholesky Decomposition is stable without pivoting. he Cholesky decomposition algorithm is quite easy to derive as well. Let, 8

9 A A11 A1... An 1 A A... A, An1 An... Ann 1 n = L L L L Ln 1 Ln... Ln3 1 = Multiplying L with L yields the following system of equations, L L11 L1... Ln 1 L 11 L1 L L... L n L1L11 L1 + L = L L... L L n1 n n3 n3 his indicates that we can follow the following process to determine L ij. L = A L = 1 A1 L11 L = A L 1 L = 31 A31 L11 L 3 = A3 L11 A31 L L = A L L and so on. Once we have determined the Cholesky factor L or a matrix A, we can solve the linear system Ax hen Ly solving = b in On ( ) operations. We can write Ax = LL x = b. Define = b. We can thus solve the system by first solving Ly = b for y and then Lx = yfor x. Each of these systems can be solved in On ( ). y = L x. Now, suppose we want to invert a general or symmetric positive definite matrix. Computing the inverse then simply requires inverting upper and lower-triangular 9

10 matrices. he inverse of as lower triangular matrix will be lower triangular itself. Let A be a lower triangular matrix and let B be its inverse. hen we have, b a b1 b... 0 a1 a = b b... b a a... a n1 n nn n1 n nn We can solve this system recursively as before. he final case is the symmetric indefinite matrix. Here, the Bunch-Kaufman algorithm provides one fast algorithm for matrix decomposition, which is implemented in LAPACK. his algorithm is hard to get a hold of, but good implementations compare in speed to the Cholesky Decomposition algorithms, are faster that the LU decomposition by about a factor of 3. Figure 3.4 reports the running time for each of these three algorithms on a number of large symmetric positive definite matrix. I tested the CLAPACK version of all three algorithms. I tested the NR C++ version of the LU and Cholesky decomposition algorithms, as well as NR s recommend Guass Jordan algorithm for computing a matrix inverse. Finally, I tested LU and Cholesky algorithms that I optimized for my machine (which probably won t be as fast as properly optimized algorithms). For both the LU and Cholesky decompositions, the un-optimized CLAPACK implementations are faster that NR C++ code by a factor of 3. My optimized code is somewhat faster than the LAPACK implementations. he LU Decompositions takes about twice as long as the BK and Cholesky Decompositions (which take about an equal amount of time). he Guass Jordan inverse is substantially slower. 10

11 able 3.4 Computation ime for the Inverse of a Symmetric Positive Definite Matrix Algorithm Implementation My Desktop (AMD) My Laptop (Intel) n=500 n=1,000 n=500 n=1,000 Guass Jordan Elimination NR C LU Decomposition NR C CLAPACK My Optimizations BK Decomposition LAPACK Cholesky Decomposition NR C LAPACK MY Optimizations All three matrix decomposition algorithms can be further specialized. For example, we can develop versions of the LU, Bunch-Kaufman, and Cholesky decompositions for band diagonal matrices that requires Onk ( ) operations (here, k is the number of band in the matrix). For block diagonal matrices, we can simply apply each of these algorithms for each block at a time. he Cholesky decomposition can be specialized to oeplitz matrices provides an algorithm that requires Onk ( ) operations. 11

12 implements an he decomposition can be used to perform a number of operations. he BLAS On ( ) algorithm for solving y = Lx, where L is a lower triangular matrix. A similar algorithm applies to an upper triangular matrix. Now, given the LU decomposition A = LU, we can solve Ax = y by solving Lz = y and then solving Ux = z. We can compute the determinant of a matrix by computing the product of the diagonal terms of L. We can compute the inverse of a matrix and condition number of a matrix using related algorithms. Specialized algorithms exist to factorize a matrix that is subject to a rank-one update. A rank one update to the matrix A is a matrix of the form A' = A+ uv. If we already have an LU of Cholesky Decomposition of A, we can determine the decomposition of A ' in On ( ) operations. his type of update is used extensively in optimization and nonlinear equations, where the BFGS and Broyden updates are rank two updates. Sparse matrices have their own special algorithms. Let us let s denote the number of non-zero elements in a sparse matrix. he a matrix-vector multiply requires Os () operations. Algorithms for performing the decomposition of sparse matrices fall into two classes. he first class of algorithms rely on direct factorization. hese algorithms rely on the fact that if the matrix is sparse, them it may be possible to generate a decomposition that preserves this sparsity. Pivoting is used here are a way to ensure sparsity of the decomposition rather than ensure numerical stability. he second class of algorithms are iterative in nature. hey rely on the fact that the operation y Ax is relatively cheap. 1

13 3.5 he Schur and Singular Value Decompositions A related problem is computing the eigenvalues of a matrix. Computing the eigenvalues of a matrix is more complicated than solving a set of linear equations. We can immediately see that no algorithm will be able to exactly compute the eigenvalues of a matrix. his is a direct consequence of Galois heory, which proves that the problem of finding the roots of a polynomial of order 5 or larger does not admit a closed form expression in radicals. Now, recall that the eigenvalues of an n -dimensional matrix can be found by solving an n th order polynomial. Hence, we cannot expect an algorithm of the type we used for the LU and Cholesky decompositions. Actual algorithms for computing the eigenvalues and singular value decomposition are iterative in nature. he actual algorithms for both cases are quiet complicated. Numerical recipes actually suggests that this is one of the few areas where relying on black-box code is actually recommended. It able 3.5, we summarize the various linear algebra problems for a number of different types of matrices. 3.6 Estimation of Linear ARMA Models Consider the following time series regression model, y = β ' x + σε where t t t εt = ut + ρut 1 and ut ~ N (0,1). his process for ε t is called a first-order moving average process, or an MA1 process. Notice that, 13

14 able 3.5 Matrix Algorithms Factor Solve Linear System Compute Inverse Compute Eigenvalues Determinant Condition Number General Matrix LU ri. Solve ri. Inverse Schur Gen. Multiply Diag. O(n^3) O(n^) given LU O(n^3) given LU Iterative O(n) given LU O(n^) given LU dgetrf dgetrs dgetri dgeev dgecon Sym. Indef. BK Spec. Spec. Schur. Symmetric??? O(n^3) O(n^) given BK O(n^3) given BK Iterative O(n^) given BK dsytrf dsytrs dsytri dsyev dsycon Sym. Pos. Def. Cholesky ri. Solve Cholesky Schur. Symmetric Multiply Diag. O(n^3) O(n^) given Chold. O(n^3) given Chold. Iterative O(n) given Chold. O(n^) given Chold. dpotrf dpotrs dpotri dsyev dpocon Lower/Upper N/A ri. Solve ri. Inverse Direct Direct Direct O(n^) O(n^3) O(n) O(n) O(n) dtrsv (BLAS) dtrtri dtrcon Band Band LU ri. Band Solve Schur Gen. Band LU Band LU O(n*k^) O(n*k) given LU O(n^*k) given LU Iterative O(n) given LU O(n*k) given LU dgbtrf dgbtrs dgbtri dgeev dgbcon Sym. Indef. Band Band LU ri. Band Solve Schur Band Sym. Band LU Band LU O(n*k^) O(n*k) given LU O(n^*k) given LU Iterative O(n) given LU O(n*k) given LU dgbtrf dgbtrs dgbtri dsbev dgbcon Sym. Pos. Def. Band Band Cholesky Band Cholesky Band Cholesky Schur Band SPD Band Cholesky Band Cholesky O(n*k^) O(n*k^) O(n^*k) Iter. O(n^*k) O(n^*k) dpbtrf dpbtrs dpbtri dsbev dpbcon Lower/Upper Band N/A ri. Solve??? Direct Direct Direct O(n*k) O(n) O(n) O(n) dtbsv (BLAS) dtbcon Block Diagonal LU by Block ri. Solve by Block ri. Inv. By Block??? Multiply Diag. LU by Block O(b*nb^3) O(b*nb^) O(b*nb^3) O(n) O(b*nb^) oeplitz SPD eop Chold. O(n^) Band oeplitz SPD Band eop Chold. O(n*k) 14

15 Y X ' β ~ N(0, Ω ) where, σ (1 + ρ ) σ ρ σ ρ σ(1 + ρ ) σ ρ 0... Ω= 0 σ ρ σ(1 + ρ ) σ ρ σ ρ σ(1 + ρ ) he maximum likelihood estimator is given by, ˆ β ˆ ρ = β Ω β (, ) argmax( Y X ' )' 1 ( Y X ' ) ( β, ρ) A naive approach to computing this estimator would form 1 Ω using a standard algorithm (e.g. the Cholesky decomposition). his system is special in a number of ways. First, the matrix Ω is band diagonal. A band diagonal matrix can be decomposed in Onk ( ) operations instead of if bands. Second, forming 3 On ( ) where n is the number of rows and k is the number 1 Ω using the band-cholesky decomposition is not efficient either because the inverse of a band matrix is not necessarily a band matrix with the same number of bands. More generally, sparse matrices do not have sparse inverse. Instead, we can take the following approach. First, from the Cholesky decomposition Ω= LL using the band Cholesky decomposition. his decomposition preserves the banded structure. Next, compute v= Y X ' β. hird, solve Lx = v for x. his can be preformed in Onk ( ) operations. Finally, evaluate the likelihood function as x ' x. All the operations together mean that the likelihood can be evaluated in On ( ) operations, since k is fixed at. 15

16 Now consider the more general problem of the ARMA(1,1) model. We have yt = β ' xt + σεt where εt = ρεt 1+ ut + γ ut 1 and ut ~ N (0,1). In this case, we have, ˆ β ˆ ργˆ = β Ω β (,, ) argmax( Y X ' )' 1 ( Y X ' ) ( β, ρ, γ) where, ( ρ+ γ) ( ρ+ γ) ρ ( ρ+ γ) ρ ( ρ+ γ) ρ σ (1 + ) σ ( ρ+ γ + ) σ ρ( ρ + γ + ) σ ρ ( ρ+ γ + )... 1 ρ 1 ρ 1 ρ 1 ρ ( ρ+ γ) ρ ( ρ+ γ) ( ρ+ γ) ρ ( ρ+ γ ) ρ σ ( ρ+ γ + ) σ (1 + ) σ ( ρ+ γ + ) σ ρ( ρ+ γ + )... 1 ρ 1 ρ 1 ρ 1 ρ Ω= ( ρ+ γ) ρ ( ρ+ γ) ρ ( ρ+ γ) σ ρ( ρ + γ + ) σ ( ρ+ γ + ) σ ( ( ρ+ γ) ρ 1 + ) σ ( ρ+ γ + )... 1 ρ 1 ρ 1 ρ 1 ρ ( ρ+ γ) ρ ( ρ+ γ) ρ ( ρ+ γ ) ρ ( ρ+ γ ) σ ρ ( ρ + γ + ) σ ρ( ρ+ γ + ) σ ( ρ + γ + ) σ (1 + )... 1 ρ 1 ρ 1 ρ 1 ρ his matrix is no longer a band diagonal matrix. It is, however, a oeplitz matrix. Hence, we can apply a similar procedure to the one above. In this case, we have an On ( ) algorithm for the Cholesky factorization. When is quite large, then the elements far away from the diagonal will get quite small. Suppose that we trim all elements that are less than In this case, we will have a band oeplitz matrix. We can then compute the Cholesky decomposition in Onk ( ) operations. his is a huge difference from the naïve 3 On ( )! For example, when n = 1,000 and k = 30 3 n, then 1111 nk. 16

17 3.7 Solving Linear Differential Equations Initial Value Problems A linear first order differential equation can be written as, dy dx = f ( xy ) + gx ( ) he simplest method of solving such an equation is to use finite difference methods. We compute a solution y k on a grid of equally spaced points [ y1, y,..., y K ] with a spacing of δ. We can then approximate dy using y k+ 1 yk dx. Using this, we can write, δ y y δ k+ 1 k = fkyk + gk his system will be pinned down if we specify y 0. hen, we can compute all future values using yk+ 1 = yk + δ fkyk + δ gk. his type of problem is called an initial value problem, because it can be solved by iteration. Boundary Value Problems he above example is somewhat atypical. In most social science applications, we have a second order derivative as follows, d y dx dy = f ( x) + g( x) y+ h( x) dx We can similarly use a finite difference scheme to give, 17

18 We can rearrange this to give, yk+ 1 yk + yk 1 yk+ 1 yk = f + gy + h δ δ k k k k y = δ( y y ) f + δ g y + δ h y + y k+ 1 k+ 1 k k k k k k 1 k If we have the values y 0 and y 1, then we can solve the system by iteration. Usually, this is not the case, however. In most real problems, we have boundary value problems. Suppose that we have y 0 and y K. It turns outs that combining the above equation with the equation y0 = a and yk = b, we have a tri-diagonal system of linear equations. his turns out to be a rather generally property of boundary value problems- they can be solved by applying band-diagonal factorizations. 3.8 References [1] Numerical Recipes in C. [] Golub and Van Loan. Matrix Computations. [3] Hamilton- ime Series Analysis 18