Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18
Part 2: Direct Methods PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 2
overview definitions GAUSSIAN elimination CHOLESKY decomposition QR decomposition PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 3
Definitions direct methods algorithms for exactly solving linear systems (avoiding round off errors) within finite amount of steps nowadays seldom used for solving huge linear systems (due to their large complexities) nevertheless, frequently used (in incomplete form) as preconditioners within iterative methods typical methods GAUSSIAN elimination CHOLESKY decomposition QR decomposition GRAM SCHMIDT method GIVENS method PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 4
overview definitions GAUSSIAN elimination CHOLESKY decomposition QR decomposition PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 5
GAUSSIAN Elimination basic concept GAUSSIAN elimination (GE) successively transforms a linear system Ax b into an equivalent system LRx b with upper triangular matrix R and lower triangular matrix L that can be solved via simple forward / backward substitution Definition 2.1 The decomposition of a matrix A into a product A LR consisting of a lower triangular matrix L and an upper triangular matrix R is called LR decomposition. In literature this is also referred to as LU (lower, upper) decomposition. PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 6
GAUSSIAN Elimination basic concept (cont d) we denote k th row P kj (2.1.1) j th row as permutation matrix that emerges from the identity matrix I via transposition of j th and k th row (j k) for k j follows P kj I PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 7
GAUSSIAN Elimination basic concept (cont d) furthermore, the lower triangular matrix L to be represented via multiplicative combination of matrices L k (2.1.2) those matrices differ at most in one column from identity matrix with (2.1.1) and (2.1.2) we are able to formulate the essential part of GE PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 8
GAUSSIAN Elimination algorithm: GAUSSIAN elimination (LR decomposition) A (1) : A for k 1,..., n 1 choose from k th column of A (k) some arbitrary element 0 with j k define P kj with above j and k according to (2.1.1) à (k) : P kj A (k) define L k according to (2.1.2) with, i k 1,..., n A (k 1) : L k à (k) with A (n) we get an upper triangular matrix R that can be used for a simple solution of the system question: where to get matrix L from? PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 9
GAUSSIAN Elimination existence and uniqueness of LR decomposition simple examples such as A show not each regular matrix necessarily exhibits an LR decomposition let A be regular, then A exhibits an LR decomposition if and only if det A[k] 0 k 1,..., n with A[k] : for k 1,..., n being the principal k k submatrix of A and det A[k] the principal determinant of A proof is lengthy PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 10
GAUSSIAN Elimination algorithm: GAUSSIAN elimination w/o pivoting for k 1,..., n 1 for i k 1,..., n a ik : a ik / a kk for j k 1,..., n a ij : a ij a ik a kj for k 2,..., n for i 1,..., k 1 b k : b k a ki b i for k n,..., 1 for i k 1,..., n b k : b k a ki x i LR decomposition A : LR forward substitution b : L 1 b backward substitution x : R 1 b x k : b k / a kk PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 11
GAUSSIAN Elimination complexity only expensive multiplications and divisions to be considered for k 1,..., n 1 for i k 1,..., n a ik : a ik / a kk for j k 1,..., n a ij : a ij a ik a kj for k 2,..., n for i 1,..., k 1 b k : b k a ki b i for k n,..., 1 for i k 1,..., n b k : b k a ki x i x k : b k / a kk (n k) times (inner loop) for k 1,..., n 1 (n k) 2 times (inner loops) for k 1,..., n 1 (k 1) times (inner loop) for k 2,..., n (n k) times (inner loop) for k 1,..., n n times PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 12
GAUSSIAN Elimination complexity (cont d) only expensive multiplications and divisions to be considered #divisions #multiplications hence, #divisions #multiplications such that the total complexity of GAUSSIAN elimination can be estimated as (n 3 ) on a standard computer (3 GHz) we need for n 10 4 approx. 5 minutes PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 13
GAUSSIAN Elimination error analysis let 1, such that (according to machine precision) resp. applies consider the following linear system Ax b with and its exact solution PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 14
GAUSSIAN Elimination error analysis (cont d) with GAUSSIAN elimination we get (1) and due to present computational accuracy substituting this into first equation of (1) yields hence x 1 0 follows PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 15
GAUSSIAN Elimination error analysis (cont d) a previous row interchange yields the following linear system hence, with GAUSSIAN elimination follows and we get x 2 0.5 and x 1 1 x 2 0.5 as (correct) solution permutation of rows and columns not only makes sense in case such strategies are called pivoting PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 16
GAUSSIAN Elimination pivoting we consider three different types of pivoting: column pivoting define P kj according to (2.1.1) with j index and consider for A (k) x b the equivalent linear system P kj A (k) x P kj b PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 17
GAUSSIAN Elimination pivoting (cont d) we consider three different types of pivoting: row pivoting define P kj according to (2.1.1) with j index and consider the linear system A (k) P kj y b x P kj y PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 18
GAUSSIAN Elimination pivoting (cont d) we consider three different types of pivoting: total pivoting define P kj1, P kj2 according to (2.1.1) with j 1 index j 2 index and consider the linear system P kj1 A (k) P kj2 y P kj1 b x P kj2 y PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 19
overview definitions GAUSSIAN elimination CHOLESKY decomposition QR decomposition PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 20
CHOLESKY Decomposition basic concept complexity for LR decomposition of symmetric positive definite (SPD) matrices during GAUSSIAN elimination can be further reduced Definition 2.2 The decomposition of a matrix A into a product A LL T with a lower triangular matrix L is called CHOLESKY decomposition. for each SPD matrix A exists exactly one lower triangular matrix L with l ii 0, i 1,..., n, such that A LL T applies PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 21
CHOLESKY Decomposition basic concept (cont d) let s consider a column wise computation of matrix coefficients we assume all l ij for i 1,..., n and j k 1 are known hence, from follows the relation thus l kk can be computed according to (2.2.1) PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 22
CHOLESKY Decomposition basic concept (cont d) from for i k 1,..., n follows a rule for computing elements of k th column (below diagonal) via for i k 1,..., n (2.2.2) with (2.2.1) and (2.2.2) we are able to formulate the algorithm for the CHOLESKY decomposition PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 23
CHOLESKY Decomposition algorithm CHOLESKY decomposition A : LL T forward substitution b : L 1 b backward substitution x : L T b for k 1,..., n for j 1,..., k 1 a kk : a kk a kj a kj a kk : for i k 1,..., n for j 1,..., k 1 a ik : a ik a ij a kj a ik : a ik / a kk for k 1,..., n for i 1,..., k 1 b k : b k a ki b i b k : b k / a kk for k n,..., 1 for i k 1,..., n b k : b k a ik x i x k : b k / a kk (2.2.1) (2.2.2) PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 24
CHOLESKY Decomposition complexity solely decomposition (as most complex part) to be considered only expensive multiplications, divisions, and roots for k 1,..., n for j 1,..., k 1 a kk : a kk a kj a kj a kk : for i k 1,..., n for j 1,..., k 1 a ik : a ik a ij a kj a ik : a ik / a kk (k 1) times (inner loop) for k 1,..., n n times (n k) (k 1) times (inner loops) for k 1,..., n (n k) times (inner loop) for k 1,..., n PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 25
CHOLESKY Decomposition complexity (cont d) #multiplications #divisions #roots 0 0 hence, for large n CHOLESKY decomposition needs approximately only half of the expensive operations of GAUSSIAN elimination PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 26
overview definitions GAUSSIAN elimination CHOLESKY decomposition QR decomposition PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 27
QR Decomposition basic concept essential foundation for GMRES ( iterative method) and many methods for solving eigenvalue problems or linear regressions due to Q Q 1, the linear system Ax b is easily to be solved via Ax b QRx b Rx Q b the three most well known methods for this decomposition are GIVENS GRAM SCHMIDT or HOUSEHOLDER (both not to be considered here) Definition 2.3 The decomposition of a matrix A into a product A QR with a unitary matrix Q and an upper triangular matrix R is called QR decomposition. PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 28
QR Decomposition GIVENS method let s assume the following matrix (confined to the case A ) A j th row i th column PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 29
QR Decomposition GIVENS method (cont d) idea: successively eliminate elements below main diagonal starting from first column, sub diagonal elements of each column become nullified in ascending order via orthogonal rotation matrices for previous matrix A applies a kl 0 l 1,..., i 1 with l k 1,..., n, a i 1,i... a j 1,i 0 (2.3.1) (2.3.2) and a ji 0 in order to nullify a ji, we look for an orthogonal matrix G ji PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 30
QR Decomposition GIVENS method (cont d) let G ji be an orthogonal matrix PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 31
QR Decomposition GIVENS method (cont d) such that for à G ji A in addition to ã kl 0 l 1,..., i 1 with l k 1,..., n, (2.3.3) and ã i 1,i... ã j 1,i 0 (2.3.4) also ã ji 0 (2.3.5) applies PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 32
QR Decomposition GIVENS method (cont d) at first, Ã and A solely differ in i th and j th row, and for l 1,..., n ã il g ii a il g ij a jl ã jl g ji a il g jj a jl applies with(2.3.1) follows a il a jl 0 for l i j, thus ã il ã jl 0 for l 1,..., i 1 and hence the requirements (2.3.3) and (2.3.4) are fulfilled well defined via a ji 0 we set g ij g ji and g ii g jj PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 33
QR Decomposition GIVENS method (cont d) thus, G ji represents an orthogonal rotation matrix with angle arccos g ii and the following applies defining G ji I in case of a matrix A that satisfies (2.3.1) and (2.3.2) and furthermore implies a ji 0, then with ~ Q : G ji : G n,n 1... G 3,2 G n,1... G 3,1 G 2,1 we get an orthogonal matrix for which R QA yields an upper triangular matrix with Q Q~ T follows A QR ~ PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 34
QR Decomposition algorithm for i 1,..., n 1 for j i 1,..., n Y a ji 0 t : 1 / s : ta ji c : ta ii for k i,..., n 1 t : ca ik sa jk Y k i N N QR decomposition A : QR By using GIVENS method, there is no need to explicitly store the orthogonal matrix. Here, A will be extended by the right hand side b according to a n 1 b. for i n,..., 1 for j i 1,..., n a i,n 1 : a i,n 1 a ij x j a ik : t a ji : 0 a jk : sa ik ca jk x i : a i,n 1 / a ii back substitution x : R 1 b PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 35
QR Decomposition complexity for i 1,..., n 1 for j i 1,..., n Y a ji 0 t : 1 / s : ta ji c : ta ii for k i,..., n 1 t : ca ik sa jk Y a ik : t a ji : 0 k i N N a jk : sa ik ca jk solely decomposition (as most complex part) to be considered w/o right hand side b only expensive multiplications, divisions, and roots (n i) times (inner loop) for i 1,..., n 1 (n i) times (inner loop) for i 1,..., n 1 (n i) times (inner loop) for i 1,..., n 1 (n i) times (inner loop) for i 1,..., n 1 2(n i 1)(n i) times (inner loops) for i 1,..., n 1 2(n i 1)(n i) times (inner loops) for i 1,..., n 1 PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 36
QR Decomposition complexity (cont d) #multiplications #divisions #roots hence, the complexity is approximately four times larger than GAUSSIAN elimination PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 37
overview definitions GAUSSIAN elimination CHOLESKY decomposition QR decomposition PD Dr. Ralf Peter Mundani Computational Linear Algebra Winter Term 2017/18 38