BLAS: Basic Linear Algebra Subroutines Analysis of the Matrix-Vector-Product Analysis of Matrix-Matrix Product
|
|
- Jasper Harvey
- 5 years ago
- Views:
Transcription
1 Level-1 BLAS: SAXPY BLAS-Notation: S single precision (D for double, C for complex) A α scalar X vector P plus operation Y vector SAXPY: y = αx + y Vectorization of SAXPY (αx + y) by pipelining: page 8 of 39
2 SAXPY Parallelization by Partitioning {1, 2,, n} = 1, n = I 1 I 2 I R x 1 x 2 x = = x n X 1 X R, y = y 1 y 2 = y n Y 1 Y R Using short vectors X j and Y j of length n R Each processor P j gets partial vector X j and Y j and computes Y j = αx j + Y j, j = 1, 2,, R Result: SAXPY very good vectorizable and parallelizable page 9 of 39
3 Further Level-1 BLAS Routines SCOPY: y = x or y x (compare SAXPY) DOT-product x T y: n norm: x 2 = j=1 i x iy i x 2 j = x T x (compare DOT-product) page 10 of 39
4 Level-2 BLAS Matrix-Vector operations with O(n 2 ) operations (sequentially) BLAS-Notation: S single precision G E } general matrix M V vector defines SGEMV, matrix-vector product: y = αax + βy Other Level-2 BLAS: solving triangular system Lx = b with triangular matrix L page 11 of 39
5 Level-3 BLAS Matrix-Matrix operations with O(n 3 ) operations (sequentially) BLAS-Notation: S single precision G E } general matrix M M matrix defines SGEMM, matrix-matrix product: C = αab + βc page 12 of 39
6 Granularity for BLAS BLAS level operation formula memory granularity BLAS-1 AXPY: 2n αx + y 2n + 1 < 1 BLAS-2 GEMV: 2n 2 αax + βy n 2 + 2n 2 BLAS-3 GEMM: 2n 3 αab + βc 4n 2 n 2 BLAS-3 has best operations to memory ratio! page 13 of 39
7 22 Analysis of the Matrix-Vector-Product A = (a ij ) i=1,,n j=1,,m 221 Vectorization c 1 a 11 a 1m = c n a n1 a nm m a 1j b j j=1 = m a nj b j j=1 R n m, b R m, c R n b 1 b n = a 11 b a 1m b m = a n1 b a nm b m m j=1 b j a 1j a nj = n DOT-products of length m m SAXPYs of length n (GAXPY) page 14 of 39
8 Pseudocode: ij-form c = 0; for i=1,,n for j=1,,m c i = c i + a ij b j end end } DOT-product c i = A i b, DOT-product of ith row of A with vector b page 15 of 39
9 Pseudocode: ji-form c = 0; for j=1,,m for i=1,,n c i = c i + a ij b j end end SAXPY updating vector c with jth column of A GAXPY: Sequence of SAXPYs related to the same vector Advantage: vector c, that is updated, can be kept in fast memory No additional data transfer page 16 of 39
10 GAXPY (repetition) SAXPY: y := y + αx GAXPY: y = y 0 for i = 1 : n y := y + α i x i end Series of SAXPYs regarding the same vector y length(gaxpy) = length(y) Advantage: less data transfer! page 17 of 39
11 222 Parallelization by Building Blocks Reduce matrix-vector product on smaller matrix-vector products {1, 2,, n} = 1, n = I 1 I 2 I R disjunct: I j I k = 0 for j k {1, 2,, m} = 1, m = J 1 J 2 J S disjunct: J j J k = 0 for j k Use 2-dimensional array of processors P rs P rs gets matrix block A rs := A(I r, J s ), b s := b(j s ), c r := c(i r ) c r = S A rs b s =: s=1 S s=1 c (s) r page 18 of 39
12 Pseudocode for r = 1,, R for s = 1,, S c (s) r = A rs b s ; end end for r = 1,, R c r = 0 for s = 1,, S c r = c r + c (s) r ; end end Small, independent matrix-vector products No communication necessary during computations! Blockwise collection and addition of vectors Rowwise communication! Fan-in page 19 of 39
13 Blocking: Special Cases S = 1: The computation of A i b is vectorizable by GAXPYs c = A 1 A 2 A 1 b b = A 2 b No communication necessary between processor P 1,, P R R = 1: A j b j are independent c = (A 1 A 2 ) b 1 b 2 = A 1b 1 + A 2 b 2 + Then collection of partial results from processor P 1,, P S Fan-in Final sum in one processor: vectorizable by GAXPYs page 20 of 39
14 Rules 1 Inner loops of a program should be simple, vectorizable 2 Outer loop of a programm should be substantial, independent, parallelizable substantial and parallelizable simple, vectorizable 3 Reuse of data (cache, minimal data transfer, blocking) page 21 of 39
15 223 c = Ab for Banded Matrix Bandwidth β (symmetric) 2β+1 diagonals: main diag + β subdiag + β superdiag β = 1: tridiagonal page 22 of 39
16 Notation: Banded Matrices A and à a 11 a 1,β a 22 a A = β+1,1 0 0 an β,n an 1,n a n,n β a nn ã 10 ã 1,β 0 0 ã 20 ã à = β+1, β 0 0 ãn β,β ãn 1,0 0 0 ã n, β ã n,0 page 23 of 39
17 c = Ab for Banded Matrix Storing entries diagonalwise: n(2β + 1) matrix instead of n 2 ã i,s = a i,i+s for row i = 1,, n 1 i + s n and β s β and 1 i n 1 i s n i and β s β in row i s [l i, r i ] = [max{ β, 1 i}, min{β, n i}] 1 s i n s and 1 i n in diag s i [ l s, r s ] = [max{1, 1 s}, min{n, n s}] page 24 of 39
18 Computation of the mtx-vec product based on storage scheme on vector CPUs For i = 1,, n : c i = A i b = j General TRIAD, no SAXPY: for s = β : β r i a ij b j = a i,i+s b i+s = s=l i for i = max{1 s, 1} : min{n s, n} c i = c i + ã i,s b i+s end end or, partial DOT-product: for i = 1 : n for s = max{ β, 1 i} : max{β, n i} c i = c i + ã i,s b i+s end end Sparsity less operations, but also loss of efficiency r i s=l i ã i,s b i+s page 25 of 39
19 Band Ab in Parallel Partitioning: R 1, n = I r, disjunct r=1 for i I r c i = end r i s=l i ã is b i+s Processor P r gets rows to index set I r := [m r, M r ] in order to compute its part of the final vector c What part of vector b does processor P r need in order to compute its part of c? page 26 of 39
20 Band Ab in Parallel Necessary for I r : b j = b i+s : j = i + s m r + max{ β, 1 m r } = max{m r β, 1} j = i + s M r + r Mr = M r + min{β, n M r } = min{m r + β, n} Processor P r with index set I r needs from b the indices j [max{1, m r β}, min{n, M r + β}] page 27 of 39
21 23 Analysis of Matrix-Matrix Product A = (a ij ) i=1,,n j=1,,m R n m, B(b ij ) i=1,,m j=1,,q R m q, for i = 1 : n for j = 1 : q m c ij = a ik b kj end end k=1 C = AB = (c ij ) i=1,,n j=1,,q R n q page 28 of 39
22 231 Vectorization Algorithm 1: (ijk)-form: for i = 1 : n for j = 1 : q for k = 1 : m c ij = c ij + a ik b kj } DOT-product of length m end end end c ij = A i B j for all i, j All entries c ij are fully computed, one after another Access to A and C is rowwise, to B columnwise (depends on inner most loops!) page 29 of 39
23 Other View on the Matrix-Matrix Product Matrix A considered as combination of columns or rows A = A 1 e T 1 + +A m e T m = (A 1 0 )+(0 A 2 0 )+ +( 0 A m ) a 1 = e 1 a e n a n = a n AB = n A j ej T j=1 m k=1 e k b k = k,j A j (e T j e k )b k = m A k b k }{{} k=1 full n q matrices as a sum of full matrices A k b k by outer product of the kth column of A and the kth row of B page 30 of 39
24 Algorithm 2: (jki)-form for j=1,,q for k=1,,m for i=1,,n c ij = c ij + a ik b kj end end end Vector update: c j = c j + a k b kj Sequence of SAXPYs for the same vector: c j = k b kj a k C computed columnwise; access to A columnwise Access to B columnwise, but delayed page 31 of 39
25 Algorithm 3: (kji)-form for k=1,,m for j=1,,q for i=1,,n c ij = c ij + a ik b kj end end end Vector update: c j = c j + a k b kj Sequence of SAXPYs for different vectors c j (no GAXPY) Access to A columnwise Access to B rowwise + delayed C computed with intermediate values c (k) ij which are computed columnwise page 32 of 39
26 Overview of Different Forms Access A by Access B by to to Comput of C ijk ikj kij jik jki kji Alg 1 Alg 2 Alg 3 row row column column column row row column row row row column column column Computat ion of c ij direct delayed delayed direct delayed delayed Vector operation Vector length DOT GAXPY SAXPY DOT GAXPY SAXPY m q q m n n Better: GAXPY (longer vector length) Access to matrices according to storage scheme (rowwise or columnwise) page 33 of 39
27 232 Matrix-Matrix Product in Parallel 1, n = R I r, 1, m = r=1 S T K s, 1, q = Distribute the blocks relative to index sets I r, K s, and J t to processor array P rst : s=1 t=1 J t 1 Processor P rst computes small matrix-matrix product All processors in parallel: c (s) rt = A rs B st 2 Compute sum by fan-in in s: c rt = S s=1 c (s) rt page 34 of 39
28 Mtx-Mtx in Parallel: Special Case S = 1 Each processor P rt can compute its part of c, c rt, independently without communication Each processor needs full block of rows of A, relative to index set I r, and full block of columns of B, relative to index set J t, to compute c rt relative to rows I k and columns J t page 35 of 39
29 Mtx-Mtx in Parallel: Special Case S = 1 With n q processors each processor has to compute one DOT-product with O(m) parallel time steps c rt = m a rk b kt k=1 Fan-in by m nq additional processors for all DOT-products reduces number of parallel time steps to O(log(m)) page 36 of 39
30 1D-Parallelization of A B 1D: p processors linear, each processor gets full A and column slice of B, computing the related column slice of C = AB Communication: N 2 p for A and (N N p ) p = N2 Granularity: N 3 N 2 (1 + p) = N 1 + p Blocking only in i, the columns of B! for i = 1 : n for j = 1 : n for k = 1 : n C j,i = C j,i + A j,k B k,i for B page 37 of 39
31 2D-Parallelization of A B 2D: p processors square, q := p, each proc gets row slice of A and column slice of B computing full subblock of C = AB Communication: N 2 p for A and N 2 p for B Granularity: N 3 2N 2 p = N 2 p Blocking in i and j, the columns of B and the rows of A! for i = 1 : n for j = 1 : n for k = 1 : n C j,i = C j,i + A j,k B k,i page 38 of 39
32 3D-Parallelization A B 3D: p processors cubic, each processor gets subblock of A and subblock of B, computing part of subblock of C = AB Additional fan-in to collect parts to full subblock of C (q = p 1 3 ) Communication: N 2 p 1 3 for A and for B ( = p N2 p 2 3 ) = p blocksize, fan-in: N 2 p 1 3 Granularity: N 3 3N 2 p 1 3 = N 3p 1 3 Blocking in i, j, and k! for i = 1 : n for j = 1 : n for k = 1 : n C j,i = C j,i + A j,k B k,i page 39 of 39
33 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 1 of 36
34 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Contents 1 Introduction 11 Computer Science Aspects 12 Numerical Problems 13 Graphs 14 Loop Manipulations 2 Elementary Linear Algebra Problems 21 BLAS: Basic Linear Algebra Subroutines 22 Matrix-Vector Operations 23 Matrix-Matrix-Product 3 Linear Systems of Equations with Dense Matrices 31 Gaussian Elimination 32 Parallelization 33 QR-Decomposition with Householder matrices 4 Sparse Matrices 41 General Properties, Storage 42 Sparse Matrices and Graphs 43 Reordering 44 Gaussian Elimination for Sparse Matrices 5 Iterative Methods for Sparse Matrices 51 Stationary Methods 52 Nonstationary Methods 53 Preconditioning 6 Domain Decomposition 61 Overlapping Domain Decomposition 62 Non-overlapping Domain Decomposition 63 Schur Complements Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 2 of 36
35 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition 31 Linear Systems of Equations with Dense Matrices 311 Gaussian Elimination: Basic Properties Linear system of equations: a 11 x a 1n x n = b 1 a n1 x a nn x n = b n Solve Ax = b a 11 a 1n a n1 a nn x 1 x n = b 1 b n Generate simpler linear equations (matrices) Transform A in triangular form: A = A (1) A (2) A (n) = U Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 3 of 36
36 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Transformation to Upper Triangular Form a 11 a 12 a 1n a 21 a 22 a 2n a n1 a n2 a nn row transformations: (2) (2) a 21 a 11 (1),, (n) (n) a n1 a 11 (1) leads to a 11 a 12 a 13 a 1n 0 a (2) 22 a (2) 23 a (2) 2n A (2) = 0 a (2) 32 a (2) 33 a (2) 3n 0 a (2) n2 a (2) n3 a (2) nn next transformations: (3) (3) a(2) 32 a (2) 22 Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices (2),, (n) (n) a(2) n2 (2) a (2) 22 page 4 of 36
37 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Transformation to Triangular Form (cont) a 11 a 12 a 13 a 1n 0 a (2) 22 a (2) 23 a (2) 2n A (3) = 0 0 a (3) 33 a (3) 3n 0 0 a (3) n3 a (3) nn next transformations: (4) (4) a(3) 43 (3),, (n) (n) a(3) n3 (3) a (3) 33 a (3) 33 a 11 a 12 a 13 a 1n 0 a (2) 22 a (2) 23 a (2) 2n A (n) = 0 0 a (3) 33 a (3) 3n = U a (n) nn Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 5 of 36
38 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Pseudocode Gaussian Elimination (GE) Simplification: assume that no pivoting is necessary a (k) kk 0 or a(k) kk ρ > 0 for k = 1, 2,, n for k = 1 : n 1 for i = k + 1 : n l i,k = a i,k a k,k end for i = k + 1 : n for j = k + 1 : n a i,j = a i,j l i,k a k,j end end end In practice: Include pivoting and include right hand side b There is still to solve a triangular system in U! Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 6 of 36
39 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Intermediate Systems A (k), k = 1, 2,, n with A = A (1) and U = A (n) a (1) 11 a (1) 1,k 1 a (1) 1,k a (1) 1,n 0 (k 1) a k 1,k 1 a (k 1) k 1,k a (k 1) k 1,n 0 0 a (k) k,k a (k) k,n 0 0 a (k) n,k a (k) n,n Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 7 of 36
40 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Define Auxiliary Matrices L = l 2,1 1 0 l n,1 l n,n 1 1 and U = A (n) L k := , L = I l k+1,k 0 0 k 0 0 l n,k 0 0 L k Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 8 of 36
41 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Elimination Step in Terms of Auxiliary Matrices A (k+1) = (I L k ) A (k) = A (k) L k A (k) U = A (n) = (I L n 1 ) A (n 1) = = (I L n 1 ) (I L 1 )A (1) = L A L := (I L n 1 ) (I L 1 ) A = L 1 U with U upper triangular and L lower triangular Theorem 2: L 1 = L and therefore A = LU Advantage: Every further problem Ax = b j can be reduced to (LU)x = b j for arbitrary j Solve two triangular problems (LU)x = Ly = b and Ux = y Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 9 of 36
42 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Theorem 2: L 1 = L A = LU for i j : L i L j = (I + L j )(I L j ) = I + L j L j L 2 j = I (I L j ) 1 = I + L j (I + L i )(I + L j ) = I + L i + L j + L i L j = I + L i + L j }{{} L 1 = [(I L n 1 ) (I L 1 )] 1 = (I L 1 ) 1 (I L n 1 ) 1 = (I + L 1 )(I + L 2 ) (I + L n 1 ) = I + L 1 + L L n 1 = L Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 10 of 36
43 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition 32 GE in Parallel: Blockwise Main idea: Blocking of GE to avoid data transfer between processors Basic Concepts: Replace GE or large LU-decomposition of full matrix by small intermediate steps (by sequence of small block operations): Solving collection of small triangular systems LU k = B k (parallelism in columns of U) A A LU updating matrices (also easy to parallelize) small B = LU-decompositions (parallelism in rows of B) Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 11 of 36
44 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition How to Choose Blocks in L/U Satisfying LU = A L U 11 U 12 U 13 A 11 A 12 A 13 L 21 L U 22 U 23 = = L 31 L 32 L U 33 A 21 A 22 A 23 A 31 A 32 A 33 L 11 U 11 L 11 U 12 L 11 U 13 = L 21 U 11 L 21 U 12 + L 22 U 22 L 21 U 13 + L 22 U 23 L 31 U 11 L 31 U 12 + L 32 U 22 Different ways of computing L and U depending on start (assume first entry/row/column of L/U as given) how to compute new entry/row/column of L/U update of block structure of L/U by grouping in known blocks blocks newly to compute blocks to be computed later Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 12 of 36
45 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Crout Form Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 13 of 36
46 Linear Systems of Equations with Dense Matrices GE in Parallel: Blockwise QR-Decomposition Crout Form (cont) 1 Solve by small LU-decomposition of the modified part of A L 22, L 32, and U 22 2 Solve by solving small triangular systems of equations in L 22 U 23 Initial steps: L 11 U 11 = A 11, ( ) ( ) L21 A21 U L 11 =, L 31 A 11 (U 12 U 13 ) = (A 12 A 13 ) 31 Parallel Numerics, WT 2014/ Linear Systems of Equations with Dense Matrices page 14 of 36
Computation of the mtx-vec product based on storage scheme on vector CPUs
BLAS: Basic Linear Algebra Subroutines BLAS: Basic Linear Algebra Subroutines BLAS: Basic Linear Algebra Subroutines Analysis of the Matrix Computation of the mtx-vec product based on storage scheme on
More information2.1 Gaussian Elimination
2. Gaussian Elimination A common problem encountered in numerical models is the one in which there are n equations and n unknowns. The following is a description of the Gaussian elimination method for
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More information1 Multiply Eq. E i by λ 0: (λe i ) (E i ) 2 Multiply Eq. E j by λ and add to Eq. E i : (E i + λe j ) (E i )
Direct Methods for Linear Systems Chapter Direct Methods for Solving Linear Systems Per-Olof Persson persson@berkeleyedu Department of Mathematics University of California, Berkeley Math 18A Numerical
More informationLU Factorization. LU Decomposition. LU Decomposition. LU Decomposition: Motivation A = LU
LU Factorization To further improve the efficiency of solving linear systems Factorizations of matrix A : LU and QR LU Factorization Methods: Using basic Gaussian Elimination (GE) Factorization of Tridiagonal
More informationDirect Methods for Solving Linear Systems. Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le
Direct Methods for Solving Linear Systems Simon Fraser University Surrey Campus MACM 316 Spring 2005 Instructor: Ha Le 1 Overview General Linear Systems Gaussian Elimination Triangular Systems The LU Factorization
More informationParallel Numerics. Prof. Dr. Thomas Huckle. July 2, Technische Universität München, Institut für Informatik
Parallel Numerics Prof Dr Thomas Huckle July 2, 2006 Technische Universität München, Institut für Informatik 1 Contents 1 Introduction 4 11 Computer Science Aspects of Parallel Numerics 4 111 Parallelism
More informationMODULE 7. where A is an m n real (or complex) matrix. 2) Let K(t, s) be a function of two variables which is continuous on the square [0, 1] [0, 1].
Topics: Linear operators MODULE 7 We are going to discuss functions = mappings = transformations = operators from one vector space V 1 into another vector space V 2. However, we shall restrict our sights
More informationDirect Methods for Solving Linear Systems. Matrix Factorization
Direct Methods for Solving Linear Systems Matrix Factorization Numerical Analysis (9th Edition) R L Burden & J D Faires Beamer Presentation Slides prepared by John Carroll Dublin City University c 2011
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More informationDense LU factorization and its error analysis
Dense LU factorization and its error analysis Laura Grigori INRIA and LJLL, UPMC February 2016 Plan Basis of floating point arithmetic and stability analysis Notation, results, proofs taken from [N.J.Higham,
More informationGaussian Elimination and Back Substitution
Jim Lambers MAT 610 Summer Session 2009-10 Lecture 4 Notes These notes correspond to Sections 31 and 32 in the text Gaussian Elimination and Back Substitution The basic idea behind methods for solving
More informationLecture Notes to Accompany. Scientific Computing An Introductory Survey. by Michael T. Heath. Chapter 2. Systems of Linear Equations
Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T. Heath Chapter 2 Systems of Linear Equations Copyright c 2001. Reproduction permitted only for noncommercial,
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 2 Systems of Linear Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More information5. Direct Methods for Solving Systems of Linear Equations. They are all over the place...
5 Direct Methods for Solving Systems of Linear Equations They are all over the place Miriam Mehl: 5 Direct Methods for Solving Systems of Linear Equations They are all over the place, December 13, 2012
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationLU Factorization. LU factorization is the most common way of solving linear systems! Ax = b LUx = b
AM 205: lecture 7 Last time: LU factorization Today s lecture: Cholesky factorization, timing, QR factorization Reminder: assignment 1 due at 5 PM on Friday September 22 LU Factorization LU factorization
More informationCS412: Lecture #17. Mridul Aanjaneya. March 19, 2015
CS: Lecture #7 Mridul Aanjaneya March 9, 5 Solving linear systems of equations Consider a lower triangular matrix L: l l l L = l 3 l 3 l 33 l n l nn A procedure similar to that for upper triangular systems
More informationAx = b. Systems of Linear Equations. Lecture Notes to Accompany. Given m n matrix A and m-vector b, find unknown n-vector x satisfying
Lecture Notes to Accompany Scientific Computing An Introductory Survey Second Edition by Michael T Heath Chapter Systems of Linear Equations Systems of Linear Equations Given m n matrix A and m-vector
More informationThe Solution of Linear Systems AX = B
Chapter 2 The Solution of Linear Systems AX = B 21 Upper-triangular Linear Systems We will now develop the back-substitution algorithm, which is useful for solving a linear system of equations that has
More informationDirect solution methods for sparse matrices. p. 1/49
Direct solution methods for sparse matrices p. 1/49 p. 2/49 Direct solution methods for sparse matrices Solve Ax = b, where A(n n). (1) Factorize A = LU, L lower-triangular, U upper-triangular. (2) Solve
More informationLinear Algebra Section 2.6 : LU Decomposition Section 2.7 : Permutations and transposes Wednesday, February 13th Math 301 Week #4
Linear Algebra Section. : LU Decomposition Section. : Permutations and transposes Wednesday, February 1th Math 01 Week # 1 The LU Decomposition We learned last time that we can factor a invertible matrix
More informationCS475: Linear Equations Gaussian Elimination LU Decomposition Wim Bohm Colorado State University
CS475: Linear Equations Gaussian Elimination LU Decomposition Wim Bohm Colorado State University Except as otherwise noted, the content of this presentation is licensed under the Creative Commons Attribution
More informationNumerical Linear Algebra
Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix
More informationBasic Concepts in Linear Algebra
Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University February 2, 2015 Grady B Wright Linear Algebra Basics February 2, 2015 1 / 39 Numerical Linear Algebra Linear
More informationNumerical Methods I Solving Square Linear Systems: GEM and LU factorization
Numerical Methods I Solving Square Linear Systems: GEM and LU factorization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 18th,
More informationLinear Equations and Matrix
1/60 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Gaussian Elimination 2/60 Alpha Go Linear algebra begins with a system of linear
More information14.2 QR Factorization with Column Pivoting
page 531 Chapter 14 Special Topics Background Material Needed Vector and Matrix Norms (Section 25) Rounding Errors in Basic Floating Point Operations (Section 33 37) Forward Elimination and Back Substitution
More informationReview of Basic Concepts in Linear Algebra
Review of Basic Concepts in Linear Algebra Grady B Wright Department of Mathematics Boise State University September 7, 2017 Math 565 Linear Algebra Review September 7, 2017 1 / 40 Numerical Linear Algebra
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 12: Gaussian Elimination and LU Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Gaussian Elimination
More informationLU Factorization a 11 a 1 a 1n A = a 1 a a n (b) a n1 a n a nn L = l l 1 l ln1 ln 1 75 U = u 11 u 1 u 1n 0 u u n 0 u n...
.. Factorizations Reading: Trefethen and Bau (1997), Lecture 0 Solve the n n linear system by Gaussian elimination Ax = b (1) { Gaussian elimination is a direct method The solution is found after a nite
More informationSolving Dense Linear Systems I
Solving Dense Linear Systems I Solving Ax = b is an important numerical method Triangular system: [ l11 l 21 if l 11, l 22 0, ] [ ] [ ] x1 b1 = l 22 x 2 b 2 x 1 = b 1 /l 11 x 2 = (b 2 l 21 x 1 )/l 22 Chih-Jen
More informationLinear Algebraic Equations
Linear Algebraic Equations 1 Fundamentals Consider the set of linear algebraic equations n a ij x i b i represented by Ax b j with [A b ] [A b] and (1a) r(a) rank of A (1b) Then Axb has a solution iff
More informationLinear Systems of n equations for n unknowns
Linear Systems of n equations for n unknowns In many application problems we want to find n unknowns, and we have n linear equations Example: Find x,x,x such that the following three equations hold: x
More informationMTH 464: Computational Linear Algebra
MTH 464: Computational Linear Algebra Lecture Outlines Exam 2 Material Prof. M. Beauregard Department of Mathematics & Statistics Stephen F. Austin State University February 6, 2018 Linear Algebra (MTH
More informationFundamentals of Engineering Analysis (650163)
Philadelphia University Faculty of Engineering Communications and Electronics Engineering Fundamentals of Engineering Analysis (6563) Part Dr. Omar R Daoud Matrices: Introduction DEFINITION A matrix is
More informationSolution of Linear Systems
Solution of Linear Systems Parallel and Distributed Computing Department of Computer Science and Engineering (DEI) Instituto Superior Técnico May 12, 2016 CPD (DEI / IST) Parallel and Distributed Computing
More informationMATH 3511 Lecture 1. Solving Linear Systems 1
MATH 3511 Lecture 1 Solving Linear Systems 1 Dmitriy Leykekhman Spring 2012 Goals Review of basic linear algebra Solution of simple linear systems Gaussian elimination D Leykekhman - MATH 3511 Introduction
More informationConsider the following example of a linear system:
LINEAR SYSTEMS Consider the following example of a linear system: Its unique solution is x + 2x 2 + 3x 3 = 5 x + x 3 = 3 3x + x 2 + 3x 3 = 3 x =, x 2 = 0, x 3 = 2 In general we want to solve n equations
More informationChapter 1 Matrices and Systems of Equations
Chapter 1 Matrices and Systems of Equations System of Linear Equations 1. A linear equation in n unknowns is an equation of the form n i=1 a i x i = b where a 1,..., a n, b R and x 1,..., x n are variables.
More informationLinear Algebra (Review) Volker Tresp 2017
Linear Algebra (Review) Volker Tresp 2017 1 Vectors k is a scalar (a number) c is a column vector. Thus in two dimensions, c = ( c1 c 2 ) (Advanced: More precisely, a vector is defined in a vector space.
More informationScientific Computing: Dense Linear Systems
Scientific Computing: Dense Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 February 9th, 2012 A. Donev (Courant Institute)
More informationParallel Programming. Parallel algorithms Linear systems solvers
Parallel Programming Parallel algorithms Linear systems solvers Terminology System of linear equations Solve Ax = b for x Special matrices Upper triangular Lower triangular Diagonally dominant Symmetric
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationSparse BLAS-3 Reduction
Sparse BLAS-3 Reduction to Banded Upper Triangular (Spar3Bnd) Gary Howell, HPC/OIT NC State University gary howell@ncsu.edu Sparse BLAS-3 Reduction p.1/27 Acknowledgements James Demmel, Gene Golub, Franc
More informationLinear Algebra (Review) Volker Tresp 2018
Linear Algebra (Review) Volker Tresp 2018 1 Vectors k, M, N are scalars A one-dimensional array c is a column vector. Thus in two dimensions, ( ) c1 c = c 2 c i is the i-th component of c c T = (c 1, c
More informationCOURSE Numerical methods for solving linear systems. Practical solving of many problems eventually leads to solving linear systems.
COURSE 9 4 Numerical methods for solving linear systems Practical solving of many problems eventually leads to solving linear systems Classification of the methods: - direct methods - with low number of
More informationChapter 12 Block LU Factorization
Chapter 12 Block LU Factorization Block algorithms are advantageous for at least two important reasons. First, they work with blocks of data having b 2 elements, performing O(b 3 ) operations. The O(b)
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More information. =. a i1 x 1 + a i2 x 2 + a in x n = b i. a 11 a 12 a 1n a 21 a 22 a 1n. i1 a i2 a in
Vectors and Matrices Continued Remember that our goal is to write a system of algebraic equations as a matrix equation. Suppose we have the n linear algebraic equations a x + a 2 x 2 + a n x n = b a 2
More informationNumerical Linear Algebra
Numerical Linear Algebra By: David McQuilling; Jesus Caban Deng Li Jan.,31,006 CS51 Solving Linear Equations u + v = 8 4u + 9v = 1 A x b 4 9 u v = 8 1 Gaussian Elimination Start with the matrix representation
More informationSolving Linear Systems of Equations
1 Solving Linear Systems of Equations Many practical problems could be reduced to solving a linear system of equations formulated as Ax = b This chapter studies the computational issues about directly
More informationIntroduction. Vectors and Matrices. Vectors [1] Vectors [2]
Introduction Vectors and Matrices Dr. TGI Fernando 1 2 Data is frequently arranged in arrays, that is, sets whose elements are indexed by one or more subscripts. Vector - one dimensional array Matrix -
More informationA Review of Matrix Analysis
Matrix Notation Part Matrix Operations Matrices are simply rectangular arrays of quantities Each quantity in the array is called an element of the matrix and an element can be either a numerical value
More informationMatrix Computations: Direct Methods II. May 5, 2014 Lecture 11
Matrix Computations: Direct Methods II May 5, 2014 ecture Summary You have seen an example of how a typical matrix operation (an important one) can be reduced to using lower level BS routines that would
More information6. Iterative Methods for Linear Systems. The stepwise approach to the solution...
6 Iterative Methods for Linear Systems The stepwise approach to the solution Miriam Mehl: 6 Iterative Methods for Linear Systems The stepwise approach to the solution, January 18, 2013 1 61 Large Sparse
More informationApplied Numerical Linear Algebra. Lecture 8
Applied Numerical Linear Algebra. Lecture 8 1/ 45 Perturbation Theory for the Least Squares Problem When A is not square, we define its condition number with respect to the 2-norm to be k 2 (A) σ max (A)/σ
More informationMATRICES. a m,1 a m,n A =
MATRICES Matrices are rectangular arrays of real or complex numbers With them, we define arithmetic operations that are generalizations of those for real and complex numbers The general form a matrix of
More informationEngineering Computation
Engineering Computation Systems of Linear Equations_1 1 Learning Objectives for Lecture 1. Motivate Study of Systems of Equations and particularly Systems of Linear Equations. Review steps of Gaussian
More information5.1 Banded Storage. u = temperature. The five-point difference operator. uh (x, y + h) 2u h (x, y)+u h (x, y h) uh (x + h, y) 2u h (x, y)+u h (x h, y)
5.1 Banded Storage u = temperature u= u h temperature at gridpoints u h = 1 u= Laplace s equation u= h u = u h = grid size u=1 The five-point difference operator 1 u h =1 uh (x + h, y) 2u h (x, y)+u h
More informationMath 304 (Spring 2010) - Lecture 2
Math 304 (Spring 010) - Lecture Emre Mengi Department of Mathematics Koç University emengi@ku.edu.tr Lecture - Floating Point Operation Count p.1/10 Efficiency of an algorithm is determined by the total
More informationExample: Current in an Electrical Circuit. Solving Linear Systems:Direct Methods. Linear Systems of Equations. Solving Linear Systems: Direct Methods
Example: Current in an Electrical Circuit Solving Linear Systems:Direct Methods A number of engineering problems or models can be formulated in terms of systems of equations Examples: Electrical Circuit
More informationNumerical Linear Algebra
Numerical Linear Algebra R. J. Renka Department of Computer Science & Engineering University of North Texas 02/03/2015 Notation and Terminology R n is the Euclidean n-dimensional linear space over the
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationApril 26, Applied mathematics PhD candidate, physics MA UC Berkeley. Lecture 4/26/2013. Jed Duersch. Spd matrices. Cholesky decomposition
Applied mathematics PhD candidate, physics MA UC Berkeley April 26, 2013 UCB 1/19 Symmetric positive-definite I Definition A symmetric matrix A R n n is positive definite iff x T Ax > 0 holds x 0 R n.
More informationParallel Scientific Computing
IV-1 Parallel Scientific Computing Matrix-vector multiplication. Matrix-matrix multiplication. Direct method for solving a linear equation. Gaussian Elimination. Iterative method for solving a linear equation.
More informationApplied Linear Algebra
Applied Linear Algebra Gábor P. Nagy and Viktor Vígh University of Szeged Bolyai Institute Winter 2014 1 / 262 Table of contents I 1 Introduction, review Complex numbers Vectors and matrices Determinants
More informationSolving linear systems (6 lectures)
Chapter 2 Solving linear systems (6 lectures) 2.1 Solving linear systems: LU factorization (1 lectures) Reference: [Trefethen, Bau III] Lecture 20, 21 How do you solve Ax = b? (2.1.1) In numerical linear
More informationLU Factorization. Marco Chiarandini. DM559 Linear and Integer Programming. Department of Mathematics & Computer Science University of Southern Denmark
DM559 Linear and Integer Programming LU Factorization Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark [Based on slides by Lieven Vandenberghe, UCLA] Outline
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationScientific Computing: Solving Linear Systems
Scientific Computing: Solving Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 September 17th and 24th, 2015 A. Donev (Courant
More informationLinear Algebra and Matrix Inversion
Jim Lambers MAT 46/56 Spring Semester 29- Lecture 2 Notes These notes correspond to Section 63 in the text Linear Algebra and Matrix Inversion Vector Spaces and Linear Transformations Matrices are much
More informationMAT 610: Numerical Linear Algebra. James V. Lambers
MAT 610: Numerical Linear Algebra James V Lambers January 16, 2017 2 Contents 1 Matrix Multiplication Problems 7 11 Introduction 7 111 Systems of Linear Equations 7 112 The Eigenvalue Problem 8 12 Basic
More informationNumerical Analysis: Solving Systems of Linear Equations
Numerical Analysis: Solving Systems of Linear Equations Mirko Navara http://cmpfelkcvutcz/ navara/ Center for Machine Perception, Department of Cybernetics, FEE, CTU Karlovo náměstí, building G, office
More informationMA2501 Numerical Methods Spring 2015
Norwegian University of Science and Technology Department of Mathematics MA2501 Numerical Methods Spring 2015 Solutions to exercise set 3 1 Attempt to verify experimentally the calculation from class that
More information5 Solving Systems of Linear Equations
106 Systems of LE 5.1 Systems of Linear Equations 5 Solving Systems of Linear Equations 5.1 Systems of Linear Equations System of linear equations: a 11 x 1 + a 12 x 2 +... + a 1n x n = b 1 a 21 x 1 +
More informationIntroduction to Numerical Analysis
Université de Liège Faculté des Sciences Appliquées Introduction to Numerical Analysis Edition 2015 Professor Q. Louveaux Department of Electrical Engineering and Computer Science Montefiore Institute
More informationChapter 1: Systems of linear equations and matrices. Section 1.1: Introduction to systems of linear equations
Chapter 1: Systems of linear equations and matrices Section 1.1: Introduction to systems of linear equations Definition: A linear equation in n variables can be expressed in the form a 1 x 1 + a 2 x 2
More informationDeterminants. Chia-Ping Chen. Linear Algebra. Professor Department of Computer Science and Engineering National Sun Yat-sen University 1/40
1/40 Determinants Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra About Determinant A scalar function on the set of square matrices
More informationNumerical Linear Algebra
Numerical Linear Algebra Direct Methods Philippe B. Laval KSU Fall 2017 Philippe B. Laval (KSU) Linear Systems: Direct Solution Methods Fall 2017 1 / 14 Introduction The solution of linear systems is one
More informationReview of matrices. Let m, n IN. A rectangle of numbers written like A =
Review of matrices Let m, n IN. A rectangle of numbers written like a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn where each a ij IR is called a matrix with m rows and n columns or an
More informationOn the Skeel condition number, growth factor and pivoting strategies for Gaussian elimination
On the Skeel condition number, growth factor and pivoting strategies for Gaussian elimination J.M. Peña 1 Introduction Gaussian elimination (GE) with a given pivoting strategy, for nonsingular matrices
More informationHomework 2 Foundations of Computational Math 2 Spring 2019
Homework 2 Foundations of Computational Math 2 Spring 2019 Problem 2.1 (2.1.a) Suppose (v 1,λ 1 )and(v 2,λ 2 ) are eigenpairs for a matrix A C n n. Show that if λ 1 λ 2 then v 1 and v 2 are linearly independent.
More informationVECTORS, TENSORS AND INDEX NOTATION
VECTORS, TENSORS AND INDEX NOTATION Enrico Nobile Dipartimento di Ingegneria e Architettura Università degli Studi di Trieste, 34127 TRIESTE March 5, 2018 Vectors & Tensors, E. Nobile March 5, 2018 1 /
More informationA Divide-and-Conquer Algorithm for Functions of Triangular Matrices
A Divide-and-Conquer Algorithm for Functions of Triangular Matrices Ç. K. Koç Electrical & Computer Engineering Oregon State University Corvallis, Oregon 97331 Technical Report, June 1996 Abstract We propose
More informationV C V L T I 0 C V B 1 V T 0 I. l nk
Multifrontal Method Kailai Xu September 16, 2017 Main observation. Consider the LDL T decomposition of a SPD matrix [ ] [ ] [ ] [ ] B V T L 0 I 0 L T L A = = 1 V T V C V L T I 0 C V B 1 V T, 0 I where
More informationThe System of Linear Equations. Direct Methods. Xiaozhou Li.
1/16 The Direct Methods xiaozhouli@uestc.edu.cn http://xiaozhouli.com School of Mathematical Sciences University of Electronic Science and Technology of China Chengdu, China Does the LU factorization always
More informationMatrix Computations (Chapter 1-2)
Matrix Computations (Chapter 1-2) Mehrdad Sheikholeslami Department of Electrical Engineering University at Buffalo The Best Group Winter 216 Chapter 1 1.1 Basic Algorithms and Notation 1.2 Exploiting
More informationSOLVING LINEAR SYSTEMS
SOLVING LINEAR SYSTEMS We want to solve the linear system a, x + + a,n x n = b a n, x + + a n,n x n = b n This will be done by the method used in beginning algebra, by successively eliminating unknowns
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationApplied Mathematics 205. Unit II: Numerical Linear Algebra. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit II: Numerical Linear Algebra Lecturer: Dr. David Knezevic Unit II: Numerical Linear Algebra Chapter II.2: LU and Cholesky Factorizations 2 / 82 Preliminaries 3 / 82 Preliminaries
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationMatrix-Matrix Multiplication
Chapter Matrix-Matrix Multiplication In this chapter, we discuss matrix-matrix multiplication We start by motivating its definition Next, we discuss why its implementation inherently allows high performance
More information7 Matrix Operations. 7.0 Matrix Multiplication + 3 = 3 = 4
7 Matrix Operations Copyright 017, Gregory G. Smith 9 October 017 The product of two matrices is a sophisticated operations with a wide range of applications. In this chapter, we defined this binary operation,
More informationInstitute for Advanced Computer Studies. Department of Computer Science. Two Algorithms for the The Ecient Computation of
University of Maryland Institute for Advanced Computer Studies Department of Computer Science College Park TR{98{12 TR{3875 Two Algorithms for the The Ecient Computation of Truncated Pivoted QR Approximations
More informationLecture 2 Decompositions, perturbations
March 26, 2018 Lecture 2 Decompositions, perturbations A Triangular systems Exercise 2.1. Let L = (L ij ) be an n n lower triangular matrix (L ij = 0 if i > j). (a) Prove that L is non-singular if and
More informationLecture: Numerical Linear Algebra Background
1/36 Lecture: Numerical Linear Algebra Background http://bicmr.pku.edu.cn/~wenzw/opt-2017-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s and Michael Grant s lecture notes
More informationChapter 4 No. 4.0 Answer True or False to the following. Give reasons for your answers.
MATH 434/534 Theoretical Assignment 3 Solution Chapter 4 No 40 Answer True or False to the following Give reasons for your answers If a backward stable algorithm is applied to a computational problem,
More informationTopics. Vectors (column matrices): Vector addition and scalar multiplication The matrix of a linear function y Ax The elements of a matrix A : A ij
Topics Vectors (column matrices): Vector addition and scalar multiplication The matrix of a linear function y Ax The elements of a matrix A : A ij or a ij lives in row i and column j Definition of a matrix
More information