AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
|
|
- Randolph Reynolds
- 5 years ago
- Views:
Transcription
1 AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 34
2 Outline 1 Computing the SVD (NLA 31) 2 Sparse Storage Format 3 Direct Methods for Sparse Linear Systems (MC ) 4 Overview of Iterative Methods for Sparse Linear Systems Xiangmin Jiao Numerical Analysis I 2 / 34
3 SVD of A and Eigenvalues of A A Intuitive idea for computing SVD of A R m n : Form A A and compute its eigenvalue decomposition A A = V ΛV Let Σ = Λ, i.e., diag( λ 1, λ 2,..., λ n ) Solve system UΣ = AV to obtain U This method is efficient if m n. However, it may not be stable, especially for smaller singular values because of the squaring of the condition number For SVD of A, σk σ k = O(ɛ machine A ), where σ k and σ k denote the computed and exact kth singular value If computed from eigenvalue decomposition of A A, σ k σ k = O(ɛ machine A 2 /σ k ), which is problematic if σ k A If one is interested in only relatively large singular values, then computing eigenvalues of A A is not a problem. For general situations, a more stable algorithm is desired. Xiangmin Jiao Numerical Analysis I 3 / 34
4 A Different Reduction to Eigenvalue Problem Typical algorithm for computing SVD are similar to computation of eigenvalues [ ] 0 A Consider A C m n, then Hermitian matrix H = has A 0 eigenvalue decomposition [ ] [ ] [ ] V V V V Σ 0 H =, U U U U 0 Σ where A = UΣV gives the SVD. This approach is stable. In practice, such a reduction is done implicitly without forming the large matrix Typically done in two phases Xiangmin Jiao Numerical Analysis I 4 / 34
5 Two-Phase Method In the first phase, reduce to bidiagonal form by applying different orthogonal transformations on left and right, which involves O(mn 2 ) operations In the second phase, reduce to diagonal form using a variant of QR algorithm or divide-and-conquer algorithm, which involves O(n 2 ) operations for fixed precision We hereafter focus on the first phase Xiangmin Jiao Numerical Analysis I 5 / 34
6 Golub-Kahan Bidiagonalization Apply Householder reflectors on both left and right sides Work for Golub-Kahan bidiagonalization 4mn n3 flops Xiangmin Jiao Numerical Analysis I 6 / 34
7 Lawson-Hanson-Chan Bidiagonalization Speed up by first performing QR factorization on A Work for LHC bidiagonalization 2mn 2 + 2n 3 flops, which is advantageous if m 5 3 n Xiangmin Jiao Numerical Analysis I 7 / 34
8 Three-Step Bidiagonalization Hybrid approach: Apply QR at suitable time on submatrix with 5/3 aspect ratio Work for three-step bidiagonalization is 4mn n3 2 3 (m n)3 Xiangmin Jiao Numerical Analysis I 8 / 34
9 Comparison of Performance One-step (Golub-Kahan) Two-step (LHC) Three-step bidiagonalization Xiangmin Jiao Numerical Analysis I 9 / 34
10 Outline 1 Computing the SVD (NLA 31) 2 Sparse Storage Format 3 Direct Methods for Sparse Linear Systems (MC ) 4 Overview of Iterative Methods for Sparse Linear Systems Xiangmin Jiao Numerical Analysis I 10 / 34
11 Sparse Linear System Boundary value problems and implicit methods for time-dependent PDEs yield systems of linear algebraic equations to solve A matrix is sparse if it has relatively few nonzeros in its entries Sparsity can be exploited to use far less than O(n 2 ) storage and O(n 3 ) work required in standard approach to solving system with dense matrix, assuming matrix is n n Xiangmin Jiao Numerical Analysis I 11 / 34
12 Storage Format of Sparse Matrices Sparse-matrices are typically stored in special formats that store only nonzero entries, along with indices to identify their locations in matrix, such as compressed-row storage (CRS) compressed-column storage (CCS) block compressed row storage (BCRS) Banded matrices have their own special storage formats (such as Compressed Diagonal Storage (CDS)) See survey at Explicitly storing indices incurs additional storage overhead and makes arithmetic operations on nonzeros less efficient due to indirect addressing to access operands, so they are beneficial only for very sparse matrices Storage format can have big impact the effectiveness of different versions of same algorithm (with different ordering of loops) Besides direct methods, these storage formats are also important in implementing iterative and multigrid solvers Xiangmin Jiao Numerical Analysis I 12 / 34
13 Example of Compressed-Row Storage (CRS) Xiangmin Jiao Numerical Analysis I 13 / 34
14 Outline 1 Computing the SVD (NLA 31) 2 Sparse Storage Format 3 Direct Methods for Sparse Linear Systems (MC ) 4 Overview of Iterative Methods for Sparse Linear Systems Xiangmin Jiao Numerical Analysis I 14 / 34
15 Banded Linear Systems Cost of factorizing banded linear system depends on bandwidth For SPD n n matrix with semi-bandwidth s, total flop count of Cholesky factorization is about ns 2 For n n matrix with lower bandwidth p and upper bandwidth q, In A = LU (LU without pivoting), total flop count is about 2npq In PA = LU (LU with column pivoting), total flop count is about 2np(p + q) Banded matrices have their own special storage formats (such as Compressed Diagonal Storage (CDS)) Xiangmin Jiao Numerical Analysis I 15 / 34
16 Fill When applying LU or Cholesky factorization to general sparse matrix, taking linear combinations of rows or columns to annihilate unwanted nonzero entries can introduce new nonzeros into matrix locations that were initially zero Such new nonzeros, called fill or fill-in, must be stored and may themselves eventually need to be annihilated in order to obtain triangular factors Resulting triangular factors can be expected to contain at least as many nonzeros as original matrix and usually significant fill as well Xiangmin Jiao Numerical Analysis I 16 / 34
17 Sparse Cholesky Factorization In general, some heuristic algorithms are employed to reorder the matrix to reduce fills Amount of fill is sensitive to order in which rows and columns of matrix are processed, so basic problem in sparse factorization is reordering matrix to limit fill during factorization Exact minimization of fill is hard combinatorial problem (NP-complete), but heuristic algorithms such as minimum degree and nested dissection limit fill well for many types of problems For Cholesky factorization, both rows and columns are reordered Xiangmin Jiao Numerical Analysis I 17 / 34
18 Graph Model of Elimination Each step of factorization process corresponds to elimination of one node from graph Eliminating node causes its neighboring nodes to become connected to each other If any such neighbors were not already connected, then fill results (new edges in graph and new nonzeros in matrix) Commonly used reordering methods include Cuthill-McKee, approximate minimum degree ordering (AMD) and nested dissection Xiangmin Jiao Numerical Analysis I 18 / 34
19 Reordering to Reduce Bandwidth The Cuthill-McKee algorithm and reverse Cuthill-McKee algorithm The Cuthill-McKee algorithm is a variant of the breadth-first search algorithm on graphs. Starts with a peripheral node Generates levels R i for i = 1, 2,... until all nodes are exhausted The set R i+1 is created from set R i by listing all vertices adjacent to all nodes in R i Within each level, nodes are listed in increasing degree The reverse Cuthill McKee algorithm (RCM) reserves the resulting index numbers Xiangmin Jiao Numerical Analysis I 19 / 34
20 Approximate Minimum Degree Ordering Good heuristic for limiting fill is to eliminate first those nodes having fewest neighbors Number of neighbors is called degree of node, so heuristic is known as minimum degree At each step, select node of smallest degree for elimination, breaking ties arbitrarily After node has been eliminated, its neighbors become connected to each other, so degrees of some nodes may change Process is then repeated, with new node of minimum degree eliminated next, and so on until all nodes have been eliminated Xiangmin Jiao Numerical Analysis I 20 / 34
21 Minimum Degree Ordering, continued Cholesky factor suffers much less fill than with original ordering, and advantage grows with problem size Sophisticated versions of minimum degree are among most effective general-purpose orderings known Xiangmin Jiao Numerical Analysis I 21 / 34
22 Comparison of Different Orderings of Example Matrix Left: Nonzero pattern of matrix A. Right: Nonzero pattern of matrix R. Xiangmin Jiao Numerical Analysis I 22 / 34
23 Nested Dissection Ordering Nested dissection is based on divide-and-conquer First, small set of nodes is selected whose removal splits graph into two pieces of roughly equal size No node in either piece is connected to any node in other, so no fill occurs in either piece due to elimination of any node in the other Separator nodes are numbered last, then process is repeated recursively on each remaining piece of graph until all nodes have been numbered Xiangmin Jiao Numerical Analysis I 23 / 34
24 Nested Dissection Ordering Continued Dissection induces blocks of zeros in matrix that are automatically preserved during factorization Recursive nature of algorithm can be seen in hierarchical block structure of matrix, which would involve many more levels in larger problems Again, Cholesky factor suffers much less fill than with original ordering, and advantage grows with problem size Xiangmin Jiao Numerical Analysis I 24 / 34
25 Sparse Gaussian Elimination For Gaussian elimination, only columns are reordered Pivoting introduces additional fills in sparse Gaussian elimination Reordering may be done dynamically or statically The reverse Cuthill-McKee algorithm applied to A + A T may be used to reduce bandwidth Column approximate minimum-degree, may be employed to reorder matrix to reduce fills Xiangmin Jiao Numerical Analysis I 25 / 34
26 Nonzero pattern of A and L + U with column AMD ordering. Xiangmin Jiao Numerical Analysis I 26 / 34 Comparison of Different Orderings of Example Matrix Nonzero pattern of A and L + U with random ordering.
27 Comparison of Direct Methods Computational cost for Laplace equation on k k( k) grid with n unknowns method 2-D 3-D dense Cholesky k 6 n 3 k 9 n 3 banded Cholesky k 4 n 2 k 7 n 2.33 sparse Cholesky k 3 n 1.5 k 6 n 2 Reference: Michael T. Heath, Scientific Computing: An Introductory Survey, 2nd Edition, McGraw-Hill, Xiangmin Jiao Numerical Analysis I 27 / 34
28 Software of Sparse Solvers Additional implementation complexities include cache performance and parallelism It is advisable to use software packages MATLAB has its own sparse solvers if matrix is stored in sparse format Sparse matrix is created by using the sparse function Reordering is implemented as symrcm, symamd, and colamd For symmetric matrices, a good software is Taucs For non-symmetric matrices, a good software is SuperLU Xiangmin Jiao Numerical Analysis I 28 / 34
29 Outline 1 Computing the SVD (NLA 31) 2 Sparse Storage Format 3 Direct Methods for Sparse Linear Systems (MC ) 4 Overview of Iterative Methods for Sparse Linear Systems Xiangmin Jiao Numerical Analysis I 29 / 34
30 Direct vs. Iterative Methods Direct methods, or noniterative methods, compute the exact solution after a finite number of steps (in exact arithmetic) Example: Gaussian elimination, QR factorization Iterative methods produce a sequence of approximations x (1), x (2),... that hopefully converge to the true solution Example: Jacobi, Conjugate Gradient (CG), GMRES, BiCG, etc. Caution: The boundary between direct and iterative methods is vague sometimes Why use iterative methods (instead of direct methods)? may be faster than direct methods produce useful intermediate results handle sparse matrices more easily (needs only matrix-vector product) often are easier to implement on parallel computers Question: When not to use iterative methods? Xiangmin Jiao Numerical Analysis I 30 / 34
31 Two Classes of Iterative Methods Stationary iterative methods is a fixed point iteration obtained by matrix splitting Examples: Jacobi (for linear systems, not Jacobi iterations for eigenvalues), Gauss-Seidel, Successive Over-Relaxation (SOR) etc. Krylov subspace methods find optimal solution in Krylov subspace {b, Ab, A 2 b, A k b} Build subspace successively Example: Conjugate Gradient (CG), Generalized Minimum Residual (GMRES), BiCG, etc. We will focus on Krylov subspace methods Xiangmin Jiao Numerical Analysis I 31 / 34
32 Stationary Iterative Methods Stationary iterative methods find a splitting A = M N and iterates x k+1 = M 1 (Nx k + b) Suppose r k = b Ax k, we have x = x k + A 1 r k. Stationary iterative method approximates it by because x k+1 = x k + M 1 r k x k+1 = M 1 Nx k + M 1 b = M 1 Nx k + M 1 (r k + Ax k ) = M 1 (N + A)x k + M 1 r k = x k + M 1 r k A stationary iterative method is good if ρ(m 1 N) < 1, and M 1 is a good approximation to A 1 Xiangmin Jiao Numerical Analysis I 32 / 34
33 Stationary Iterative Methods Different choices of splitting will lead to various schemes Let A = L + D + U, where D is diagonal, L is strictly lower triangular, and U is strictly upper triangular Jacobi iteration: M = D, works well if A is diagonally dominant Gauss-Seidel: M = L + D, works well if A is SPD Successive Over-Relaxation (SOR): M = 1 ω D + L, where 1 ω < 2, converges quickly proper choice of ω Symmetric SOR: symmetric version of SOR These methods work for some problems, but they may converge slowly Nevertheless, stationary methods are important as preconditioners for Krylov-subspace methods and smoothers in multigrid methods (later) Xiangmin Jiao Numerical Analysis I 33 / 34
34 Stationary Iterative Methods Example For 2D Poisson equation, spectral radius of Jacobi iteration matrix is cos ( ) ( π n 1 O 1 ) n. Number of iterations required to achieve ɛ is 2 O(n 2 ln ɛ 1 ). After 5 Jacobi iterations on a Poisson equation, error decreases very slowly. Xiangmin Jiao Numerical Analysis I 34 / 34
AMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationScientific Computing with Case Studies SIAM Press, Lecture Notes for Unit VII Sparse Matrix
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 1: Direct Methods Dianne P. O Leary c 2008
More informationLecture 18 Classical Iterative Methods
Lecture 18 Classical Iterative Methods MIT 18.335J / 6.337J Introduction to Numerical Methods Per-Olof Persson November 14, 2006 1 Iterative Methods for Linear Systems Direct methods for solving Ax = b,
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 18 Outline
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 24: Preconditioning and Multigrid Solver Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 5 Preconditioning Motivation:
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More informationSOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS. Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA
1 SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA 2 OUTLINE Sparse matrix storage format Basic factorization
More informationNumerical Methods in Matrix Computations
Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationNumerical Linear Algebra
Numerical Linear Algebra Decompositions, numerical aspects Gerard Sleijpen and Martin van Gijzen September 27, 2017 1 Delft University of Technology Program Lecture 2 LU-decomposition Basic algorithm Cost
More informationProgram Lecture 2. Numerical Linear Algebra. Gaussian elimination (2) Gaussian elimination. Decompositions, numerical aspects
Numerical Linear Algebra Decompositions, numerical aspects Program Lecture 2 LU-decomposition Basic algorithm Cost Stability Pivoting Cholesky decomposition Sparse matrices and reorderings Gerard Sleijpen
More informationScientific Computing
Scientific Computing Direct solution methods Martin van Gijzen Delft University of Technology October 3, 2018 1 Program October 3 Matrix norms LU decomposition Basic algorithm Cost Stability Pivoting Pivoting
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 11 Partial Differential Equations Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002.
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 3: Positive-Definite Systems; Cholesky Factorization Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 11 Symmetric
More information9. Iterative Methods for Large Linear Systems
EE507 - Computational Techniques for EE Jitkomut Songsiri 9. Iterative Methods for Large Linear Systems introduction splitting method Jacobi method Gauss-Seidel method successive overrelaxation (SOR) 9-1
More informationAPPLIED NUMERICAL LINEAR ALGEBRA
APPLIED NUMERICAL LINEAR ALGEBRA James W. Demmel University of California Berkeley, California Society for Industrial and Applied Mathematics Philadelphia Contents Preface 1 Introduction 1 1.1 Basic Notation
More informationAM205: Assignment 2. i=1
AM05: Assignment Question 1 [10 points] (a) [4 points] For p 1, the p-norm for a vector x R n is defined as: ( n ) 1/p x p x i p ( ) i=1 This definition is in fact meaningful for p < 1 as well, although
More informationLecture 17: Iterative Methods and Sparse Linear Algebra
Lecture 17: Iterative Methods and Sparse Linear Algebra David Bindel 25 Mar 2014 Logistics HW 3 extended to Wednesday after break HW 4 should come out Monday after break Still need project description
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 19: More on Arnoldi Iteration; Lanczos Iteration Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis I 1 / 17 Outline 1
More informationContents. Preface... xi. Introduction...
Contents Preface... xi Introduction... xv Chapter 1. Computer Architectures... 1 1.1. Different types of parallelism... 1 1.1.1. Overlap, concurrency and parallelism... 1 1.1.2. Temporal and spatial parallelism
More information5.1 Banded Storage. u = temperature. The five-point difference operator. uh (x, y + h) 2u h (x, y)+u h (x, y h) uh (x + h, y) 2u h (x, y)+u h (x h, y)
5.1 Banded Storage u = temperature u= u h temperature at gridpoints u h = 1 u= Laplace s equation u= h u = u h = grid size u=1 The five-point difference operator 1 u h =1 uh (x + h, y) 2u h (x, y)+u h
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationSolving linear systems (6 lectures)
Chapter 2 Solving linear systems (6 lectures) 2.1 Solving linear systems: LU factorization (1 lectures) Reference: [Trefethen, Bau III] Lecture 20, 21 How do you solve Ax = b? (2.1.1) In numerical linear
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationCME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication.
CME342 Parallel Methods in Numerical Analysis Matrix Computation: Iterative Methods II Outline: CG & its parallelization. Sparse Matrix-vector Multiplication. 1 Basic iterative methods: Ax = b r = b Ax
More informationLecture 9: Numerical Linear Algebra Primer (February 11st)
10-725/36-725: Convex Optimization Spring 2015 Lecture 9: Numerical Linear Algebra Primer (February 11st) Lecturer: Ryan Tibshirani Scribes: Avinash Siravuru, Guofan Wu, Maosheng Liu Note: LaTeX template
More informationPreface to the Second Edition. Preface to the First Edition
n page v Preface to the Second Edition Preface to the First Edition xiii xvii 1 Background in Linear Algebra 1 1.1 Matrices................................. 1 1.2 Square Matrices and Eigenvalues....................
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 12: Gaussian Elimination and LU Factorization Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Gaussian Elimination
More informationLecture 11: CMSC 878R/AMSC698R. Iterative Methods An introduction. Outline. Inverse, LU decomposition, Cholesky, SVD, etc.
Lecture 11: CMSC 878R/AMSC698R Iterative Methods An introduction Outline Direct Solution of Linear Systems Inverse, LU decomposition, Cholesky, SVD, etc. Iterative methods for linear systems Why? Matrix
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 7: More on Householder Reflectors; Least Squares Problems Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 15 Outline
More informationStabilization and Acceleration of Algebraic Multigrid Method
Stabilization and Acceleration of Algebraic Multigrid Method Recursive Projection Algorithm A. Jemcov J.P. Maruszewski Fluent Inc. October 24, 2006 Outline 1 Need for Algorithm Stabilization and Acceleration
More informationSparse Linear Systems. Iterative Methods for Sparse Linear Systems. Motivation for Studying Sparse Linear Systems. Partial Differential Equations
Sparse Linear Systems Iterative Methods for Sparse Linear Systems Matrix Computations and Applications, Lecture C11 Fredrik Bengzon, Robert Söderlund We consider the problem of solving the linear system
More informationLab 1: Iterative Methods for Solving Linear Systems
Lab 1: Iterative Methods for Solving Linear Systems January 22, 2017 Introduction Many real world applications require the solution to very large and sparse linear systems where direct methods such as
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 9 Minimizing Residual CG
More informationCME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 0
CME 302: NUMERICAL LINEAR ALGEBRA FALL 2005/06 LECTURE 0 GENE H GOLUB 1 What is Numerical Analysis? In the 1973 edition of the Webster s New Collegiate Dictionary, numerical analysis is defined to be the
More information6. Iterative Methods for Linear Systems. The stepwise approach to the solution...
6 Iterative Methods for Linear Systems The stepwise approach to the solution Miriam Mehl: 6 Iterative Methods for Linear Systems The stepwise approach to the solution, January 18, 2013 1 61 Large Sparse
More informationMAA507, Power method, QR-method and sparse matrix representation.
,, and representation. February 11, 2014 Lecture 7: Overview, Today we will look at:.. If time: A look at representation and fill in. Why do we need numerical s? I think everyone have seen how time consuming
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Iteration basics Notes for 2016-11-07 An iterative solver for Ax = b is produces a sequence of approximations x (k) x. We always stop after finitely many steps, based on some convergence criterion, e.g.
More informationSparse Matrices and Iterative Methods
Sparse Matrices and Iterative Methods K. 1 1 Department of Mathematics 2018 Iterative Methods Consider the problem of solving Ax = b, where A is n n. Why would we use an iterative method? Avoid direct
More informationClassical iterative methods for linear systems
Classical iterative methods for linear systems Ed Bueler MATH 615 Numerical Analysis of Differential Equations 27 February 1 March, 2017 Ed Bueler (MATH 615 NADEs) Classical iterative methods for linear
More informationLinear Algebra. Brigitte Bidégaray-Fesquet. MSIAM, September Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble.
Brigitte Bidégaray-Fesquet Univ. Grenoble Alpes, Laboratoire Jean Kuntzmann, Grenoble MSIAM, 23 24 September 215 Overview 1 Elementary operations Gram Schmidt orthonormalization Matrix norm Conditioning
More informationComputational Methods. Systems of Linear Equations
Computational Methods Systems of Linear Equations Manfred Huber 2010 1 Systems of Equations Often a system model contains multiple variables (parameters) and contains multiple equations Multiple equations
More informationIterative methods for Linear System
Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and
More informationAlgebraic Multigrid as Solvers and as Preconditioner
Ò Algebraic Multigrid as Solvers and as Preconditioner Domenico Lahaye domenico.lahaye@cs.kuleuven.ac.be http://www.cs.kuleuven.ac.be/ domenico/ Department of Computer Science Katholieke Universiteit Leuven
More informationSolving PDEs with CUDA Jonathan Cohen
Solving PDEs with CUDA Jonathan Cohen jocohen@nvidia.com NVIDIA Research PDEs (Partial Differential Equations) Big topic Some common strategies Focus on one type of PDE in this talk Poisson Equation Linear
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization /36-725
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: proximal gradient descent Consider the problem min g(x) + h(x) with g, h convex, g differentiable, and h simple
More informationSolving PDEs with Multigrid Methods p.1
Solving PDEs with Multigrid Methods Scott MacLachlan maclachl@colorado.edu Department of Applied Mathematics, University of Colorado at Boulder Solving PDEs with Multigrid Methods p.1 Support and Collaboration
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Reduction to Hessenberg and Tridiagonal Forms; Rayleigh Quotient Iteration Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME MS&E 38 (CME 338 Large-Scale Numerical Optimization Course description Instructor: Michael Saunders Spring 28 Notes : Review The course teaches
More informationBlock Bidiagonal Decomposition and Least Squares Problems
Block Bidiagonal Decomposition and Least Squares Problems Åke Björck Department of Mathematics Linköping University Perspectives in Numerical Analysis, Helsinki, May 27 29, 2008 Outline Bidiagonal Decomposition
More informationFrom Stationary Methods to Krylov Subspaces
Week 6: Wednesday, Mar 7 From Stationary Methods to Krylov Subspaces Last time, we discussed stationary methods for the iterative solution of linear systems of equations, which can generally be written
More informationTopics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems
Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52 Conjugate
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 1: Course Overview & Matrix-Vector Multiplication Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 20 Outline 1 Course
More informationConjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)
Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationIterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009)
Iterative methods for Linear System of Equations Joint Advanced Student School (JASS-2009) Course #2: Numerical Simulation - from Models to Software Introduction In numerical simulation, Partial Differential
More informationA Method for Constructing Diagonally Dominant Preconditioners based on Jacobi Rotations
A Method for Constructing Diagonally Dominant Preconditioners based on Jacobi Rotations Jin Yun Yuan Plamen Y. Yalamov Abstract A method is presented to make a given matrix strictly diagonally dominant
More informationLinear Solvers. Andrew Hazel
Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction
More informationPreconditioning Techniques Analysis for CG Method
Preconditioning Techniques Analysis for CG Method Huaguang Song Department of Computer Science University of California, Davis hso@ucdavis.edu Abstract Matrix computation issue for solve linear system
More informationIncomplete Cholesky preconditioners that exploit the low-rank property
anapov@ulb.ac.be ; http://homepages.ulb.ac.be/ anapov/ 1 / 35 Incomplete Cholesky preconditioners that exploit the low-rank property (theory and practice) Artem Napov Service de Métrologie Nucléaire, Université
More informationThe Lanczos and conjugate gradient algorithms
The Lanczos and conjugate gradient algorithms Gérard MEURANT October, 2008 1 The Lanczos algorithm 2 The Lanczos algorithm in finite precision 3 The nonsymmetric Lanczos algorithm 4 The Golub Kahan bidiagonalization
More informationNumerical linear algebra
Numerical linear algebra Purdue University CS 51500 Fall 2017 David Gleich David F. Gleich Call me Prof Gleich Dr. Gleich Please not Hey matrix guy! Huda Nassar Call me Huda Ms. Huda Please not Matrix
More informationIntroduction to Scientific Computing
(Lecture 5: Linear system of equations / Matrix Splitting) Bojana Rosić, Thilo Moshagen Institute of Scientific Computing Motivation Let us resolve the problem scheme by using Kirchhoff s laws: the algebraic
More informationSparsity-Preserving Difference of Positive Semidefinite Matrix Representation of Indefinite Matrices
Sparsity-Preserving Difference of Positive Semidefinite Matrix Representation of Indefinite Matrices Jaehyun Park June 1 2016 Abstract We consider the problem of writing an arbitrary symmetric matrix as
More informationBoundary Value Problems - Solving 3-D Finite-Difference problems Jacob White
Introduction to Simulation - Lecture 2 Boundary Value Problems - Solving 3-D Finite-Difference problems Jacob White Thanks to Deepak Ramaswamy, Michal Rewienski, and Karen Veroy Outline Reminder about
More informationIterative Methods and Multigrid
Iterative Methods and Multigrid Part 3: Preconditioning 2 Eric de Sturler Preconditioning The general idea behind preconditioning is that convergence of some method for the linear system Ax = b can be
More informationSolving Ax = b, an overview. Program
Numerical Linear Algebra Improving iterative solvers: preconditioning, deflation, numerical software and parallelisation Gerard Sleijpen and Martin van Gijzen November 29, 27 Solving Ax = b, an overview
More informationParallel Iterative Methods for Sparse Linear Systems. H. Martin Bücker Lehrstuhl für Hochleistungsrechnen
Parallel Iterative Methods for Sparse Linear Systems Lehrstuhl für Hochleistungsrechnen www.sc.rwth-aachen.de RWTH Aachen Large and Sparse Small and Dense Outline Problem with Direct Methods Iterative
More informationOUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative methods ffl Krylov subspace methods ffl Preconditioning techniques: Iterative methods ILU
Preconditioning Techniques for Solving Large Sparse Linear Systems Arnold Reusken Institut für Geometrie und Praktische Mathematik RWTH-Aachen OUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative
More informationSolving Large Nonlinear Sparse Systems
Solving Large Nonlinear Sparse Systems Fred W. Wubs and Jonas Thies Computational Mechanics & Numerical Mathematics University of Groningen, the Netherlands f.w.wubs@rug.nl Centre for Interdisciplinary
More informationMotivation: Sparse matrices and numerical PDE's
Lecture 20: Numerical Linear Algebra #4 Iterative methods and Eigenproblems Outline 1) Motivation: beyond LU for Ax=b A little PDE's and sparse matrices A) Temperature Equation B) Poisson Equation 2) Splitting
More informationA Robust Preconditioned Iterative Method for the Navier-Stokes Equations with High Reynolds Numbers
Applied and Computational Mathematics 2017; 6(4): 202-207 http://www.sciencepublishinggroup.com/j/acm doi: 10.11648/j.acm.20170604.18 ISSN: 2328-5605 (Print); ISSN: 2328-5613 (Online) A Robust Preconditioned
More informationReview of matrices. Let m, n IN. A rectangle of numbers written like A =
Review of matrices Let m, n IN. A rectangle of numbers written like a 11 a 12... a 1n a 21 a 22... a 2n A =...... a m1 a m2... a mn where each a ij IR is called a matrix with m rows and n columns or an
More informationLU Factorization. Marco Chiarandini. DM559 Linear and Integer Programming. Department of Mathematics & Computer Science University of Southern Denmark
DM559 Linear and Integer Programming LU Factorization Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark [Based on slides by Lieven Vandenberghe, UCLA] Outline
More informationTopics in Numerical Linear Algebra
Topics in Numerical Linear Algebra Axel Ruhe 1 January 17, 2005 1 Royal Institute of Technology, SE-10044 Stockholm, Sweden (ruhe@kth.se). Material given to course in Applied Numerical Analysis. 2 Preface
More informationUsing semiseparable matrices to compute the SVD of a general matrix product/quotient
Using semiseparable matrices to compute the SVD of a general matrix product/quotient Marc Van Barel Yvette Vanberghen Paul Van Dooren Report TW 508, November 007 n Katholieke Universiteit Leuven Department
More informationDepartment of Mathematics California State University, Los Angeles Master s Degree Comprehensive Examination in. NUMERICAL ANALYSIS Spring 2015
Department of Mathematics California State University, Los Angeles Master s Degree Comprehensive Examination in NUMERICAL ANALYSIS Spring 2015 Instructions: Do exactly two problems from Part A AND two
More informationComputation of eigenvalues and singular values Recall that your solutions to these questions will not be collected or evaluated.
Math 504, Homework 5 Computation of eigenvalues and singular values Recall that your solutions to these questions will not be collected or evaluated 1 Find the eigenvalues and the associated eigenspaces
More informationParallel Singular Value Decomposition. Jiaxing Tan
Parallel Singular Value Decomposition Jiaxing Tan Outline What is SVD? How to calculate SVD? How to parallelize SVD? Future Work What is SVD? Matrix Decomposition Eigen Decomposition A (non-zero) vector
More informationLinear algebra & Numerical Analysis
Linear algebra & Numerical Analysis Eigenvalues and Eigenvectors Marta Jarošová http://homel.vsb.cz/~dom033/ Outline Methods computing all eigenvalues Characteristic polynomial Jacobi method for symmetric
More informationNumerical Solution Techniques in Mechanical and Aerospace Engineering
Numerical Solution Techniques in Mechanical and Aerospace Engineering Chunlei Liang LECTURE 3 Solvers of linear algebraic equations 3.1. Outline of Lecture Finite-difference method for a 2D elliptic PDE
More informationLecture 13 Stability of LU Factorization; Cholesky Factorization. Songting Luo. Department of Mathematics Iowa State University
Lecture 13 Stability of LU Factorization; Cholesky Factorization Songting Luo Department of Mathematics Iowa State University MATH 562 Numerical Analysis II ongting Luo ( Department of Mathematics Iowa
More informationIterative Methods for Sparse Linear Systems
Iterative Methods for Sparse Linear Systems Luca Bergamaschi e-mail: berga@dmsa.unipd.it - http://www.dmsa.unipd.it/ berga Department of Mathematical Methods and Models for Scientific Applications University
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 16: Rayleigh Quotient Iteration Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 10 Solving Eigenvalue Problems All
More informationAMS 209, Fall 2015 Final Project Type A Numerical Linear Algebra: Gaussian Elimination with Pivoting for Solving Linear Systems
AMS 209, Fall 205 Final Project Type A Numerical Linear Algebra: Gaussian Elimination with Pivoting for Solving Linear Systems. Overview We are interested in solving a well-defined linear system given
More informationIncomplete Factorization Preconditioners for Least Squares and Linear and Quadratic Programming
Incomplete Factorization Preconditioners for Least Squares and Linear and Quadratic Programming by Jelena Sirovljevic B.Sc., The University of British Columbia, 2005 A THESIS SUBMITTED IN PARTIAL FULFILMENT
More informationNumerical Mathematics
Alfio Quarteroni Riccardo Sacco Fausto Saleri Numerical Mathematics Second Edition With 135 Figures and 45 Tables 421 Springer Contents Part I Getting Started 1 Foundations of Matrix Analysis 3 1.1 Vector
More informationPreconditioning Techniques for Large Linear Systems Part III: General-Purpose Algebraic Preconditioners
Preconditioning Techniques for Large Linear Systems Part III: General-Purpose Algebraic Preconditioners Michele Benzi Department of Mathematics and Computer Science Emory University Atlanta, Georgia, USA
More information1. Fast Iterative Solvers of SLE
1. Fast Iterative Solvers of crucial drawback of solvers discussed so far: they become slower if we discretize more accurate! now: look for possible remedies relaxation: explicit application of the multigrid
More informationAn advanced ILU preconditioner for the incompressible Navier-Stokes equations
An advanced ILU preconditioner for the incompressible Navier-Stokes equations M. ur Rehman C. Vuik A. Segal Delft Institute of Applied Mathematics, TU delft The Netherlands Computational Methods with Applications,
More informationFINE-GRAINED PARALLEL INCOMPLETE LU FACTORIZATION
FINE-GRAINED PARALLEL INCOMPLETE LU FACTORIZATION EDMOND CHOW AND AFTAB PATEL Abstract. This paper presents a new fine-grained parallel algorithm for computing an incomplete LU factorization. All nonzeros
More informationMath 471 (Numerical methods) Chapter 3 (second half). System of equations
Math 47 (Numerical methods) Chapter 3 (second half). System of equations Overlap 3.5 3.8 of Bradie 3.5 LU factorization w/o pivoting. Motivation: ( ) A I Gaussian Elimination (U L ) where U is upper triangular
More informationComputational Methods. Eigenvalues and Singular Values
Computational Methods Eigenvalues and Singular Values Manfred Huber 2010 1 Eigenvalues and Singular Values Eigenvalues and singular values describe important aspects of transformations and of data relations
More information