Conjugate Gradient (CG) Method
|
|
- Pauline Dickerson
- 6 years ago
- Views:
Transcription
1 Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous equations. 2 Minimization problem Consider the quadratic function given by Q (x) = 1 2 xt A x b T x, A R n n, b, x R n, (1) where A is a symmetric positive definite matrix. This quadratic function attains its minimum at the solution of the linear algebraic equation A x = b, which will be denoted by x. A variety of methods for solving this minimization problem have been derived. The general form of these algorithms is x k+1 = x k α k p k, k = 0, 1,..., (2) where p k is the direction vector at the kth iteration. The parameter α k is determined so as to minimize the objective function Q(x), that is, α k is the value satisfying Since in the expression Q (x k α k p k ) = min α Q (x k α p k ). (3) Q (x k α k p k ) = 1 2 (pt k A p k ) α 2 k p T k (A x k b) α k xt k (Ax k 2 b), (4) the coefficients of α 2 k is positive, this function attains its minimum at α k = p T k (A x k b)/(pt k A p k ), (5) for the fixed x k and p k. Next we consider the method for choosing the direction vectors p k. 3 Univariate iteration If we put p k = e i (the unit vector whose ith entry is and the others are 0) in (2), then with the α k given by (5) we have ( ) x k+1 = x k α k e i = x k 1 a a ij x k j b i e i, (6) ii j=1 1
2 since e T i A e i = a ii and e T i (A x b) = a ij x j b i. This means that the point x k+1 given by (6) is the minimum point which can be obtained by changing only the ith component of x k. Thus, the first n successive iterations of (6) with j=1 p 0 = e 1, p 1 = e 2,..., p n 1 = e n (7) is equivalent to the one iteration of the well-known Gauss Seidel method. 4 Steepest descent method On the other hand, if we choose p k as the gradient vector of Q(x) at the point x k, then we have the iteration method given by since where ( Q Q(x) =,..., Q ) T. x 1 x n This method is called the steepest descent method. 5 Conjugate direction method Let us consider the case that p k are chosen as x k+1 = x k α k (A x k b), (8) p k = Q(x k ) = A x k b, (9) p T i A p j = 0, i j. (10) When the direction vectors p k are taken in this way, the method is called the conjugate direction method. The vectors p i and p j satisfying (10) are said to be A-conjugate. The conjugate direction method converges in n steps, if no roundoff occurs. Here we show the convergence of the method. First of all, notice that (A x k+1 b) T p j = (A x k α k A p k b) T p j = (A x k b) T p j α k (A p k ) T p j = (A x k b) T p j, and as a result, we have (A x k+1 b) T p j = { (A x k b) T p j, j < k, 0, j = k. (11) 2
3 Thus we have (A x n b) T p j = (A x n 1 b) T p j = = (A x j+1 b) T p j = 0, j = 0,...,n 1. Therefore, the vector A x n b is orthogonal to the n linearly independent vectors p 0,...,p n 1, that is, A x n b = 0. Here we consider the method of generating the conjugate vectors p j. One might think of the method of selecting eigenvectors of A as p j (j = 1,..., n), since the two distinct eigenvectors of A, say p i, p j, satisfy p T i A p j = λ j pt i p j = 0. However, in general, the computation of all eigenvectors is far more expensive than solving linear equations, so that this method is not practical at all. The conjugate gradient method to be explained in the next section generates the conjugate direction vectors by using a relatively cheap procedure. 6 Conjugate gradient method Here we show the algorithm: CG-1 1: Choose a small value ε > 0 and an initial guess x 0 2: p 0 = r 0 = b A x 0, and compute (r 0, r 0 ) 3: k = 0 4: while r k / b ε do 5: α k = (r k, p k )/(p k, A p k ) 6: x k+1 = x k α k p k 7: r k+1 = r k + α k A p k 8: β k = (p k, A r k+1 )/(p k, A p k ) 9: p k+1 = r k+1 + β k p k 10: k = k : end while The vectors p k generated by the algorithm satisfy the conjugacy condition (p k, A p j ) = 0, 0 j < k, k = 1,..., n 1, (12) and the residuals r j (= b A x j ) satisfy the orthogonality condition(see Appendix) (r k, r j ) = 0, 0 j < k, k = 1,...,n 1. (13) Moreover, from the definitions of α j and β j we have (p j, r j+1 ) = (p j, r j ) + α j (p j, A p j ) = 0, j = 0, 1,... (14) (p j, A p j+1 ) = (p j, A r j+1 ) + β j (p j, A p j ) = 0, j = 0, 1,... (15) Using the relations above, we have a variant of CG-1: 3
4 CG-2 1: Choose a small value ε > 0 and an initial guess x 0 2: p 0 = r 0 = b A x 0, and compute (r 0, r 0 ) 3: k = 0 4: while r k / b ε do 5: α k = (r k, r k )/(p k, A p k ) 6: x k+1 = x k α k p k 7: r k+1 = r k + α k A p k 8: β k = (r k+1, r k+1 )/(r k, r k ) 9: p k+1 = r k+1 + β k p k 10: k = k : end while The number of the operations to be performed per step in this algorithm is shown in Table 1, which is a tremendous improvement over CG-1. Table 1.Number of the operations per step in CG-2. matrix-vector product 1 inner product 2 vector addition 3 scalar division 1 scalar comparison 1 4
5 Here we show the experimantal result for some dimensional equation ( xk x k log ) 10 x ( ) rk log 10 b k Relative residual and error versus iteration number k. References [1] James M. Ortega, Introduction to Parallel and Vector Solution of Linear Systems, 1989, Prenum Press, New York. [2] Gilbert W. Stewart, Afternotes Goes to Graduate School, 1997, SIAM, Philadelphia. 5
6 Problem 1. For the function P(x) given by P(x) = 1 2 (x x ) T A (x x ), show that the relation P(x) = Q(x) (x ) T Ax, where x is the solution of A x = b, and Q(x) is the function given by (1). 2. Prove equation (4). 3. Show that the equation (9). 4. Show that the vectors p 0,...,p n 1 given by (10) are linearly independent. 5. Show that equation (11) is valid. 6. Show that diagonal elements of symmetric positive definite matrices are positive. 7. For CG-1 make the table as Table Show that the two algorithms, CG-1 and CG-2, are equivalent. 9. Show that the matrix given by A = is symmetric positive definite Solve the equation A x = b by CG-1 and CG-2, and compare the CPU times for the case that A is the matrix given by above and the vector b is given by b = A u, where the dimension of the equation is n = and each component of u is a random number in the range (0, 1). 6
7 Appendix 1 Inner product Let x = (x 1,...,x n ) T and y = (y 1,...,y n ) T be real vectors in R n. Then we define the inner product of x and y by (x, y) = x T y = x i y i. (16) The inner product has the following properties: i=1 (x, y) = (y, x) (α x, y) = α (x, y), α is scalar (x, y + z) = (x, y) + (x, z) (x, x) 0, = holds, only if x = 0. (17) For nonzero vectors x, y R n, if (x, y) = 0 then the vectors x and y are said to be orthogonal. If the vector x is orthogonal with the linearly independent n vectors p i (i = 1,...,n) in R n, then x = 0, since from the assumption we have p T 1 p T 2. x = 0, p T n and the matrix in the left-hand side is nonsingular, so that x = 0. Next we show the set of n vectors p i (i = 1,..., n) which are orthogonal with each other are linearly independent. Since, if not so, that is, if p i (i = 1,...,n) are linearly dependent, then there exist constants c i (i = 1,...,n), not necessarily all zero, such that From this we have for all j c 1 p c n p n = 0. 0 = (c 1 p c n p n, p j ) = c j (p j, p j ), which means c j = 0. This contradicts the assumption. 2 Eigenvalues and eigenvectors of real symmetric matrices When a real matrix A satisfies a ij = a ji, that is, A T = A, the matrix A is called real symmetric matrix. The eigenvalues and eigenvectors of real symmetric matrices have the following properties: 7
8 1. Eigenvalues are real Let λ and x be any eigenvalue end eigenvector of A, respectively. Then we have (A x, x) = (λx, x) = λ( x, x), where denotes complex conjugate. On the other hand, from the property of inner product we have (A x, x) = ( x, Ax) = λ ( x, x). These two expressions mean λ = λ, since ( x, x) Orthogonality of eigenvectors Let λ i and λ j be the eigenvalues of A and assume λ i λ j. Then we have and since A is symmetric, which implies Thus we have (x i, x j ) = 0. (Ax i, x j ) = λ i (x i, x j ), (Ax i, x j ) = (x i, Ax j ) = λ j (x i, x j ), (λ i λ j ) (x i, x j ) = 0. 3 Quadratic form For a real symmetric matrix A = (a ij ) and a real vector x = (x 1,...,x n ) T, the quantity given by Q (x) = x T A x = (x, A x) = a ij x i x j (18) is called quadratic form. For any x 0, if Q (x) > 0 ( 0), then the matrix A is said to be positive (semi-)definite. Positive definite matrices have the following properties: 1. The diagonal elements of symmetric positive definite matrix is positive. This is clear from a ii = (e i, A e i ). 2. All the eigenvalues of symmetric positive definite matrix A are positive. For any x 0, if we transform x by y = T x, where T = (u 1,...,u n ), and u i is the eigenvector of A corresponding to λ i, then i=1 j=1 which means λ i > 0 for all i. 0 < x T A x = y T (T T A T) y = y T (T 1 A T) y = λ i yi 2, i=1 8
9 4 Theorem Here we show again the algorithm CG-1: 1: Choose a small value ε > 0 and initial guess x 0 2: p 0 = r 0 = b A x 0, and compute (r 0, r 0 ) 3: k = 0 4: while (r k, r k ) ε do 5: α k = (r k, p k )/(p k, A p k ) 6: x k+1 = x k α k p k 7: r k+1 = r k + α k A p k 8: β k = (p k, A r k+1 )/(p k, A p k ) 9: p k+1 = r k+1 + β k p k 10: k = k : end while Theorem Let A be an n n symmetric positive definite matrix, and x be the solution of the equation A x = b. Then the vectors p k generated by CG-1 algorithm satisfy and p k 0 unless x k = x. p T k A p j = 0, 0 j < k, k = 1,...,n 1, (19) r T k r j = 0, 0 j < k, k = 1,...,n 1, (20) Proof By the definitions of α j, β j and those of p j, r j+1, we have p T j r j+1 = pt j r j + α j p T j A p j = 0, j = 0, 1,..., (21) p T j A p j+1 = p T j A r j+1 + β j p T j A p j = 0, j = 0, 1,... (22) Here we assume, as an induction hypothesis, that (19) and (20) hold for some k < n 1. Then we must show these hold for k +1. Since p 0 = r 0, these hold for k = 1. For any j < k, using the 7th and 9th lines in CG-1, we have r T j r k+1 = rt j (r k + α k A p k ) = r T j r k + α k p T k A r j = r T j r k + α k p T k A (p j β j 1 p j 1 ) = 0, (23) since all three terms in the last line vanish by induction hypothesis. Moreover, we have from (21) and (22) r T k r k+1 = (p k β k 1 p k 1 ) T r k+1 = β k 1 p T k 1 r k+1 = β k 1 p T k 1 (r k + α k A p k ) = 0. 9
10 Thus we have shown that (20) is true also for k + 1. Next we have for any j < k p T j A p k+1 = pt j A (r k+1 + β k p k ) = pt j A r k+1 = α 1 j (r j+1 r j ) T r k+1 = 0, (24) if α j 0, which we will show later. Therefore (19) is true also for k + 1. Finally, we show α j 0. By the definition of p j and (22) we have Hence we have r T j p j = rt j (r j + β j 1 p j 1 ) = rt j r j. α j = r T j r j /p T j A p j. Therefore, if α j = 0, then r j = 0, that is, x j = x so that the process stops with x j. 5 Differentiation by vectors Here we define the differentiation of the scalar-valued function J(x) with respect to its vector argument by J (x) := J x 1 J x 2. J x n According to this definition, we have ( ) 1 Q (x) = 2 xt A x b T x = A x b,. (25) (26) since and Q(x) = 1 2 b T x = a ii x 2 i + a ij x i x j, i j i=1 b i x i, i=1 x k Q(x) = a kk x k + j k x k (b T x) = b k. a kj x j = a kj x j, j=1 10
Chapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011 Lecture Objectives describe when CG can be used to solve Ax
More informationNotes on Some Methods for Solving Linear Systems
Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms
More informationLecture Note 7: Iterative methods for solving linear systems. Xiaoqun Zhang Shanghai Jiao Tong University
Lecture Note 7: Iterative methods for solving linear systems Xiaoqun Zhang Shanghai Jiao Tong University Last updated: December 24, 2014 1.1 Review on linear algebra Norms of vectors and matrices vector
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationConjugate Gradient Method
Conjugate Gradient Method Hung M Phan UMass Lowell April 13, 2017 Throughout, A R n n is symmetric and positive definite, and b R n 1 Steepest Descent Method We present the steepest descent method for
More informationIterative methods for Linear System
Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and
More informationQuadratic forms. Here. Thus symmetric matrices are diagonalizable, and the diagonalization can be performed by means of an orthogonal matrix.
Quadratic forms 1. Symmetric matrices An n n matrix (a ij ) n ij=1 with entries on R is called symmetric if A T, that is, if a ij = a ji for all 1 i, j n. We denote by S n (R) the set of all n n symmetric
More informationThe conjugate gradient method
The conjugate gradient method Michael S. Floater November 1, 2011 These notes try to provide motivation and an explanation of the CG method. 1 The method of conjugate directions We want to solve the linear
More informationMath Introduction to Numerical Analysis - Class Notes. Fernando Guevara Vasquez. Version Date: January 17, 2012.
Math 5620 - Introduction to Numerical Analysis - Class Notes Fernando Guevara Vasquez Version 1990. Date: January 17, 2012. 3 Contents 1. Disclaimer 4 Chapter 1. Iterative methods for solving linear systems
More informationChap 3. Linear Algebra
Chap 3. Linear Algebra Outlines 1. Introduction 2. Basis, Representation, and Orthonormalization 3. Linear Algebraic Equations 4. Similarity Transformation 5. Diagonal Form and Jordan Form 6. Functions
More informationReview of Linear Algebra
Review of Linear Algebra Definitions An m n (read "m by n") matrix, is a rectangular array of entries, where m is the number of rows and n the number of columns. 2 Definitions (Con t) A is square if m=
More informationLinear Solvers. Andrew Hazel
Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction
More informationLinear Algebra: Matrix Eigenvalue Problems
CHAPTER8 Linear Algebra: Matrix Eigenvalue Problems Chapter 8 p1 A matrix eigenvalue problem considers the vector equation (1) Ax = λx. 8.0 Linear Algebra: Matrix Eigenvalue Problems Here A is a given
More informationLecture 11: CMSC 878R/AMSC698R. Iterative Methods An introduction. Outline. Inverse, LU decomposition, Cholesky, SVD, etc.
Lecture 11: CMSC 878R/AMSC698R Iterative Methods An introduction Outline Direct Solution of Linear Systems Inverse, LU decomposition, Cholesky, SVD, etc. Iterative methods for linear systems Why? Matrix
More informationConjugate Gradient Method
Conjugate Gradient Method Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 10, 2011 T.M. Huang (NTNU) Conjugate Gradient Method October 10, 2011 1 / 36 Outline 1 Steepest
More informationEcon Slides from Lecture 7
Econ 205 Sobel Econ 205 - Slides from Lecture 7 Joel Sobel August 31, 2010 Linear Algebra: Main Theory A linear combination of a collection of vectors {x 1,..., x k } is a vector of the form k λ ix i for
More informationCS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3
CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3 Felix Kwok February 27, 2004 Written Problems 1. (Heath E3.10) Let B be an n n matrix, and assume that B is both
More informationJordan Journal of Mathematics and Statistics (JJMS) 5(3), 2012, pp A NEW ITERATIVE METHOD FOR SOLVING LINEAR SYSTEMS OF EQUATIONS
Jordan Journal of Mathematics and Statistics JJMS) 53), 2012, pp.169-184 A NEW ITERATIVE METHOD FOR SOLVING LINEAR SYSTEMS OF EQUATIONS ADEL H. AL-RABTAH Abstract. The Jacobi and Gauss-Seidel iterative
More informationMath 5630: Conjugate Gradient Method Hung M. Phan, UMass Lowell March 29, 2019
Math 563: Conjugate Gradient Method Hung M. Phan, UMass Lowell March 29, 219 hroughout, A R n n is symmetric and positive definite, and b R n. 1 Steepest Descent Method We present the steepest descent
More informationJACOBI S ITERATION METHOD
ITERATION METHODS These are methods which compute a sequence of progressively accurate iterates to approximate the solution of Ax = b. We need such methods for solving many large linear systems. Sometimes
More informationLecture 7. Econ August 18
Lecture 7 Econ 2001 2015 August 18 Lecture 7 Outline First, the theorem of the maximum, an amazing result about continuity in optimization problems. Then, we start linear algebra, mostly looking at familiar
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationIterative techniques in matrix algebra
Iterative techniques in matrix algebra Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan September 12, 2015 Outline 1 Norms of vectors and matrices 2 Eigenvalues and
More informationLecture 8 : Eigenvalues and Eigenvectors
CPS290: Algorithmic Foundations of Data Science February 24, 2017 Lecture 8 : Eigenvalues and Eigenvectors Lecturer: Kamesh Munagala Scribe: Kamesh Munagala Hermitian Matrices It is simpler to begin with
More informationNotes on PCG for Sparse Linear Systems
Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova e-mail luca.bergamaschi@unipd.it webpage www.dmsa.unipd.it/
More informationthe method of steepest descent
MATH 3511 Spring 2018 the method of steepest descent http://www.phys.uconn.edu/ rozman/courses/m3511_18s/ Last modified: February 6, 2018 Abstract The Steepest Descent is an iterative method for solving
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationIterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009)
Iterative methods for Linear System of Equations Joint Advanced Student School (JASS-2009) Course #2: Numerical Simulation - from Models to Software Introduction In numerical simulation, Partial Differential
More informationCS 246 Review of Linear Algebra 01/17/19
1 Linear algebra In this section we will discuss vectors and matrices. We denote the (i, j)th entry of a matrix A as A ij, and the ith entry of a vector as v i. 1.1 Vectors and vector operations A vector
More informationJae Heon Yun and Yu Du Han
Bull. Korean Math. Soc. 39 (2002), No. 3, pp. 495 509 MODIFIED INCOMPLETE CHOLESKY FACTORIZATION PRECONDITIONERS FOR A SYMMETRIC POSITIVE DEFINITE MATRIX Jae Heon Yun and Yu Du Han Abstract. We propose
More informationEigenvalues and eigenvectors
Chapter 6 Eigenvalues and eigenvectors An eigenvalue of a square matrix represents the linear operator as a scaling of the associated eigenvector, and the action of certain matrices on general vectors
More informationConjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)
Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps
More informationLecture 11. Fast Linear Solvers: Iterative Methods. J. Chaudhry. Department of Mathematics and Statistics University of New Mexico
Lecture 11 Fast Linear Solvers: Iterative Methods J. Chaudhry Department of Mathematics and Statistics University of New Mexico J. Chaudhry (UNM) Math/CS 375 1 / 23 Summary: Complexity of Linear Solves
More informationNumerical Linear Algebra Homework Assignment - Week 2
Numerical Linear Algebra Homework Assignment - Week 2 Đoàn Trần Nguyên Tùng Student ID: 1411352 8th October 2016 Exercise 2.1: Show that if a matrix A is both triangular and unitary, then it is diagonal.
More informationLecture 10: Eigenvectors and eigenvalues (Numerical Recipes, Chapter 11)
Lecture 1: Eigenvectors and eigenvalues (Numerical Recipes, Chapter 11) The eigenvalue problem, Ax= λ x, occurs in many, many contexts: classical mechanics, quantum mechanics, optics 22 Eigenvectors and
More informationHere is an example of a block diagonal matrix with Jordan Blocks on the diagonal: J
Class Notes 4: THE SPECTRAL RADIUS, NORM CONVERGENCE AND SOR. Math 639d Due Date: Feb. 7 (updated: February 5, 2018) In the first part of this week s reading, we will prove Theorem 2 of the previous class.
More informationMath 489AB Exercises for Chapter 2 Fall Section 2.3
Math 489AB Exercises for Chapter 2 Fall 2008 Section 2.3 2.3.3. Let A M n (R). Then the eigenvalues of A are the roots of the characteristic polynomial p A (t). Since A is real, p A (t) is a polynomial
More information4.6 Iterative Solvers for Linear Systems
4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More informationThis property turns out to be a general property of eigenvectors of a symmetric A that correspond to distinct eigenvalues as we shall see later.
34 To obtain an eigenvector x 2 0 2 for l 2 = 0, define: B 2 A - l 2 I 2 = È 1, 1, 1 Î 1-0 È 1, 0, 0 Î 1 = È 1, 1, 1 Î 1. To transform B 2 into an upper triangular matrix, subtract the first row of B 2
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationLecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε,
2. REVIEW OF LINEAR ALGEBRA 1 Lecture 1 Review: Linear models have the form (in matrix notation) Y = Xβ + ε, where Y n 1 response vector and X n p is the model matrix (or design matrix ) with one row for
More informationIterative Methods. Splitting Methods
Iterative Methods Splitting Methods 1 Direct Methods Solving Ax = b using direct methods. Gaussian elimination (using LU decomposition) Variants of LU, including Crout and Doolittle Other decomposition
More informationSolving Linear Systems
Solving Linear Systems Iterative Solutions Methods Philippe B. Laval KSU Fall 207 Philippe B. Laval (KSU) Linear Systems Fall 207 / 2 Introduction We continue looking how to solve linear systems of the
More informationMATRIX ALGEBRA. or x = (x 1,..., x n ) R n. y 1 y 2. x 2. x m. y m. y = cos θ 1 = x 1 L x. sin θ 1 = x 2. cos θ 2 = y 1 L y.
as Basics Vectors MATRIX ALGEBRA An array of n real numbers x, x,, x n is called a vector and it is written x = x x n or x = x,, x n R n prime operation=transposing a column to a row Basic vector operations
More informationTopics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems
Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52 Conjugate
More informationProblems in Linear Algebra and Representation Theory
Problems in Linear Algebra and Representation Theory (Most of these were provided by Victor Ginzburg) The problems appearing below have varying level of difficulty. They are not listed in any specific
More information2. Matrix Algebra and Random Vectors
2. Matrix Algebra and Random Vectors 2.1 Introduction Multivariate data can be conveniently display as array of numbers. In general, a rectangular array of numbers with, for instance, n rows and p columns
More information7.2 Steepest Descent and Preconditioning
7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider
More informationCourse Notes: Week 4
Course Notes: Week 4 Math 270C: Applied Numerical Linear Algebra 1 Lecture 9: Steepest Descent (4/18/11) The connection with Lanczos iteration and the CG was not originally known. CG was originally derived
More informationCME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication.
CME342 Parallel Methods in Numerical Analysis Matrix Computation: Iterative Methods II Outline: CG & its parallelization. Sparse Matrix-vector Multiplication. 1 Basic iterative methods: Ax = b r = b Ax
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationConjugate-Gradient. Learn about the Conjugate-Gradient Algorithm and its Uses. Descent Algorithms and the Conjugate-Gradient Method. Qx = b.
Lab 1 Conjugate-Gradient Lab Objective: Learn about the Conjugate-Gradient Algorithm and its Uses Descent Algorithms and the Conjugate-Gradient Method There are many possibilities for solving a linear
More informationJim Lambers MAT 610 Summer Session Lecture 1 Notes
Jim Lambers MAT 60 Summer Session 2009-0 Lecture Notes Introduction This course is about numerical linear algebra, which is the study of the approximate solution of fundamental problems from linear algebra
More informationR-Linear Convergence of Limited Memory Steepest Descent
R-Linear Convergence of Limited Memory Steepest Descent Frank E. Curtis, Lehigh University joint work with Wei Guo, Lehigh University OP17 Vancouver, British Columbia, Canada 24 May 2017 R-Linear Convergence
More informationMatrix Algebra: Summary
May, 27 Appendix E Matrix Algebra: Summary ontents E. Vectors and Matrtices.......................... 2 E.. Notation.................................. 2 E..2 Special Types of Vectors.........................
More informationIntroduction to Matrix Algebra
Introduction to Matrix Algebra August 18, 2010 1 Vectors 1.1 Notations A p-dimensional vector is p numbers put together. Written as x 1 x =. x p. When p = 1, this represents a point in the line. When p
More informationThe Solution of Linear Systems AX = B
Chapter 2 The Solution of Linear Systems AX = B 21 Upper-triangular Linear Systems We will now develop the back-substitution algorithm, which is useful for solving a linear system of equations that has
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More information1. What is the determinant of the following matrix? a 1 a 2 4a 3 2a 2 b 1 b 2 4b 3 2b c 1. = 4, then det
What is the determinant of the following matrix? 3 4 3 4 3 4 4 3 A 0 B 8 C 55 D 0 E 60 If det a a a 3 b b b 3 c c c 3 = 4, then det a a 4a 3 a b b 4b 3 b c c c 3 c = A 8 B 6 C 4 D E 3 Let A be an n n matrix
More informationGradient Descent Methods
Lab 18 Gradient Descent Methods Lab Objective: Many optimization methods fall under the umbrella of descent algorithms. The idea is to choose an initial guess, identify a direction from this point along
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationB553 Lecture 5: Matrix Algebra Review
B553 Lecture 5: Matrix Algebra Review Kris Hauser January 19, 2012 We have seen in prior lectures how vectors represent points in R n and gradients of functions. Matrices represent linear transformations
More informationNumerical Linear Algebra
Numerical Linear Algebra The two principal problems in linear algebra are: Linear system Given an n n matrix A and an n-vector b, determine x IR n such that A x = b Eigenvalue problem Given an n n matrix
More informationThe Conjugate Gradient Algorithm
Optimization over a Subspace Conjugate Direction Methods Conjugate Gradient Algorithm Non-Quadratic Conjugate Gradient Algorithm Optimization over a Subspace Consider the problem min f (x) subject to x
More informationLecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University
Lecture 17 Methods for System of Linear Equations: Part 2 Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of
More information, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues
More informationMath 577 Assignment 7
Math 577 Assignment 7 Thanks for Yu Cao 1. Solution. The linear system being solved is Ax = 0, where A is a (n 1 (n 1 matrix such that 2 1 1 2 1 A =......... 1 2 1 1 2 and x = (U 1, U 2,, U n 1. By the
More informationFrom Stationary Methods to Krylov Subspaces
Week 6: Wednesday, Mar 7 From Stationary Methods to Krylov Subspaces Last time, we discussed stationary methods for the iterative solution of linear systems of equations, which can generally be written
More informationMAT Linear Algebra Collection of sample exams
MAT 342 - Linear Algebra Collection of sample exams A-x. (0 pts Give the precise definition of the row echelon form. 2. ( 0 pts After performing row reductions on the augmented matrix for a certain system
More informationELE/MCE 503 Linear Algebra Facts Fall 2018
ELE/MCE 503 Linear Algebra Facts Fall 2018 Fact N.1 A set of vectors is linearly independent if and only if none of the vectors in the set can be written as a linear combination of the others. Fact N.2
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method The minimization problem We are given a symmetric positive definite matrix R n n and a right hand side vector b R n We want to solve the linear system Find u R n such that
More informationMath 108b: Notes on the Spectral Theorem
Math 108b: Notes on the Spectral Theorem From section 6.3, we know that every linear operator T on a finite dimensional inner product space V has an adjoint. (T is defined as the unique linear operator
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationChapter 3 Transformations
Chapter 3 Transformations An Introduction to Optimization Spring, 2014 Wei-Ta Chu 1 Linear Transformations A function is called a linear transformation if 1. for every and 2. for every If we fix the bases
More information7.3 The Jacobi and Gauss-Siedel Iterative Techniques. Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.
7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 7.3 The Jacobi and Gauss-Siedel Iterative Techniques
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationM.A. Botchev. September 5, 2014
Rome-Moscow school of Matrix Methods and Applied Linear Algebra 2014 A short introduction to Krylov subspaces for linear systems, matrix functions and inexact Newton methods. Plan and exercises. M.A. Botchev
More informationAN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES
AN ELEMENTARY PROOF OF THE SPECTRAL RADIUS FORMULA FOR MATRICES JOEL A. TROPP Abstract. We present an elementary proof that the spectral radius of a matrix A may be obtained using the formula ρ(a) lim
More informationSTAT200C: Review of Linear Algebra
Stat200C Instructor: Zhaoxia Yu STAT200C: Review of Linear Algebra 1 Review of Linear Algebra 1.1 Vector Spaces, Rank, Trace, and Linear Equations 1.1.1 Rank and Vector Spaces Definition A vector whose
More information1 Linear Algebra Problems
Linear Algebra Problems. Let A be the conjugate transpose of the complex matrix A; i.e., A = A t : A is said to be Hermitian if A = A; real symmetric if A is real and A t = A; skew-hermitian if A = A and
More informationMath 411 Preliminaries
Math 411 Preliminaries Provide a list of preliminary vocabulary and concepts Preliminary Basic Netwon s method, Taylor series expansion (for single and multiple variables), Eigenvalue, Eigenvector, Vector
More informationEcon Slides from Lecture 8
Econ 205 Sobel Econ 205 - Slides from Lecture 8 Joel Sobel September 1, 2010 Computational Facts 1. det AB = det BA = det A det B 2. If D is a diagonal matrix, then det D is equal to the product of its
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationPositive Definite Matrix
1/29 Chia-Ping Chen Professor Department of Computer Science and Engineering National Sun Yat-sen University Linear Algebra Positive Definite, Negative Definite, Indefinite 2/29 Pure Quadratic Function
More informationSpectral Theorem for Self-adjoint Linear Operators
Notes for the undergraduate lecture by David Adams. (These are the notes I would write if I was teaching a course on this topic. I have included more material than I will cover in the 45 minute lecture;
More informationChapter 10 Conjugate Direction Methods
Chapter 10 Conjugate Direction Methods An Introduction to Optimization Spring, 2012 1 Wei-Ta Chu 2012/4/13 Introduction Conjugate direction methods can be viewed as being intermediate between the method
More informationLinear System Theory
Linear System Theory Wonhee Kim Lecture 4 Apr. 4, 2018 1 / 40 Recap Vector space, linear space, linear vector space Subspace Linearly independence and dependence Dimension, Basis, Change of Basis 2 / 40
More informationMath 5630: Iterative Methods for Systems of Equations Hung Phan, UMass Lowell March 22, 2018
1 Linear Systems Math 5630: Iterative Methods for Systems of Equations Hung Phan, UMass Lowell March, 018 Consider the system 4x y + z = 7 4x 8y + z = 1 x + y + 5z = 15. We then obtain x = 1 4 (7 + y z)
More informationCS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares
CS 542G: Robustifying Newton, Constraints, Nonlinear Least Squares Robert Bridson October 29, 2008 1 Hessian Problems in Newton Last time we fixed one of plain Newton s problems by introducing line search
More information