Notes on PCG for Sparse Linear Systems
|
|
- Clinton Martin
- 5 years ago
- Views:
Transcription
1 Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova webpage berga 1 The method of the steepest descent Let us now focus on symmetric positive definite matrices. Let us define a function φ R n R φ(x) = 1 2 (x h)t A(x h) = 1 2 et Ae φ(x) 0 x 0 (by definition of spd matrix). Moreover φ(x) = 0 x = h = A 1 b. Summarizing, φ has a unique minimum x = h, so that minimizing φ can be regarded as a method to solve the linear system Ax = b. Let us suppose to have an approximate solution x k. φ(x k ) = constant represents a quadratic form in n dimensions. Geometrically it is an hyperellipsoid in the n-dimensional subspace whose geometric center is the solution h. The method of the steepest descent is aimed to go from x k toward the center of the hyperellipsoid going in the direction of the gradient. The gradient of φ at the point x k is φ(x k ) = 1 ( 1 2 et k Ae k = 2 xt k Ax k x T b + 1 ) 2 ht h = Ax k b = r k Since the direction of the gradient vector is outward, the wanted direction coincides with the residual of our linear system in the current approximation x k. Once we have obtained the search direction (WITHOUT knowing the exact solution) we can write our next approximation as x k+1 = x k + α k r k where α k is a suitable constant to be determined so as to minimize function φ(x) along the search direction r k. This is an easier problem as compared to the initial one. We have to perform a monodimensional optimization of function φ. Formally α k = argmin t φ(x k + tr k ) 1
2 1 The method of the steepest descent 2 Let us write in detail φ(x k + tr k ): it turns out to be a second order polynomial in the variable t. φ(x k + tr k ) = (x k + tr k h) T A(x k + tr k h) = (x k + tr k h) T (Ax k + tar k b) = (x k + tr k h) T (tar k r k ) = (tr k e k ) T (tar k r k ) Being r T k Ar k > 0 the minimum of φ is attained for Alternatively:solve φ t 1.1 Steepest descent. Algorithm = t 2 r T k Ar k t ( r T k r k + e T k Ar k ) + h T r k = t 2 r T k Ar k 2tr T k r k + const. rt k r k α k t = r T k Ar k = 0 (use the chain rule to differentiate). The k + 1th iterate of the method of the steepest descent can be written as r k = b Ax k α k = rt k r k r T k Ar k x k+1 = x k + α k r k Note that this iteration costs mainly two matrix-vector products. We can even do better by observing that from x k+1 = x k + α k r k it follows that r k+1 = b Ax k+1 = b Ax k Aα k r k = r k α k Ar k. This allows us to write a more efficient algorithm for the SD method. Consecutive residuals r k+1, r k are orthogonal (proof by Exercise). The error e k+1 is A orthogonal to the search direction r k. (proof by Exercise). Algorithm. Steepest Descent Input: x 0, A, b, k max, tol r 0 = b Ax 0, k = 0 while r k > tol b and k < k max do 1. z = Ar k 2. α k = rt k r k z T r k 3. x k+1 = x k + α k r k 4. r k+1 = r k α k z 5. k = k + 1 end while
3 1 The method of the steepest descent Convergence of the SD Theorem 1.1. (proof not given) φ(xk ) = e T k Ae k = e k A,2 µ k e 0 A,2, where µ = κ(a) 1 κ(a) + 1 Order the eigenvalues of A as 0 < λ n λ 2 λ 1, then κ(a) = λ 1. λ n Again when dealing with system arising from PDEs, κ(a) may be very large. It is useful to compute an estimate of the number of iteration needed to gain p digits in the approximation of the solution: e k A e 0 A 10 p by solving the inequality ( ) k κ(a) 1 10 p κ(a) + 1 Taking logarithms and using the first order Taylor approximation log κ(a) 1 κ(a) + 1 2, we obtain κ(a) + 1 log 10 k p (κ(a) + 1) 2 ( ) 1 0 Exercise Solve the system Ax = 0 with A = using the SD method. with initial 0 9 guess x 0 = (9, 1) T. Find x 1, x 2. prove that x k = 0.8 k (9, 1) T. Verify the convergence bounds of previous frame ( ) Solve the system Ax = 0 with A = using the SD method. with initial guess x = (9, 1/9) T. Find x 1, x Slow convergence: consecutive residuals are orthogonal. It depends on relation between e 0 and the eigenvectors of A. Note that (exercise) if e 0 is an eigenvector then x 1 = h.
4 2 The Conjugate Gradient (CG) Method Faster convergence: why? 2 The Conjugate Gradient (CG) Method The Conjugate Gradient method can be viewed as an improvement of the SD method. We start from a similar recurrence relation between consecutive iterates. x k+1 = x k + α k p k Now we allow our search direction to be different from the gradient, namely p 0 = r 0 p k+1 = r k+1 + β k p k, k 0 Differently from SD we require that the error at step k+1 is also A-orthogonal to the previous direction p k. To this aim we impose that wich gives 0 = p T k Ap k+1 = p T k A (r k+1 + β k p k ) = p T k Ar k+1 + β k p T k Ap k, β k = rt k+1 Ap k p T k Ap k From this definition of β k we have that p T k Ap k+1 = 0 i.e. p k+1 and p k are A-orthogonal. Exercise 2.1. Prove the following results: a) As in the SD case choice of α k is aimed to minimize φ(x k+1 ) = φ(x k + α k p k ) giving the following formula rt k p k α k = p T k Ap k b) Consecutive residuals as in the SD satisfy a recurrence relation c) Consecutive residuals are orthogonal namely r k+1 = r k α k Ap k r T k+1r k = 0.
5 2 The Conjugate Gradient (CG) Method The finite termination property of CG Theorem 2.1. The CG algorithm generates the exact solution h in at most n steps. Moreover the sequences e k, r k and p k satisfy: Proof. Assume that {r j } 0, j k. Basis of the induction. e T k+1ap j = 0, p T k+1ap j = 0, r T k+1r j = 0, j k p T 1 Ap 0 = 0 by the definition of β 0 r T 0 r 1 = r T 0 (r 0 α 0 Ar 0 ) = r T 0 r 0 α 0 r T 0 Ar 0 = 0 since α 0 = rt 0 p 0 p T 0 Ap = rt 0 r 0 0 r T 0 Ar 0 e T 1 Ap 0 = r T 1 r 0 = 0. We assume as inductive hypothesis that e T k Ap j = 0, p T k Ap j = 0, r T k r j = 0, j k 1 Note that the first relation also implies that r T k p j = 0, j k 1. Then for j k 1 we have e T k+1ap j = (e k + α k p k ) T Ap j = e T k Ap j + α k p T k Ap j = 0 (= r T k+1p j = 0); r T k+1r j = r T ( ) k+1 pj β j 1 p j 1 = r T k+1 p j β j 1 r T k p j 1 = 0 p T k+1ap j = (r k+1 + β k p k ) T Ap j = r T k+1ap j + β k p T k Ap j = r T 1 k+1 (r j+1 r j ) = α j = 1 α j ( r T k+1 r j+1 r T k+1r j ) = 0 (by Exercise 2.1 c) and the previous relation) It remains to prove that e T k+1ap k = 0, p T k+1ap k = 0, r T k+1r k = 0. The relation p T k+1 Ap k = 0 immediately follow from the definition of β k. Moreover r T k+1 r k = 0 by Exercise 2.1. Finally by definition of α k. e T k+1ap k = (e k + α k p k ) T Ap k = e T k Ap k + α k p T k Ap k = r T k p k + α k p T k Ap k = 0, 2.2 Consequences of Theorem 2.1 From the Theorem we have that r n = 0, p n = 0 which implies x n = h. This is the finite termination property. CG stops (in exact arithmetics!!) after at most n iterations. The error at step k + 1 can be defined in terms of the initial error and a polynomial of degree k in matrix A. In fact the following result holds: Theorem 2.2. with Q k (0) = 1. k+1 e k+1 = e 0 + c j A j e 0 = (I + AP k (A))e 0 = Q k+1 (A)e 0 j=1
6 2 The Conjugate Gradient (CG) Method 6 Proof. By induction. e 1 = e 0 + α 0 p 0 = e 0 + α 0 r 0 = e 0 α 0 Ae 0 = Q 1 (A)e 0 With Q 1 (x) = 1 α 0 x, Q 1 (0) = 1. Suppose e j = Q j (A)e 0, j k, then e k+1 = e k + α k p k = e k + α k ( rk + β k 1 p k 1 ) = by setting Q k+1 (A) = (1 α k A)Q k (A) + c 1 Q k 1 (A). = e k α k Ae k + β k 1 α k 1 (e k e k 1 ) = = (1 α k A)e k + c 1 e k 1 = Q k+1 e 0 Minimization property ( of the CG. Actually, e k+1 minimizes the A norm on all the vectors which k+1 ) can be written as e 0 + j=1 c ja j e 0. This is the outcome of the following Theorem 2.3. The sequences of errors produce by the CG method satisfies: e k+1 A = min (I + AP k (A))e 0 A P k Proof. From we have, recursively that x k+1 = x k + α k p k, x k+1 = x 0 + with S k = [ p 0 p 1 p k ] k α k p k = x 0 + S k z = e k+1 = e 0 + S k z. j=0 α 0 z = α 1. α k We have to prove that the A norm of the error is the smallest possible over all choices of vector z. Hence g(z) = e k+1 2 A = e T k+1ae k+1 = e T 0 Ae 0 + 2e T 0 AS k z + z T S T k AS k z = e 0 2 A 2r T 0 S k z + z T Λ k z where the (i, j) entry of Λ k is given by p T i Ap j, which is zero if i j. Matrix Λ k is therefore diagonal. Function g(z) takes its minimum value in ẑ such that g(z) = 0. Componentwise g = 2Λ k z 2S T k r 0 ; g = 0 z = Λ 1 k ST k r 0 α k = z k = pt k r 0 p T k Ap, k which are exactly the α s provided by the CG algorithm, if we finally observe that k 1 r k = r k 1 α k 1 Ap k 1 = = r 0 α j Ap j, hence p T k r k = p T k r 0. j=0
7 2 The Conjugate Gradient (CG) Method Other useful relations. 1. p T k r k = r T k r k. From e T k Ap j = 0 immediately follows r T k p j = 0 and hence 2. r T k Ap k = p T k Ap k (left by Exercise). p T k r k = ( r k + β k 1 p k 1 ) T rk = r T k r k 3. Combining 1. and 2. we obtain an alternative definition of α k : rt k p k α k = p T k Ap k = rt k r k r T k Ap k 4. Alternative (and more numerically stable) formulation for scalars α k, β k p T k Ap k = p T k 1 α k (r k r k+1 ) = 1 α k r T k r k = α k = pt k Ap k r T k r k Therefore r T k+1ap k = r T 1 k+1 (r k r k+1 ) = 1 r T α k α k+1r k+1 k β k = rt k+1 Ap k p T k Ap k = rt k+1 r k+1 r T k r k After these rewritings we are able to show the final, and most efficient, version of the Conjugate Gradient method: Algorithm: Conjugate Gradient Input: x 0, A, b, k max, tol r 0 = p 0 = b Ax 0, k = 0 ρ 0 = r T 0 r 0 while r k > tol b and k < k max do 1. z = Ap k 2. α k = ρ k z T p k = rt k r k p T k Ap k 3. x k+1 = x k + α k p k 4. r k+1 = r k α k z 5. ρ k+1 = r T k+1 r k+1 6. β k = ρ k+1 ρ k = rt k+1 r k+1 r T k r k 7. p k+1 = r k+1 + β k p k 8. k = k + 1 end while
8 2 The Conjugate Gradient (CG) Method Convergence rate of the CG Method Finite termination property does not have practical utility (on a computer no termination due to loss of orthogonality) + n iterations are too many. Can we measure how fast do iterate approach the solution? Start from: e k A = min P k 1 (I + AP k 1 (A))e 0 A We define Q k (A) = I + AP k 1 (A) (that is Q k (0) = 1). Now being A normal we can orthogonally diagonalize it as A = UΛU T and since A is also spd it exists A = Udiag ( λ 1,, λ n ) U T. Then e k 2 A = e T k Ae k = ( Ae k ) T ( Ae k ) = Ae k 2 e k A = min Q k = min Q k AQ k (A)e 0 = min Q k max Q k(λ i ) e 0 A i [1,n] UQ k (Λ)U T Ae 0 min Q k (Λ) e 0 A Q k If λ n and λ 1 (the smallest and largest eigenvalue, respectively) are known then a bound of the error norm reduction can be given in terms of shifted and scaled Chebyshev polynomials which satisfy a so called minimax condition. Theorem 2.4. The A-norm of the error at the k-th CG iterate satissfies: ( ) k e k A κ(a) 1 2. e 0 A κ(a) + 1 Proof. See Luenberger, 1969 and many other books on iteriatve methods. It is useful to compute an estimate of the number of iteration needed to gain p digits in the approximation of the solution: ( ) k e k A κ(a) 1 10 p by solving the inequality 10 p e 0 A κ(a) + 1 Taking logarithms and using the first order Taylor approximation log κ(a) 1 κ(a) κ(a) + 1, we log 10 obtain k p ( κ(a) + 1) 2 Example. Optimality of Chebyshev polynomials. Suppose eigenvalues of A are 1, 2, 10. Scaled and shifted 2nd order Chebyshev polynomial on [1, 10] has the following form T 2 (x) = 2 ( ) 2x ( ) 11 2, max{ T 2 (λ i ) } = T 2 (1) (or T 2 (10) ) = It satisfies T 2 (x) = arg min max P 2(x). What we really want is something that minimizes the P 2(x) x [1,10] absolute value of the polynomial at the eigenvalues not on the whole spectral interval. So our Q 2 (x) = 1 + ax + bx 2. We can find optimal a, b by imposing the conditions Q 2 (1) = Q 2 (2) = Q 2 (10) i.e. solving the following linear system (Q 2 (1) =) 1 + a + b = γ (Q 2 (2) =) 1 + 2a + 4b = γ (Q 2 (10) =) a + 100b = γ
9 3 Preconditioning 9 which gives a = 33 42, b = 1 14, γ = max Q 2(λ i ) = Note that, as known, the Chebyshev polynomial T 2 (x) provides the minimum of the maximum value 1.5 Chebyshev polynomial 1 (33/42)x + x*x/ in the interval [1, 10] while the computed polynomial Q 2 (x), though achieving larger values within the interval minimizes max{ Q 2 (1), Q 2 (2), Q 2 (10) }. Summing up: e Chebyshev polynomials A optimal polynomial e 0 A = Computed ratio (A = diag(1, 2, 10), b = (1, 2, 10) T, x 0 = (0, 0, 0) T.) 3 Preconditioning Fast convergence of the CG method is guaranteed if κ(a) is sufficiently small or, at least a sort of partial condition number λ s+1 λ n p 1 very large and separated eigenvalues. To obtain a faster convergence we would like either to reduce the condition number is small in the case matrix A has s n very small and p n or to obtain a clustering of the majority of the eigenvalues around a single value In both cases we are trying to round the ellipsoid. Preconditioning a linear system is nothing but (pre)multiplying the system by a nonsingular matrix, which we call M 1, thus yielding an equivalent linear system. M 1 Ax = M 1 b (1)
10 3 Preconditioning 10 The preconditioner M 1, should be also easily invertible, or in other words, should allow a fast solution of a linear system of the form My = c. The CG method is applicable only if system matrix is spd. However, the preconditioned system is no longer symmetric. We now show that system (1) is equivalent to another one in which the coefficient matrix is spd provided that also M is. If M is spd then m admits a real square root that we call X so that M = XX. System (1) can be written as X 1 X 1 Ax = X 1 X 1 b or, premultiplying by X, X 1 Ax = X 1 b Let us now define: x = Xx, b = X 1 b, B = X 1 AX 1 ; we can rewrite the system as Bx = b (2) with B spd matrix (also note that B and M 1 A, being similar, have the same eigenvalues). We then can apply the CG method to the transformed system 2 whose coefficient matrix is now SPD. Unfortunately we are not able to compute explicitly the square root of a large and sparse matrix, so computation of X can not be done in practice. However, the Preconditioned Conjugate Gradient method can be efficiently implemented as we will see later. Let us now write in detail the PCG in the new variables. x k = Xx k, p k = Xp k, r k = b Bx k = X 1 b X 1 AX 1 Xx k = X 1 r k α k = r T k p k p T k Bp k r T k = X 1 Xp k p T = α k XX 1 AX 1 k Xp k β k = r T k+1r k+1 = rt k+1 M 1 r k+1 r T k r r T k k M 1 r k x k+1 = x k + α k p k = Xx k+1 = Xx k + α k Xp k = x k+1 = x k + α k p k r k+1 = r k α k Bp k = X 1 r k+1 = X 1 r k α k X 1 AX 1 Xp k = r k+1 = r k α k Ap k p k+1 = r k+1 + β p k = Xp k+1 = X 1 r k+1 + β Xp k = p k+1 = M 1 r k+1 + β p k It turns out that every CG equation in the new variable can be rewritten in terms of the old variable. The result is that matrix X completely disappears in the algorithm. To be able to implement the PCG method we must compute efficiently the operation M 1 r k+1, which is needed to compute the residual and the scalar β. 3.1 Computational issues A single iteration of PCG is made of 1. 3 scalar products + 3 daxpys 1 = 6 O(n) operations 2. 1 matrix vector product 3. 1 application of preconditioner (solution of Mg k+1 = r k+1 ). 1 A daxpy operation is of the form x = x + αy, with x, p vectors anf α a scalar.
11 4 Classical preconditioners 11 Algorithm. Preconditioned Conjugate Gradient (PCG) In Input: x 0, A, M, b, k max, tol r 0 = b Ax 0, p 0 = M 1 r 0, k = 0, ρ 0 = r T 0 M 1 r 0 while r k > tol b and k < k max do 1. z k = Ap k 2. α k = pt k r k z T k p k 3. x k+1 = x k + α k p k 4. r k+1 = r k α k z k 5. g k+1 = M 1 r k+1, ρ k+1 = r T k+1 g k+1 6. β k = rt k+1 M 1 r k+1 r T k M 1 r k = ρ k+1 ρ k 7. p k+1 = g k+1 + β k p k 8. k = k + 1 end while 4 Classical preconditioners How to choose the preconditioner? It should mediate between two opposite requirements: 1. Reducing the number of iterations (by clustering eigenvalues). 2. Allowing cheap solution of system Mg k+1 = r k+1. Jacobi preconditioner M = D where D is the (spd) diagonal matrix satisfying d ii = a ii, i = 1,..., n Computation of M 1 r k+1 costs O(n) operations and hence is very cheap. Does this preconditioner improve the condition number of the system? This is the case e.g. for diagonally dominant matrices. In general we can expect only a slight reduction in the iteration number using this preconditioner. Incomplete Cholesky factorization M = L L T where L is an approximation (sparsification) of the triangular factor obtained by the Cholesky factorization. Since we need to solve a system with matrix M we require L be as sparse as possible. A common choice (which leads to the IC(0) preconditioner) is to allow that L has nonzeros only in those position where also A is different from zero, that is a ij = 0 = l ij 0 Fixed a subset S [1,..., n] [1,..., n] of positions in the matrix then the factor L is initialized to the lower triangular factor of A and then updated according to the following procedure:
12 5 Numerical Example 12 for k=1:n A(k,k)=sqrt(A(k,k)) for i=k+1:n if (A(i,k) ~=0) A(i,k)=A(i,k)/A(k,k) end end for j=k+1:n for i=j:n if (A(i,j) ~=0) A(i,j)=A(i,j)-A(i,k)*A(j,k) end end end end The incomplete Cholesky factorization may not be properly defined even if A is spd (Existence results are known for M-matrices and H-matrices). Breakdown may occur when A kk = 0. However in practice very seldom IC(0) breaks down. Here the keyword is sparsity since computation of g k+1 = M 1 r k+1 is performed by a sequential solution of two sparse triangular linear systems: from g k+1 = (LL T ) 1 r k+1 we obtain LL T g k+1 = r k+1 and hence: Ly = r k+1 L T g k+1 = y When using IC(0) the computational cost of a single iteration is roughly doubled with respect to the non-preconditioned CG. Moreover the condition number of the preconditioned system may decrease of a few orders of magnitude thus producing an effective acceleration of the iterative method. Approximate inverse preconditioners They build M 1 = Z T Z where Z is a sparse lower triangular matrix approximating L 1. In the AINV approach (Benzi et al.) Z is constructed by means of a sparse Gram-Schmidt orthogonalization. 5 Numerical Example A is the result of a Mixed Finite Element discretization of a 2D diffusion equation. n = 77120, nz = , κ 2 (A) = Results on a Pentium 1GHz (PCG Fortran implementation) with tol=10 7 : Preconditioner iterations cpu cpu per iter Jacobi Cholesky Unfortunately, not even IC(0) avoids dependence of condition number on h. Preconditioners which do this are multigrid/domain decomposition based preconditioners.
13 5 Numerical Example An important example: the diffusion equation A number of different physical processes can be described by the diffusion equation: u( x, t) t (K u( x, t)) = f( x), x Ω, t (0, T ) Here u might represent the temperature distribution at time t in an object Ω, to which an external heat source f is applied. The positive definite matrix K( x) is the thermal conductivity of the material. To determine the temperature at time t, we need to know an initial temperature distribution u( x, 0) and some boundary conditions, of the form u( x, t) = 0 on Ω 1 u( x, t) n = 0 on Ω 2 where n is the outward normal to the domain Ω. Two standard method are commonly used for obtaining approximate solutions to such partial differential equations: The Finite Difference method. 1. The region Ω is divided into small pieces 2. At each point of a grid on Ω the derivatives are replaced by difference quotients as a linear combination of the unknown values at the gridpoints. The Finite Element method ff 1. The region Ω is divided into small pieces 2. The differential equation is projected onto a finite dimensional space and its solution is find so as to minimize the residual onto another finite dimensional space. A simplified form of the diffusion equation is the Laplace equation whose solution gives the steady state solution. 2 u x u y 2 = 0 x T=0 o C i,j y T=100 o C T=0 o C i-1,j i,j i+1,j Ap- i,j-1 o T=100 C proximations of second derivatives can be made by 2nd order Taylor expansion as: 2 u x 2 = u i+1,j 2u i,j + u i 1,j x 2, 2 u y 2 = u i,j+1 2u i,j + u i,j 1 y 2
14 5 Numerical Example 14 Assuming x = y the Laplace equation on node i, j is approximated by u i+1,j + u i 1,j + u i,j+1 + u i,j 1 4u i,j = 0 The result is a system with as many rows as the interior nodes of the grid. Every interior node produces a linear equation with the values of temperature at nodes as unknowns. Temperature on boundary nodes is known and contributes to the rhs Discretized diffusion equation u 1 u 2 u 3 u 4 u 5 u 6 u 7 u 8 u 9 = After discretization in the space, the PDE is converted in a system of ODEs like P u t = Hu + b where P = I in FD discretization. Stiffness matrix H is symmetric definite negative, capacity or mass matrix P is symmetric positive definite. Finally discretization in time transforms the system of ODEs to a linear system of size equal to the number of gridpoints to be solved at each time step. Time discretization divides the time interval (0, T ] into subintervals of length t i so that solution in time is computed at t k = k i=1 i and uses Finite Differences to approximate the time derivative. For example applying the implicit Euler method (u u(t k+1) u(t k ) t k ). (I + t k H)u(t k+1 ) = t k P u(t k ) + t k b The 2D FD discretization of the Laplace equation takes a general form (in the case x = y = h): S I 4 1 H = 1 I S I h where S = I S I I S 1 4 Eigenvalues of S and hence those of H are explicitly known: λ j,k (H) = 4 ( ) jπh h 2 sin2 + 4 ( ) kπh 2 h 2 sin2, j, k = 1,..., n x = 1 2 h 1 Theorem 5.1. The smallest and largest eigenvalues of H behave like: λ n = 2π 2 + O(h 2 ), λ 1 = 8h 2 + O(1).
15 5 Numerical Example 15 Proof. λ n = 8 ( ) πh h 2 sin2 = 8 2 h 2 (hπ/2 + O(h3 )) 2 = 2π 2 + O(h 2 ). λ 1 = 8 ( ) nx πh h 2 sin2 = 8 ( π 2 h 2 sin2 2 πh ) = 2 = 8 ( ) πh h 2 cos2 = 8 2 h 2 (1 O(h2 )) 2 = 8h 2 + O(1). Corollary 5.2. The condition number of H behaves like Corollary 5.3. κ(h) = 4 π 2 h 2 + O(1) (κ(h) = 2d π d h 2 when Ω R d ) The number of iteration of the CG for solving Hx = b is proportional to n 1D discretizations h 1 = n 2D discretizations. 3 n 3D discretizations 5.3 Numerical Results Finite Difference discretization of the steady-state Laplace equation on a unitary square domain with n x = n y = {100, 200, 400, 800}. The system has been solved by the Conjugate Gradient method with no preconditioner (CG) or accelerated by the Incomplete Cholesky preconditioner provided by MatLAB L = ichol(a,struct( type, ict, droptol,tol)); base on drop tolerances of tol = 10 2, n x N CG PCG (10 2 ) PCG (10 3 ) Tab. 1: Number of iterations of CG and PCG with the IC factorization Note from the table the dipendence of the number of iterations on h 1 N.
Iterative Methods for Sparse Linear Systems
Iterative Methods for Sparse Linear Systems Luca Bergamaschi e-mail: berga@dmsa.unipd.it - http://www.dmsa.unipd.it/ berga Department of Mathematical Methods and Models for Scientific Applications University
More informationApplied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationConjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)
Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps
More informationNotes on Some Methods for Solving Linear Systems
Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms
More informationCS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3
CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3 Felix Kwok February 27, 2004 Written Problems 1. (Heath E3.10) Let B be an n n matrix, and assume that B is both
More informationTopics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems
Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52 Conjugate
More informationITERATIVE METHODS BASED ON KRYLOV SUBSPACES
ITERATIVE METHODS BASED ON KRYLOV SUBSPACES LONG CHEN We shall present iterative methods for solving linear algebraic equation Au = b based on Krylov subspaces We derive conjugate gradient (CG) method
More informationLecture # 20 The Preconditioned Conjugate Gradient Method
Lecture # 20 The Preconditioned Conjugate Gradient Method We wish to solve Ax = b (1) A R n n is symmetric and positive definite (SPD). We then of n are being VERY LARGE, say, n = 10 6 or n = 10 7. Usually,
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationNumerical Methods I Non-Square and Sparse Linear Systems
Numerical Methods I Non-Square and Sparse Linear Systems Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 MATH-GA 2011.003 / CSCI-GA 2945.003, Fall 2014 September 25th, 2014 A. Donev (Courant
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationM.A. Botchev. September 5, 2014
Rome-Moscow school of Matrix Methods and Applied Linear Algebra 2014 A short introduction to Krylov subspaces for linear systems, matrix functions and inexact Newton methods. Plan and exercises. M.A. Botchev
More informationIterative techniques in matrix algebra
Iterative techniques in matrix algebra Tsung-Ming Huang Department of Mathematics National Taiwan Normal University, Taiwan September 12, 2015 Outline 1 Norms of vectors and matrices 2 Eigenvalues and
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 23: GMRES and Other Krylov Subspace Methods; Preconditioning Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 18 Outline
More information1 Conjugate gradients
Notes for 2016-11-18 1 Conjugate gradients We now turn to the method of conjugate gradients (CG), perhaps the best known of the Krylov subspace solvers. The CG iteration can be characterized as the iteration
More informationLecture 18 Classical Iterative Methods
Lecture 18 Classical Iterative Methods MIT 18.335J / 6.337J Introduction to Numerical Methods Per-Olof Persson November 14, 2006 1 Iterative Methods for Linear Systems Direct methods for solving Ax = b,
More informationA Chebyshev-based two-stage iterative method as an alternative to the direct solution of linear systems
A Chebyshev-based two-stage iterative method as an alternative to the direct solution of linear systems Mario Arioli m.arioli@rl.ac.uk CCLRC-Rutherford Appleton Laboratory with Daniel Ruiz (E.N.S.E.E.I.H.T)
More informationConjugate Gradient Method
Conjugate Gradient Method direct and indirect methods positive definite linear systems Krylov sequence spectral analysis of Krylov sequence preconditioning Prof. S. Boyd, EE364b, Stanford University Three
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More information7.2 Steepest Descent and Preconditioning
7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider
More informationPreconditioning Techniques Analysis for CG Method
Preconditioning Techniques Analysis for CG Method Huaguang Song Department of Computer Science University of California, Davis hso@ucdavis.edu Abstract Matrix computation issue for solve linear system
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More information4.8 Arnoldi Iteration, Krylov Subspaces and GMRES
48 Arnoldi Iteration, Krylov Subspaces and GMRES We start with the problem of using a similarity transformation to convert an n n matrix A to upper Hessenberg form H, ie, A = QHQ, (30) with an appropriate
More informationPreliminary Examination, Numerical Analysis, August 2016
Preliminary Examination, Numerical Analysis, August 2016 Instructions: This exam is closed books and notes. The time allowed is three hours and you need to work on any three out of questions 1-4 and any
More informationLinear Solvers. Andrew Hazel
Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction
More informationLecture Note 7: Iterative methods for solving linear systems. Xiaoqun Zhang Shanghai Jiao Tong University
Lecture Note 7: Iterative methods for solving linear systems Xiaoqun Zhang Shanghai Jiao Tong University Last updated: December 24, 2014 1.1 Review on linear algebra Norms of vectors and matrices vector
More information4.6 Iterative Solvers for Linear Systems
4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often
More informationAlgebra C Numerical Linear Algebra Sample Exam Problems
Algebra C Numerical Linear Algebra Sample Exam Problems Notation. Denote by V a finite-dimensional Hilbert space with inner product (, ) and corresponding norm. The abbreviation SPD is used for symmetric
More informationLecture 11. Fast Linear Solvers: Iterative Methods. J. Chaudhry. Department of Mathematics and Statistics University of New Mexico
Lecture 11 Fast Linear Solvers: Iterative Methods J. Chaudhry Department of Mathematics and Statistics University of New Mexico J. Chaudhry (UNM) Math/CS 375 1 / 23 Summary: Complexity of Linear Solves
More informationFEM and sparse linear system solving
FEM & sparse linear system solving, Lecture 9, Nov 19, 2017 1/36 Lecture 9, Nov 17, 2017: Krylov space methods http://people.inf.ethz.ch/arbenz/fem17 Peter Arbenz Computer Science Department, ETH Zürich
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011 Lecture Objectives describe when CG can be used to solve Ax
More informationLecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University
Lecture 17 Methods for System of Linear Equations: Part 2 Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of
More informationConjugate Gradient Method
Conjugate Gradient Method Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 10, 2011 T.M. Huang (NTNU) Conjugate Gradient Method October 10, 2011 1 / 36 Outline 1 Steepest
More information7.3 The Jacobi and Gauss-Siedel Iterative Techniques. Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP.
7.3 The Jacobi and Gauss-Siedel Iterative Techniques Problem: To solve Ax = b for A R n n. Methodology: Iteratively approximate solution x. No GEPP. 7.3 The Jacobi and Gauss-Siedel Iterative Techniques
More informationPreliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012
Instructions Preliminary/Qualifying Exam in Numerical Analysis (Math 502a) Spring 2012 The exam consists of four problems, each having multiple parts. You should attempt to solve all four problems. 1.
More informationyou expect to encounter difficulties when trying to solve A x = b? 4. A composite quadrature rule has error associated with it in the following form
Qualifying exam for numerical analysis (Spring 2017) Show your work for full credit. If you are unable to solve some part, attempt the subsequent parts. 1. Consider the following finite difference: f (0)
More informationLecture 9: Krylov Subspace Methods. 2 Derivation of the Conjugate Gradient Algorithm
CS 622 Data-Sparse Matrix Computations September 19, 217 Lecture 9: Krylov Subspace Methods Lecturer: Anil Damle Scribes: David Eriksson, Marc Aurele Gilles, Ariah Klages-Mundt, Sophia Novitzky 1 Introduction
More informationMath 1080: Numerical Linear Algebra Chapter 4, Iterative Methods
Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 March 2015 1 / 70 Topics Introduction to Iterative Methods
More informationThe amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A.
AMSC/CMSC 661 Scientific Computing II Spring 2005 Solution of Sparse Linear Systems Part 2: Iterative methods Dianne P. O Leary c 2005 Solving Sparse Linear Systems: Iterative methods The plan: Iterative
More informationIntroduction. Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods. Example: First Order Richardson. Strategy
Introduction Math 1080: Numerical Linear Algebra Chapter 4, Iterative Methods M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 Solve system Ax = b by repeatedly computing
More informationIterative methods for Linear System
Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and
More informationJae Heon Yun and Yu Du Han
Bull. Korean Math. Soc. 39 (2002), No. 3, pp. 495 509 MODIFIED INCOMPLETE CHOLESKY FACTORIZATION PRECONDITIONERS FOR A SYMMETRIC POSITIVE DEFINITE MATRIX Jae Heon Yun and Yu Du Han Abstract. We propose
More informationBindel, Fall 2016 Matrix Computations (CS 6210) Notes for
1 Iteration basics Notes for 2016-11-07 An iterative solver for Ax = b is produces a sequence of approximations x (k) x. We always stop after finitely many steps, based on some convergence criterion, e.g.
More informationScientific Computing with Case Studies SIAM Press, Lecture Notes for Unit VII Sparse Matrix
Scientific Computing with Case Studies SIAM Press, 2009 http://www.cs.umd.edu/users/oleary/sccswebpage Lecture Notes for Unit VII Sparse Matrix Computations Part 1: Direct Methods Dianne P. O Leary c 2008
More informationConjugate Gradient Tutorial
Conjugate Gradient Tutorial Prof. Chung-Kuan Cheng Computer Science and Engineering Department University of California, San Diego ckcheng@ucsd.edu December 1, 2015 Prof. Chung-Kuan Cheng (UC San Diego)
More informationNotes for CS542G (Iterative Solvers for Linear Systems)
Notes for CS542G (Iterative Solvers for Linear Systems) Robert Bridson November 20, 2007 1 The Basics We re now looking at efficient ways to solve the linear system of equations Ax = b where in this course,
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method The minimization problem We are given a symmetric positive definite matrix R n n and a right hand side vector b R n We want to solve the linear system Find u R n such that
More informationPDE Solvers for Fluid Flow
PDE Solvers for Fluid Flow issues and algorithms for the Streaming Supercomputer Eran Guendelman February 5, 2002 Topics Equations for incompressible fluid flow 3 model PDEs: Hyperbolic, Elliptic, Parabolic
More informationIterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009)
Iterative methods for Linear System of Equations Joint Advanced Student School (JASS-2009) Course #2: Numerical Simulation - from Models to Software Introduction In numerical simulation, Partial Differential
More informationFrom Stationary Methods to Krylov Subspaces
Week 6: Wednesday, Mar 7 From Stationary Methods to Krylov Subspaces Last time, we discussed stationary methods for the iterative solution of linear systems of equations, which can generally be written
More informationConjugate Gradient Method
Conjugate Gradient Method Hung M Phan UMass Lowell April 13, 2017 Throughout, A R n n is symmetric and positive definite, and b R n 1 Steepest Descent Method We present the steepest descent method for
More informationIterative solution methods and their rate of convergence
Uppsala University Graduate School in Mathematics and Computing Institute for Information Technology Numerical Linear Algebra FMB and MN Fall 2007 Mandatory Assignment 3a: Iterative solution methods and
More informationITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD
ITERATIVE PROJECTION METHODS FOR SPARSE LINEAR SYSTEMS AND EIGENPROBLEMS CHAPTER 4 : CONJUGATE GRADIENT METHOD Heinrich Voss voss@tu-harburg.de Hamburg University of Technology Institute of Numerical Simulation
More information5.3 The Power Method Approximation of the Eigenvalue of Largest Module
192 5 Approximation of Eigenvalues and Eigenvectors 5.3 The Power Method The power method is very good at approximating the extremal eigenvalues of the matrix, that is, the eigenvalues having largest and
More informationOUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative methods ffl Krylov subspace methods ffl Preconditioning techniques: Iterative methods ILU
Preconditioning Techniques for Solving Large Sparse Linear Systems Arnold Reusken Institut für Geometrie und Praktische Mathematik RWTH-Aachen OUTLINE ffl CFD: elliptic pde's! Ax = b ffl Basic iterative
More informationSome definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts
Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric
More informationLast Time. Social Network Graphs Betweenness. Graph Laplacian. Girvan-Newman Algorithm. Spectral Bisection
Eigenvalue Problems Last Time Social Network Graphs Betweenness Girvan-Newman Algorithm Graph Laplacian Spectral Bisection λ 2, w 2 Today Small deviation into eigenvalue problems Formulation Standard eigenvalue
More informationThe Conjugate Gradient Method for Solving Linear Systems of Equations
The Conjugate Gradient Method for Solving Linear Systems of Equations Mike Rambo Mentor: Hans de Moor May 2016 Department of Mathematics, Saint Mary s College of California Contents 1 Introduction 2 2
More informationMath 5630: Conjugate Gradient Method Hung M. Phan, UMass Lowell March 29, 2019
Math 563: Conjugate Gradient Method Hung M. Phan, UMass Lowell March 29, 219 hroughout, A R n n is symmetric and positive definite, and b R n. 1 Steepest Descent Method We present the steepest descent
More informationBoundary Value Problems - Solving 3-D Finite-Difference problems Jacob White
Introduction to Simulation - Lecture 2 Boundary Value Problems - Solving 3-D Finite-Difference problems Jacob White Thanks to Deepak Ramaswamy, Michal Rewienski, and Karen Veroy Outline Reminder about
More informationIncomplete Cholesky preconditioners that exploit the low-rank property
anapov@ulb.ac.be ; http://homepages.ulb.ac.be/ anapov/ 1 / 35 Incomplete Cholesky preconditioners that exploit the low-rank property (theory and practice) Artem Napov Service de Métrologie Nucléaire, Université
More informationIterative methods for positive definite linear systems with a complex shift
Iterative methods for positive definite linear systems with a complex shift William McLean, University of New South Wales Vidar Thomée, Chalmers University November 4, 2011 Outline 1. Numerical solution
More informationMaster Thesis Literature Study Presentation
Master Thesis Literature Study Presentation Delft University of Technology The Faculty of Electrical Engineering, Mathematics and Computer Science January 29, 2010 Plaxis Introduction Plaxis Finite Element
More informationJACOBI S ITERATION METHOD
ITERATION METHODS These are methods which compute a sequence of progressively accurate iterates to approximate the solution of Ax = b. We need such methods for solving many large linear systems. Sometimes
More informationCME342 Parallel Methods in Numerical Analysis. Matrix Computation: Iterative Methods II. Sparse Matrix-vector Multiplication.
CME342 Parallel Methods in Numerical Analysis Matrix Computation: Iterative Methods II Outline: CG & its parallelization. Sparse Matrix-vector Multiplication. 1 Basic iterative methods: Ax = b r = b Ax
More informationKrylov Subspace Methods that Are Based on the Minimization of the Residual
Chapter 5 Krylov Subspace Methods that Are Based on the Minimization of the Residual Remark 51 Goal he goal of these methods consists in determining x k x 0 +K k r 0,A such that the corresponding Euclidean
More informationSOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS. Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA
1 SOLVING SPARSE LINEAR SYSTEMS OF EQUATIONS Chao Yang Computational Research Division Lawrence Berkeley National Laboratory Berkeley, CA, USA 2 OUTLINE Sparse matrix storage format Basic factorization
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 19: Computing the SVD; Sparse Linear Systems Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationMath Introduction to Numerical Analysis - Class Notes. Fernando Guevara Vasquez. Version Date: January 17, 2012.
Math 5620 - Introduction to Numerical Analysis - Class Notes Fernando Guevara Vasquez Version 1990. Date: January 17, 2012. 3 Contents 1. Disclaimer 4 Chapter 1. Iterative methods for solving linear systems
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationLab 1: Iterative Methods for Solving Linear Systems
Lab 1: Iterative Methods for Solving Linear Systems January 22, 2017 Introduction Many real world applications require the solution to very large and sparse linear systems where direct methods such as
More informationThe Lanczos and conjugate gradient algorithms
The Lanczos and conjugate gradient algorithms Gérard MEURANT October, 2008 1 The Lanczos algorithm 2 The Lanczos algorithm in finite precision 3 The nonsymmetric Lanczos algorithm 4 The Golub Kahan bidiagonalization
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview
More informationLINEAR SYSTEMS (11) Intensive Computation
LINEAR SYSTEMS () Intensive Computation 27-8 prof. Annalisa Massini Viviana Arrigoni EXACT METHODS:. GAUSSIAN ELIMINATION. 2. CHOLESKY DECOMPOSITION. ITERATIVE METHODS:. JACOBI. 2. GAUSS-SEIDEL 2 CHOLESKY
More informationNumerical Methods in Matrix Computations
Ake Bjorck Numerical Methods in Matrix Computations Springer Contents 1 Direct Methods for Linear Systems 1 1.1 Elements of Matrix Theory 1 1.1.1 Matrix Algebra 2 1.1.2 Vector Spaces 6 1.1.3 Submatrices
More information5.6. PSEUDOINVERSES 101. A H w.
5.6. PSEUDOINVERSES 0 Corollary 5.6.4. If A is a matrix such that A H A is invertible, then the least-squares solution to Av = w is v = A H A ) A H w. The matrix A H A ) A H is the left inverse of A and
More informationMatrices, Moments and Quadrature, cont d
Jim Lambers CME 335 Spring Quarter 2010-11 Lecture 4 Notes Matrices, Moments and Quadrature, cont d Estimation of the Regularization Parameter Consider the least squares problem of finding x such that
More informationCourse Notes: Week 4
Course Notes: Week 4 Math 270C: Applied Numerical Linear Algebra 1 Lecture 9: Steepest Descent (4/18/11) The connection with Lanczos iteration and the CG was not originally known. CG was originally derived
More informationLecture 22. r i+1 = b Ax i+1 = b A(x i + α i r i ) =(b Ax i ) α i Ar i = r i α i Ar i
8.409 An Algorithmist s oolkit December, 009 Lecturer: Jonathan Kelner Lecture Last time Last time, we reduced solving sparse systems of linear equations Ax = b where A is symmetric and positive definite
More informationChapter 6: Orthogonality
Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products
More informationThe Conjugate Gradient Method
CHAPTER The Conjugate Gradient Method Exercise.: A-norm Let A = LL be a Cholesy factorization of A, i.e.l is lower triangular with positive diagonal elements. The A-norm then taes the form x A = p x T
More informationSolving linear systems (6 lectures)
Chapter 2 Solving linear systems (6 lectures) 2.1 Solving linear systems: LU factorization (1 lectures) Reference: [Trefethen, Bau III] Lecture 20, 21 How do you solve Ax = b? (2.1.1) In numerical linear
More informationMath 577 Assignment 7
Math 577 Assignment 7 Thanks for Yu Cao 1. Solution. The linear system being solved is Ax = 0, where A is a (n 1 (n 1 matrix such that 2 1 1 2 1 A =......... 1 2 1 1 2 and x = (U 1, U 2,, U n 1. By the
More informationMath 504 (Fall 2011) 1. (*) Consider the matrices
Math 504 (Fall 2011) Instructor: Emre Mengi Study Guide for Weeks 11-14 This homework concerns the following topics. Basic definitions and facts about eigenvalues and eigenvectors (Trefethen&Bau, Lecture
More informationClass notes: Approximation
Class notes: Approximation Introduction Vector spaces, linear independence, subspace The goal of Numerical Analysis is to compute approximations We want to approximate eg numbers in R or C vectors in R
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More informationIntroduction to the Numerical Solution of IVP for ODE
Introduction to the Numerical Solution of IVP for ODE 45 Introduction to the Numerical Solution of IVP for ODE Consider the IVP: DE x = f(t, x), IC x(a) = x a. For simplicity, we will assume here that
More informationPreconditioned inverse iteration and shift-invert Arnoldi method
Preconditioned inverse iteration and shift-invert Arnoldi method Melina Freitag Department of Mathematical Sciences University of Bath CSC Seminar Max-Planck-Institute for Dynamics of Complex Technical
More informationMultigrid absolute value preconditioning
Multigrid absolute value preconditioning Eugene Vecharynski 1 Andrew Knyazev 2 (speaker) 1 Department of Computer Science and Engineering University of Minnesota 2 Department of Mathematical and Statistical
More informationThe quadratic eigenvalue problem (QEP) is to find scalars λ and nonzero vectors u satisfying
I.2 Quadratic Eigenvalue Problems 1 Introduction The quadratic eigenvalue problem QEP is to find scalars λ and nonzero vectors u satisfying where Qλx = 0, 1.1 Qλ = λ 2 M + λd + K, M, D and K are given
More informationIterative Methods for Linear Systems of Equations
Iterative Methods for Linear Systems of Equations Projection methods (3) ITMAN PhD-course DTU 20-10-08 till 24-10-08 Martin van Gijzen 1 Delft University of Technology Overview day 4 Bi-Lanczos method
More information1 Solutions to selected problems
Solutions to selected problems Section., #a,c,d. a. p x = n for i = n : 0 p x = xp x + i end b. z = x, y = x for i = : n y = y + x i z = zy end c. y = (t x ), p t = a for i = : n y = y(t x i ) p t = p
More informationPENNON A Generalized Augmented Lagrangian Method for Convex NLP and SDP p.1/39
PENNON A Generalized Augmented Lagrangian Method for Convex NLP and SDP Michal Kočvara Institute of Information Theory and Automation Academy of Sciences of the Czech Republic and Czech Technical University
More information