The Conjugate Gradient Method
|
|
- August Lyons
- 6 years ago
- Views:
Transcription
1 The Conjugate Gradient Method Jason E. Hicken Aerospace Design Lab Department of Aeronautics & Astronautics Stanford University 14 July 2011
2 Lecture Objectives describe when CG can be used to solve Ax = b relate CG to the method of conjugate directions describe what CG does geometrically explain each line in the CG algorithm
3 We are interested in solving the linear system Ax = b where x, b R n and A R n n Matrix is symmetric positive-definite (SPD) A T = A (symmetric) x T Ax > 0, x 0 (positive-definite) discretization of elliptic PDEs optimization of quadratic functionals nonlinear optimization problems
4 We are interested in solving the linear system Ax = b where x, b R n and A R n n Matrix is symmetric positive-definite (SPD) A T = A (symmetric) x T Ax > 0, x 0 (positive-definite) discretization of elliptic PDEs optimization of quadratic functionals nonlinear optimization problems
5 When A is SPD, solving the linear system is the same as minimizing the quadratic form f(x) = 1 2 xt Ax b T x. Why? If x is the minimizing point, then f(x ) = Ax b = 0 and, for x x f(x) f(x ) > 0. (homework)
6 When A is SPD, solving the linear system is the same as minimizing the quadratic form f(x) = 1 2 xt Ax b T x. Why? If x is the minimizing point, then f(x ) = Ax b = 0 and, for x x f(x) f(x ) > 0. (homework)
7 Definitions Let x i be the approximate solution to Ax = b at iteration i. error: e i x i x residual: r i b Ax i The following identities for the residual will be useful later. r i = Ae i r i = f(x i )
8 Model problem [ ]( x1 ) ( 4 = x 2 4) x x 1 + 5x 2 = 4 5x 1 3x 2 = 4 x = [2 2] T x 1
9 Model problem [ ]( x1 ) ( 4 = x 2 4) x x 1 + 5x 2 = 4 5x 1 3x 2 = 4 x = [2 2] T x 1
10 Review: Steepest Descent Method Qualitatively, how will steepest descent proceed on our model problem, starting at x 0 = ( 1 3,1)T? x x x 1
11 Review: Steepest Descent Method Qualitatively, how will steepest descent proceed on our model problem, starting at x 0 = ( 1 3,1)T? x x 0 1 x x 1
12 Review: Steepest Descent Method Qualitatively, how will steepest descent proceed on our model problem, starting at x 0 = ( 1 3,1)T? x x x 1
13 How can we eliminate this zig-zag behaviour? To find the answer, we begin by considering the easier problem [ ]( ) ( ) 2 0 x1 4 2 =, 0 8 x 2 0 f(x) = x x x 1. Here, the equations are decoupled, so we can minimize in each direction independently. What do the contours of the corresponding quadratic form look like?
14 How can we eliminate this zig-zag behaviour? To find the answer, we begin by considering the easier problem [ ]( ) ( ) 2 0 x1 4 2 =, 0 8 x 2 0 f(x) = x x x 1. Here, the equations are decoupled, so we can minimize in each direction independently. What do the contours of the corresponding quadratic form look like?
15 Simplified problem [ ]( x1 ) = x 2 ( ) x x 1
16 Simplified problem [ ]( x1 ) = x 2 ( ) x x 1
17 Simplified problem [ ]( x1 ) = x 2 ( ) x 2 e x 1
18 Simplified problem [ ]( x1 ) = x 2 ( ) x 2 e x 1
19 Method of Orthogonal Directions Idea: Express error as a sum of n orthogonal search directions n 1 e x 0 x = α i d i. i=0 At iteration i +1, eliminate component α i d i. never need to search along d i again converge in n iterations! How would we apply the method of orthogonal directions to a non-diagonal matrix?
20 Method of Orthogonal Directions Idea: Express error as a sum of n orthogonal search directions n 1 e x 0 x = α i d i. i=0 At iteration i +1, eliminate component α i d i. never need to search along d i again converge in n iterations! How would we apply the method of orthogonal directions to a non-diagonal matrix?
21 Method of Orthogonal Directions Idea: Express error as a sum of n orthogonal search directions n 1 e x 0 x = α i d i. i=0 At iteration i +1, eliminate component α i d i. never need to search along d i again converge in n iterations! How would we apply the method of orthogonal directions to a non-diagonal matrix?
22 Review of Inner Products The search directions in the method of orthogonal directions are orthogonal with respect to the dot product. The dot product is an example of an inner product. Inner Product For x,y,z R n and α R, an inner product (,) : R n R n R satisfies symmetry: (x,y) = (y,x) linearity: (αx +y,z) = α(x,z)+(y,z) positive-definiteness: (x,x) > 0 x 0
23 Review of Inner Products The search directions in the method of orthogonal directions are orthogonal with respect to the dot product. The dot product is an example of an inner product. Inner Product For x,y,z R n and α R, an inner product (,) : R n R n R satisfies symmetry: (x,y) = (y,x) linearity: (αx +y,z) = α(x,z)+(y,z) positive-definiteness: (x,x) > 0 x 0
24 Fact: (x,y) A x T Ay is an inner product A-orthogonality (conjugacy) We say two vectors x,y R n are A-orthogonal, or conjugate, if (x,y) A = x T Ay = 0. What happens if we use A-orthogonality rather than standard orthogonality in the method of orthogonal directions?
25 Let {p 0,p 1,...,p n 1 } be a set of n linearly independent vectors that are A-orthogonal. If p i is the i th column of P, then P T AP = Σ where Σ is a diagonal matrix. Substitute x = Py into the quadratic form: f(py) = y T Σy (P T b) T y. We can apply the method of orthogonal directions in y-space.
26 Let {p 0,p 1,...,p n 1 } be a set of n linearly independent vectors that are A-orthogonal. If p i is the i th column of P, then P T AP = Σ where Σ is a diagonal matrix. Substitute x = Py into the quadratic form: f(py) = y T Σy (P T b) T y. We can apply the method of orthogonal directions in y-space.
27 New Problem: how do we get the set {p i } of conjugate vectors? Gram-Schmidt Conjugation Let {d 0,d 1,...,d n 1 } be a set of linearly independent vectors, e.g., coordinate axes. set p 0 = d 0 for i > 0 i 1 p i = d i β ij p j j=0 where β ij = (d i,p j ) A /(p j,p j ) A.
28 New Problem: how do we get the set {p i } of conjugate vectors? Gram-Schmidt Conjugation Let {d 0,d 1,...,d n 1 } be a set of linearly independent vectors, e.g., coordinate axes. set p 0 = d 0 for i > 0 i 1 p i = d i β ij p j j=0 where β ij = (d i,p j ) A /(p j,p j ) A.
29 The Method of Conjugate Directions Force the error at iteration i +1 to be conjugate to the search direction p i. p T i Ae i+1 = p T i A(e i +α i p i ) = 0 α i = pt i Ae i p T i Ap i = p T i r i p T i Ap i never need to search along p i again converge in n iterations!
30 The Method of Conjugate Directions Force the error at iteration i +1 to be conjugate to the search direction p i. p T i Ae i+1 = p T i A(e i +α i p i ) = 0 α i = pt i Ae i p T i Ap i = p T i r i p T i Ap i never need to search along p i again converge in n iterations!
31 The Method of Conjugate Directions Force the error at iteration i +1 to be conjugate to the search direction p i. p T i Ae i+1 = p T i A(e i +α i p i ) = 0 α i = pt i Ae i p T i Ap i = p T i r i p T i Ap i never need to search along p i again converge in n iterations!
32 The Method of Conjugate Directions [ ]( x1 ) ( 4 = x 2 4) x d 1 1 d x 1
33 The Method of Conjugate Directions [ ]( x1 ) ( 4 = x 2 4) x p 1 1 p x 1
34 The Method of Conjugate Directions [ ]( x1 ) ( 4 = x 2 4) x x x 1
35 The Method of Conjugate Directions [ ]( x1 ) ( 4 = x 2 4) x x x 1
36 The Method of Conjugate Directions [ ]( x1 ) ( 4 = x 2 4) x x x 1
37 The Method of Conjugate Directions is well defined, and avoids the zig-zagging of Steepest Descent. What about computational expense? If we choose the d i in Gram-Schmidt conjugation to be the coordinate axes, the Method of Conjugate Directions is equivalent to Gaussian elimination. Keeping all the p i is the same as storing a dense matrix! Can we find a smarter choice for d i?
38 The Method of Conjugate Directions is well defined, and avoids the zig-zagging of Steepest Descent. What about computational expense? If we choose the d i in Gram-Schmidt conjugation to be the coordinate axes, the Method of Conjugate Directions is equivalent to Gaussian elimination. Keeping all the p i is the same as storing a dense matrix! Can we find a smarter choice for d i?
39 Error Decomposition Using p i x e x 1
40 Error Decomposition Using p i x e 0 α 1 p 1 1 α 0 p x 1
41 The error at iteration i can be expressed as n 1 e i = α k p k, k=i so the error must be conjugate to p j for j < i: pj T Ae i = 0, pj T r i = 0, but from Gram-Schmidt conjugation we have = =
42 The error at iteration i can be expressed as n 1 e i = α k p k, k=i so the error must be conjugate to p j for j < i: pj T Ae i = 0, pj T r i = 0, but from Gram-Schmidt conjugation we have = =
43 The error at iteration i can be expressed as n 1 e i = α k p k, k=i so the error must be conjugate to p j for j < i: pj T Ae i = 0, pj T r i = 0, but from Gram-Schmidt conjugation we have = =
44 The error at iteration i can be expressed as n 1 e i = α k p k, k=i so the error must be conjugate to p j for j < i: pj T Ae i = 0, pj T r i = 0, but from Gram-Schmidt conjugation we have j 1 p j = d j β jk p k. k=0
45 The error at iteration i can be expressed as n 1 e i = α k p k, k=i so the error must be conjugate to p j for j < i: p T j Ae i = 0, p T j r i = 0, but from Gram-Schmidt conjugation we have p T j r i = d T j r i j 1 k=0 0 = d T j r i, j < i. β jk p T k r i
46 Thus, the residual at iteration i is orthogonal to the vectors d j used in the previous iterations: d T j r i = 0, j < i Idea: what happens if we choose d i = r i? residuals become mutually orthogonal r i is orthogonal to p j, for j < i r i+1 becomes conjugate to p j, for j < i This last point is not immediately obvious, so we will prove it. This result has significant implications for Gram-Schmidt conjugation. we showed this is true for any choice of d i
47 Thus, the residual at iteration i is orthogonal to the vectors d j used in the previous iterations: d T j r i = 0, j < i Idea: what happens if we choose d i = r i? residuals become mutually orthogonal r i is orthogonal to p j, for j < i r i+1 becomes conjugate to p j, for j < i This last point is not immediately obvious, so we will prove it. This result has significant implications for Gram-Schmidt conjugation. we showed this is true for any choice of d i
48 The solution is updated according to x j+1 = x j +α j p j r j+1 = r j α j Ap j Ap j = 1 α j (r j r j+1 ). Next, take the dot product of both sides with an arbitrary residual r i : ri T r i α i, i = j ri T Ap j = rt i r i α i 1, i = j +1 0, otherwise.
49 The solution is updated according to x j+1 = x j +α j p j r j+1 = r j α j Ap j Ap j = 1 α j (r j r j+1 ). Next, take the dot product of both sides with an arbitrary residual r i : ri T r i α i, i = j ri T Ap j = rt i r i α i 1, i = j +1 0, otherwise.
50 The solution is updated according to x j+1 = x j +α j p j r j+1 = r j α j Ap j Ap j = 1 α j (r j r j+1 ). Next, take the dot product of both sides with an arbitrary residual r i : ri T r i α i, i = j ri T Ap j = rt i r i α i 1, i = j +1 0, otherwise.
51 The solution is updated according to x j+1 = x j +α j p j r j+1 = r j α j Ap j Ap j = 1 α j (r j r j+1 ). Next, take the dot product of both sides with an arbitrary residual r i : ri T r i α i, i = j ri T Ap j = rt i r i α i 1, i = j +1 0, otherwise.
52 The solution is updated according to x j+1 = x j +α j p j r j+1 = r j α j Ap j Ap j = 1 α j (r j r j+1 ). Next, take the dot product of both sides with an arbitrary residual r i : ri T r i α i, i = j ri T Ap j = rt i r i α i 1, i = j +1 0, otherwise.
53 We can show that the first case (i = j) contains no new information (homework). Divide the remaining cases by p T j Ap j and insert the definition of α i 1 : ri T Ap j pj T = Ap j }{{} β ij rt i r i ri 1 T r i 1, i = j +1 0, otherwise. We recognize the L.H.S. as the coefficients in Gram-Schmidt conjugation only one coefficient is nonzero!
54 We can show that the first case (i = j) contains no new information (homework). Divide the remaining cases by p T j Ap j and insert the definition of α i 1 : r T i Ap j p T j Ap j }{{} β ij = rt i r i ri 1 T r i 1, i = j +1 0, otherwise. We recognize the L.H.S. as the coefficients in Gram-Schmidt conjugation only one coefficient is nonzero!
55 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
56 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
57 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
58 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
59 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
60 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
61 The Conjugate Gradient Method Set p 0 = r 0 = b Ax 0 and i = 0 α i = (p T i r i )/(p T i Ap i ) x i+1 = x i +α i p i r i+1 = r i α i Ap i (step length) (sol. update) (resid. update) β i+1,i = (r T i+1r i+1 )/(r T i r i ) (G.S. coeff.) p i+1 = r i+1 β i+1,i p i (Gram Schmidt) i := i +1
62 The Conjugate Gradient Method [ ]( x1 ) ( 4 = x 2 4) x x x 1
63 The Conjugate Gradient Method [ ]( x1 ) ( 4 = x 2 4) x x 1
64 The Conjugate Gradient Method [ ]( x1 ) ( 4 = x 2 4) x x 1
65 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
66 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
67 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
68 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
69 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
70 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
71 Lecture Objectives describe when CG can be used to solve Ax = b A must be symmetric positive-definite relate CG to the method of conjugate directions CG is a method of conjugate directions with the choice d i = r i, which simplifies Gram-Schmidt conjugation describe what CG does geometrically Performs the method of orthogonal directions in a transformed space where the contours of the quadratic form are aligned with the coordinate axes explain each line in the CG algorithm
72 References Saad, Y., Iterative Methods for Sparse Linear Systems, second edition Shewchuk, J. R., An introduction to the Conjugate Gradient method without the agonizing pain
The conjugate gradient method
The conjugate gradient method Michael S. Floater November 1, 2011 These notes try to provide motivation and an explanation of the CG method. 1 The method of conjugate directions We want to solve the linear
More informationEECS 275 Matrix Computation
EECS 275 Matrix Computation Ming-Hsuan Yang Electrical Engineering and Computer Science University of California at Merced Merced, CA 95344 http://faculty.ucmerced.edu/mhyang Lecture 20 1 / 20 Overview
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationthe method of steepest descent
MATH 3511 Spring 2018 the method of steepest descent http://www.phys.uconn.edu/ rozman/courses/m3511_18s/ Last modified: February 6, 2018 Abstract The Steepest Descent is an iterative method for solving
More informationConjugate Gradient (CG) Method
Conjugate Gradient (CG) Method by K. Ozawa 1 Introduction In the series of this lecture, I will introduce the conjugate gradient method, which solves efficiently large scale sparse linear simultaneous
More informationMathematical optimization
Optimization Mathematical optimization Determine the best solutions to certain mathematically defined problems that are under constrained determine optimality criteria determine the convergence of the
More informationChapter 7 Iterative Techniques in Matrix Algebra
Chapter 7 Iterative Techniques in Matrix Algebra Per-Olof Persson persson@berkeley.edu Department of Mathematics University of California, Berkeley Math 128B Numerical Analysis Vector Norms Definition
More informationNotes on Some Methods for Solving Linear Systems
Notes on Some Methods for Solving Linear Systems Dianne P. O Leary, 1983 and 1999 and 2007 September 25, 2007 When the matrix A is symmetric and positive definite, we have a whole new class of algorithms
More informationCourse Notes: Week 4
Course Notes: Week 4 Math 270C: Applied Numerical Linear Algebra 1 Lecture 9: Steepest Descent (4/18/11) The connection with Lanczos iteration and the CG was not originally known. CG was originally derived
More informationThe Conjugate Gradient Method for Solving Linear Systems of Equations
The Conjugate Gradient Method for Solving Linear Systems of Equations Mike Rambo Mentor: Hans de Moor May 2016 Department of Mathematics, Saint Mary s College of California Contents 1 Introduction 2 2
More information1 Conjugate gradients
Notes for 2016-11-18 1 Conjugate gradients We now turn to the method of conjugate gradients (CG), perhaps the best known of the Krylov subspace solvers. The CG iteration can be characterized as the iteration
More informationConjugate-Gradient. Learn about the Conjugate-Gradient Algorithm and its Uses. Descent Algorithms and the Conjugate-Gradient Method. Qx = b.
Lab 1 Conjugate-Gradient Lab Objective: Learn about the Conjugate-Gradient Algorithm and its Uses Descent Algorithms and the Conjugate-Gradient Method There are many possibilities for solving a linear
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More informationPETROV-GALERKIN METHODS
Chapter 7 PETROV-GALERKIN METHODS 7.1 Energy Norm Minimization 7.2 Residual Norm Minimization 7.3 General Projection Methods 7.1 Energy Norm Minimization Saad, Sections 5.3.1, 5.2.1a. 7.1.1 Methods based
More informationConjugate Gradient Tutorial
Conjugate Gradient Tutorial Prof. Chung-Kuan Cheng Computer Science and Engineering Department University of California, San Diego ckcheng@ucsd.edu December 1, 2015 Prof. Chung-Kuan Cheng (UC San Diego)
More informationLinear Solvers. Andrew Hazel
Linear Solvers Andrew Hazel Introduction Thus far we have talked about the formulation and discretisation of physical problems...... and stopped when we got to a discrete linear system of equations. Introduction
More informationIterative methods for Linear System of Equations. Joint Advanced Student School (JASS-2009)
Iterative methods for Linear System of Equations Joint Advanced Student School (JASS-2009) Course #2: Numerical Simulation - from Models to Software Introduction In numerical simulation, Partial Differential
More informationAn Iterative Descent Method
Conjugate Gradient: An Iterative Descent Method The Plan Review Iterative Descent Conjugate Gradient Review : Iterative Descent Iterative Descent is an unconstrained optimization process x (k+1) = x (k)
More informationConjugate Gradient Method
Conjugate Gradient Method Hung M Phan UMass Lowell April 13, 2017 Throughout, A R n n is symmetric and positive definite, and b R n 1 Steepest Descent Method We present the steepest descent method for
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 21: Sensitivity of Eigenvalues and Eigenvectors; Conjugate Gradient Method Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical Analysis
More informationIterative Methods for Solving A x = b
Iterative Methods for Solving A x = b A good (free) online source for iterative methods for solving A x = b is given in the description of a set of iterative solvers called templates found at netlib: http
More informationSome minimization problems
Week 13: Wednesday, Nov 14 Some minimization problems Last time, we sketched the following two-step strategy for approximating the solution to linear systems via Krylov subspaces: 1. Build a sequence of
More informationNonlinearOptimization
1/35 NonlinearOptimization Pavel Kordík Department of Computer Systems Faculty of Information Technology Czech Technical University in Prague Jiří Kašpar, Pavel Tvrdík, 2011 Unconstrained nonlinear optimization,
More informationMATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2 1 Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 8 topics: quadratic optimisation, least squares,
More informationMAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012
(Homework 1: Chapter 1: Exercises 1-7, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions
More informationCourse Notes: Week 1
Course Notes: Week 1 Math 270C: Applied Numerical Linear Algebra 1 Lecture 1: Introduction (3/28/11) We will focus on iterative methods for solving linear systems of equations (and some discussion of eigenvalues
More informationChapter 10 Conjugate Direction Methods
Chapter 10 Conjugate Direction Methods An Introduction to Optimization Spring, 2012 1 Wei-Ta Chu 2012/4/13 Introduction Conjugate direction methods can be viewed as being intermediate between the method
More informationChapter 7. Iterative methods for large sparse linear systems. 7.1 Sparse matrix algebra. Large sparse matrices
Chapter 7 Iterative methods for large sparse linear systems In this chapter we revisit the problem of solving linear systems of equations, but now in the context of large sparse systems. The price to pay
More information7.2 Steepest Descent and Preconditioning
7.2 Steepest Descent and Preconditioning Descent methods are a broad class of iterative methods for finding solutions of the linear system Ax = b for symmetric positive definite matrix A R n n. Consider
More informationGradient Descent Methods
Lab 18 Gradient Descent Methods Lab Objective: Many optimization methods fall under the umbrella of descent algorithms. The idea is to choose an initial guess, identify a direction from this point along
More informationConjugate Gradient Method
Conjugate Gradient Method Tsung-Ming Huang Department of Mathematics National Taiwan Normal University October 10, 2011 T.M. Huang (NTNU) Conjugate Gradient Method October 10, 2011 1 / 36 Outline 1 Steepest
More informationMath 5630: Conjugate Gradient Method Hung M. Phan, UMass Lowell March 29, 2019
Math 563: Conjugate Gradient Method Hung M. Phan, UMass Lowell March 29, 219 hroughout, A R n n is symmetric and positive definite, and b R n. 1 Steepest Descent Method We present the steepest descent
More informationIterative methods for Linear System
Iterative methods for Linear System JASS 2009 Student: Rishi Patil Advisor: Prof. Thomas Huckle Outline Basics: Matrices and their properties Eigenvalues, Condition Number Iterative Methods Direct and
More informationConjugate Gradients: Idea
Overview Steepest Descent often takes steps in the same direction as earlier steps Wouldn t it be better every time we take a step to get it exactly right the first time? Again, in general we choose a
More informationMODULE 8 Topics: Null space, range, column space, row space and rank of a matrix
MODULE 8 Topics: Null space, range, column space, row space and rank of a matrix Definition: Let L : V 1 V 2 be a linear operator. The null space N (L) of L is the subspace of V 1 defined by N (L) = {x
More informationLec10p1, ORF363/COS323
Lec10 Page 1 Lec10p1, ORF363/COS323 This lecture: Conjugate direction methods Conjugate directions Conjugate Gram-Schmidt The conjugate gradient (CG) algorithm Solving linear systems Leontief input-output
More information17 Solution of Nonlinear Systems
17 Solution of Nonlinear Systems We now discuss the solution of systems of nonlinear equations. An important ingredient will be the multivariate Taylor theorem. Theorem 17.1 Let D = {x 1, x 2,..., x m
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 3: Iterative Methods PD
More informationNotes on PCG for Sparse Linear Systems
Notes on PCG for Sparse Linear Systems Luca Bergamaschi Department of Civil Environmental and Architectural Engineering University of Padova e-mail luca.bergamaschi@unipd.it webpage www.dmsa.unipd.it/
More informationCS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3
CS137 Introduction to Scientific Computing Winter Quarter 2004 Solutions to Homework #3 Felix Kwok February 27, 2004 Written Problems 1. (Heath E3.10) Let B be an n n matrix, and assume that B is both
More informationNumerical Methods - Numerical Linear Algebra
Numerical Methods - Numerical Linear Algebra Y. K. Goh Universiti Tunku Abdul Rahman 2013 Y. K. Goh (UTAR) Numerical Methods - Numerical Linear Algebra I 2013 1 / 62 Outline 1 Motivation 2 Solving Linear
More informationConjugate gradient method. Descent method. Conjugate search direction. Conjugate Gradient Algorithm (294)
Conjugate gradient method Descent method Hestenes, Stiefel 1952 For A N N SPD In exact arithmetic, solves in N steps In real arithmetic No guaranteed stopping Often converges in many fewer than N steps
More information9.1 Preconditioned Krylov Subspace Methods
Chapter 9 PRECONDITIONING 9.1 Preconditioned Krylov Subspace Methods 9.2 Preconditioned Conjugate Gradient 9.3 Preconditioned Generalized Minimal Residual 9.4 Relaxation Method Preconditioners 9.5 Incomplete
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf-Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2018/19 Part 4: Iterative Methods PD
More information6.4 Krylov Subspaces and Conjugate Gradients
6.4 Krylov Subspaces and Conjugate Gradients Our original equation is Ax = b. The preconditioned equation is P Ax = P b. When we write P, we never intend that an inverse will be explicitly computed. P
More informationApplied Mathematics 205. Unit V: Eigenvalue Problems. Lecturer: Dr. David Knezevic
Applied Mathematics 205 Unit V: Eigenvalue Problems Lecturer: Dr. David Knezevic Unit V: Eigenvalue Problems Chapter V.4: Krylov Subspace Methods 2 / 51 Krylov Subspace Methods In this chapter we give
More informationDS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationTopics. The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems
Topics The CG Algorithm Algorithmic Options CG s Two Main Convergence Theorems What about non-spd systems? Methods requiring small history Methods requiring large history Summary of solvers 1 / 52 Conjugate
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra)
AMS526: Numerical Analysis I (Numerical Linear Algebra) Lecture 24: Preconditioning and Multigrid Solver Xiangmin Jiao SUNY Stony Brook Xiangmin Jiao Numerical Analysis I 1 / 5 Preconditioning Motivation:
More informationNumerical Linear Algebra Primer. Ryan Tibshirani Convex Optimization
Numerical Linear Algebra Primer Ryan Tibshirani Convex Optimization 10-725 Consider Last time: proximal Newton method min x g(x) + h(x) where g, h convex, g twice differentiable, and h simple. Proximal
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationGeometric Modeling Summer Semester 2010 Mathematical Tools (1)
Geometric Modeling Summer Semester 2010 Mathematical Tools (1) Recap: Linear Algebra Today... Topics: Mathematical Background Linear algebra Analysis & differential geometry Numerical techniques Geometric
More informationSolution to Homework 8, Math 2568
Solution to Homework 8, Math 568 S 5.4: No. 0. Use property of heorem 5 to test for linear independence in P 3 for the following set of cubic polynomials S = { x 3 x, x x, x, x 3 }. Solution: If we use
More informationLecture 22. r i+1 = b Ax i+1 = b A(x i + α i r i ) =(b Ax i ) α i Ar i = r i α i Ar i
8.409 An Algorithmist s oolkit December, 009 Lecturer: Jonathan Kelner Lecture Last time Last time, we reduced solving sparse systems of linear equations Ax = b where A is symmetric and positive definite
More informationThe amount of work to construct each new guess from the previous one should be a small multiple of the number of nonzeros in A.
AMSC/CMSC 661 Scientific Computing II Spring 2005 Solution of Sparse Linear Systems Part 2: Iterative methods Dianne P. O Leary c 2005 Solving Sparse Linear Systems: Iterative methods The plan: Iterative
More informationConjugate Gradients I: Setup
Conjugate Gradients I: Setup CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Justin Solomon CS 205A: Mathematical Methods Conjugate Gradients I: Setup 1 / 22 Time for Gaussian Elimination
More informationx 3y 2z = 6 1.2) 2x 4y 3z = 8 3x + 6y + 8z = 5 x + 3y 2z + 5t = 4 1.5) 2x + 8y z + 9t = 9 3x + 5y 12z + 17t = 7
Linear Algebra and its Applications-Lab 1 1) Use Gaussian elimination to solve the following systems x 1 + x 2 2x 3 + 4x 4 = 5 1.1) 2x 1 + 2x 2 3x 3 + x 4 = 3 3x 1 + 3x 2 4x 3 2x 4 = 1 x + y + 2z = 4 1.4)
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Classical Iterations We have a problem, We assume that the matrix comes from a discretization of a PDE. The best and most popular model problem is, The matrix will be as large
More informationSome definitions. Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization. A-inner product. Important facts
Some definitions Math 1080: Numerical Linear Algebra Chapter 5, Solving Ax = b by Optimization M. M. Sussman sussmanm@math.pitt.edu Office Hours: MW 1:45PM-2:45PM, Thack 622 A matrix A is SPD (Symmetric
More informationM.A. Botchev. September 5, 2014
Rome-Moscow school of Matrix Methods and Applied Linear Algebra 2014 A short introduction to Krylov subspaces for linear systems, matrix functions and inexact Newton methods. Plan and exercises. M.A. Botchev
More informationLecture 11: CMSC 878R/AMSC698R. Iterative Methods An introduction. Outline. Inverse, LU decomposition, Cholesky, SVD, etc.
Lecture 11: CMSC 878R/AMSC698R Iterative Methods An introduction Outline Direct Solution of Linear Systems Inverse, LU decomposition, Cholesky, SVD, etc. Iterative methods for linear systems Why? Matrix
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationAlgorithm for Sparse Approximate Inverse Preconditioners in the Conjugate Gradient Method
Algorithm for Sparse Approximate Inverse Preconditioners in the Conjugate Gradient Method Ilya B. Labutin A.A. Trofimuk Institute of Petroleum Geology and Geophysics SB RAS, 3, acad. Koptyug Ave., Novosibirsk
More informationLecture 17 Methods for System of Linear Equations: Part 2. Songting Luo. Department of Mathematics Iowa State University
Lecture 17 Methods for System of Linear Equations: Part 2 Songting Luo Department of Mathematics Iowa State University MATH 481 Numerical Methods for Differential Equations Songting Luo ( Department of
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method The minimization problem We are given a symmetric positive definite matrix R n n and a right hand side vector b R n We want to solve the linear system Find u R n such that
More informationLecture # 20 The Preconditioned Conjugate Gradient Method
Lecture # 20 The Preconditioned Conjugate Gradient Method We wish to solve Ax = b (1) A R n n is symmetric and positive definite (SPD). We then of n are being VERY LARGE, say, n = 10 6 or n = 10 7. Usually,
More informationLinear Models Review
Linear Models Review Vectors in IR n will be written as ordered n-tuples which are understood to be column vectors, or n 1 matrices. A vector variable will be indicted with bold face, and the prime sign
More informationwhich arises when we compute the orthogonal projection of a vector y in a subspace with an orthogonal basis. Hence assume that P y = A ij = x j, x i
MODULE 6 Topics: Gram-Schmidt orthogonalization process We begin by observing that if the vectors {x j } N are mutually orthogonal in an inner product space V then they are necessarily linearly independent.
More information4.6 Iterative Solvers for Linear Systems
4.6 Iterative Solvers for Linear Systems Why use iterative methods? Virtually all direct methods for solving Ax = b require O(n 3 ) floating point operations. In practical applications the matrix A often
More informationNeural Network Training
Neural Network Training Sargur Srihari Topics in Network Training 0. Neural network parameters Probabilistic problem formulation Specifying the activation and error functions for Regression Binary classification
More informationProperties of Matrices and Operations on Matrices
Properties of Matrices and Operations on Matrices A common data structure for statistical analysis is a rectangular array or matris. Rows represent individual observational units, or just observations,
More informationSOLUTIONS to Exercises from Optimization
SOLUTIONS to Exercises from Optimization. Use the bisection method to find the root correct to 6 decimal places: 3x 3 + x 2 = x + 5 SOLUTION: For the root finding algorithm, we need to rewrite the equation
More informationLecture Note 7: Iterative methods for solving linear systems. Xiaoqun Zhang Shanghai Jiao Tong University
Lecture Note 7: Iterative methods for solving linear systems Xiaoqun Zhang Shanghai Jiao Tong University Last updated: December 24, 2014 1.1 Review on linear algebra Norms of vectors and matrices vector
More information5. Orthogonal matrices
L Vandenberghe EE133A (Spring 2017) 5 Orthogonal matrices matrices with orthonormal columns orthogonal matrices tall matrices with orthonormal columns complex matrices with orthonormal columns 5-1 Orthonormal
More informationNonlinear Optimization
Nonlinear Optimization (Com S 477/577 Notes) Yan-Bin Jia Nov 7, 2017 1 Introduction Given a single function f that depends on one or more independent variable, we want to find the values of those variables
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationAMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences)
AMS526: Numerical Analysis I (Numerical Linear Algebra for Computational and Data Sciences) Lecture 1: Course Overview; Matrix Multiplication Xiangmin Jiao Stony Brook University Xiangmin Jiao Numerical
More informationLECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION
15-382 COLLECTIVE INTELLIGENCE - S19 LECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION TEACHER: GIANNI A. DI CARO WHAT IF WE HAVE ONE SINGLE AGENT PSO leverages the presence of a swarm: the outcome
More informationLecture 9: Krylov Subspace Methods. 2 Derivation of the Conjugate Gradient Algorithm
CS 622 Data-Sparse Matrix Computations September 19, 217 Lecture 9: Krylov Subspace Methods Lecturer: Anil Damle Scribes: David Eriksson, Marc Aurele Gilles, Ariah Klages-Mundt, Sophia Novitzky 1 Introduction
More informationParallel Numerics, WT 2016/ Iterative Methods for Sparse Linear Systems of Equations. page 1 of 1
Parallel Numerics, WT 2016/2017 5 Iterative Methods for Sparse Linear Systems of Equations page 1 of 1 Contents 1 Introduction 1.1 Computer Science Aspects 1.2 Numerical Problems 1.3 Graphs 1.4 Loop Manipulations
More informationMATH 304 Linear Algebra Lecture 19: Least squares problems (continued). Norms and inner products.
MATH 304 Linear Algebra Lecture 19: Least squares problems (continued). Norms and inner products. Orthogonal projection Theorem 1 Let V be a subspace of R n. Then any vector x R n is uniquely represented
More informationScientific Computing II
Technische Universität München ST 008 Institut für Informatik Dr. Miriam Mehl Scientific Computing II Final Exam, July, 008 Iterative Solvers (3 pts + 4 extra pts, 60 min) a) Steepest Descent and Conjugate
More informationKrylov Subspace Methods that Are Based on the Minimization of the Residual
Chapter 5 Krylov Subspace Methods that Are Based on the Minimization of the Residual Remark 51 Goal he goal of these methods consists in determining x k x 0 +K k r 0,A such that the corresponding Euclidean
More informationFurther Mathematical Methods (Linear Algebra)
Further Mathematical Methods (Linear Algebra) Solutions For The 2 Examination Question (a) For a non-empty subset W of V to be a subspace of V we require that for all vectors x y W and all scalars α R:
More informationBasic concepts in Linear Algebra and Optimization
Basic concepts in Linear Algebra and Optimization Yinbin Ma GEOPHYS 211 Outline Basic Concepts on Linear Algbra vector space norm linear mapping, range, null space matrix multiplication terative Methods
More informationKRYLOV SUBSPACE ITERATION
KRYLOV SUBSPACE ITERATION Presented by: Nab Raj Roshyara Master and Ph.D. Student Supervisors: Prof. Dr. Peter Benner, Department of Mathematics, TU Chemnitz and Dipl.-Geophys. Thomas Günther 1. Februar
More informationGradient Based Optimization Methods
Gradient Based Optimization Methods Antony Jameson, Department of Aeronautics and Astronautics Stanford University, Stanford, CA 94305-4035 1 Introduction Consider the minimization of a function J(x) where
More informationConjugate Gradient Method
Conjugate Gradient Method direct and indirect methods positive definite linear systems Krylov sequence spectral analysis of Krylov sequence preconditioning Prof. S. Boyd, EE364b, Stanford University Three
More informationComputational Linear Algebra
Computational Linear Algebra PD Dr. rer. nat. habil. Ralf Peter Mundani Computation in Engineering / BGU Scientific Computing in Computer Science / INF Winter Term 2017/18 Part 2: Direct Methods PD Dr.
More informationDesigning Information Devices and Systems I Spring 2018 Lecture Notes Note 25
EECS 6 Designing Information Devices and Systems I Spring 8 Lecture Notes Note 5 5. Speeding up OMP In the last lecture note, we introduced orthogonal matching pursuit OMP, an algorithm that can extract
More informationLecture 7 Conjugate Gradient Method(CG)
Lecture 7 Conjugate Gradient Method(CG) Jinn-Liang Liu 2017/6/7 The CG method is used for solving symmetric and positive definite(spd) systems A x = b (7.1) with A = A T, (7.2) x T A x > 0 forallnon-zerovectors
More informationLinear Algebra. Session 12
Linear Algebra. Session 12 Dr. Marco A Roque Sol 08/01/2017 Example 12.1 Find the constant function that is the least squares fit to the following data x 0 1 2 3 f(x) 1 0 1 2 Solution c = 1 c = 0 f (x)
More informationMATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors.
MATH 304 Linear Algebra Lecture 20: The Gram-Schmidt process (continued). Eigenvalues and eigenvectors. Orthogonal sets Let V be a vector space with an inner product. Definition. Nonzero vectors v 1,v
More informationStatistical Geometry Processing Winter Semester 2011/2012
Statistical Geometry Processing Winter Semester 2011/2012 Linear Algebra, Function Spaces & Inverse Problems Vector and Function Spaces 3 Vectors vectors are arrows in space classically: 2 or 3 dim. Euclidian
More informationAM 205: lecture 8. Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition
AM 205: lecture 8 Last time: Cholesky factorization, QR factorization Today: how to compute the QR factorization, the Singular Value Decomposition QR Factorization A matrix A R m n, m n, can be factorized
More informationVector Spaces, Orthogonality, and Linear Least Squares
Week Vector Spaces, Orthogonality, and Linear Least Squares. Opening Remarks.. Visualizing Planes, Lines, and Solutions Consider the following system of linear equations from the opener for Week 9: χ χ
More information1. Subspaces A subset M of Hilbert space H is a subspace of it is closed under the operation of forming linear combinations;i.e.,
Abstract Hilbert Space Results We have learned a little about the Hilbert spaces L U and and we have at least defined H 1 U and the scale of Hilbert spaces H p U. Now we are going to develop additional
More informationChapter 4 Euclid Space
Chapter 4 Euclid Space Inner Product Spaces Definition.. Let V be a real vector space over IR. A real inner product on V is a real valued function on V V, denoted by (, ), which satisfies () (x, y) = (y,
More informationElementary linear algebra
Chapter 1 Elementary linear algebra 1.1 Vector spaces Vector spaces owe their importance to the fact that so many models arising in the solutions of specific problems turn out to be vector spaces. The
More information