Iterative Methods for Smooth Objective Functions

Transcription

1 Optimization Iterative Methods for Smooth Objective Functions Quadratic Objective Functions Stationary Iterative Methods (first/second order) Steepest Descent Method Landweber/Projected Landweber Methods Conjugate Gradient Method Non-Quadratic Smooth Objective Functions Conjugate Gradient Method Newton s Method Trust Region Globalization of Newton s Method BFGS Method IPIM, IST, José Bioucas,

2 References [1] O. Axelsson, Iterative Solution Methods. New York: Cambridge Univ. Press, [2] Golub, G.H. and Van Loan, C.F., Matrix Computations, Johns Hopkins University Press, Baltimore, Maryland, [3] C. Byrne, A unified treatment of some iterative algorithms in signal processing and image reconstruction, Inverse Problems, vol. 20, pp , IPIM, IST, José Bioucas,

3 Rates of convergence Suppose that as Linear convergence rate: there exits a constant for which Superlinear convergence rate: there exits a sequence of real numbers such that and Quadratic convergence rate: there exits a constant for which IPIM, IST, José Bioucas,

4 10 0 linear Rates of convergence: example superlinear quadratic IPIM, IST, José Bioucas,

5 Comparing linear convergence rates Many iterative methods for large scale inverse problems have linear converge rate: convergence factor r - log 10 convergence rate number of iterations to reduce the error by a factor of 10 IPIM, IST, José Bioucas,

6 Induced norms and spectral radius Given a vector norm, the matrix norm induced by the vector norm is When the vector norm is the Euclidian norm, the induced norm is termed the spectral norm and is given by If is Hermitian, the matrix norm is given by the spectral radius of A, IPIM, IST, José Bioucas,

7 Key results involving the spectral radius IPIM, IST, José Bioucas,

8 Tikhonov regularization/gaussian priors Assume that is non-singular. Then The solution is obtained by solving the system IPIM, IST, José Bioucas,

9 Stationary iterative methods Consider the system, where is non-singular First Order Stationary Iterative Methods Let be a splitting of is nonsingular for must be ease to invert Jacobi Gauss-Seidel IPIM, IST, José Bioucas,

10 Stationary iterative methods Frequently, we can not access to the elements of A or D, but only apply these operators. Thus C should depend only on these operators Example 1: Landweber iterations Example 2: Easy to compute when D is diagonal or a convolution IPIM, IST, José Bioucas,

11 First order stationary iterative methods: convergence Consider the system Let be a splitting of is nonsingular and Then iff IPIM, IST, José Bioucas,

12 First order stationary iterative methods: convergence Consider the system Let be a splitting of for Convergence iff IPIM, IST, José Bioucas,

13 First order stationary iterative methods (cont.) Ill-conditioned systems Number of iterations to attenuate the error norm by 10 Landweber C = I Under what conditions? The eigenvalues of tend to be less spread than those of IPIM, IST, José Bioucas,

14 Second order stationary iterative methods: convergence Consider the system Let be a splitting of Convergence [1] iff IPIM, IST, José Bioucas,

15 First/second order stationary iterative methods: comparison Ill-conditioned systems First order Second order Example Second order is 100 times faster IPIM, IST, José Bioucas,

16 Steepest descent method non-stationary first order iterative method Optimal (line search) Convergence IPIM, IST, José Bioucas,

17 Conjugate gradient method Consider the system Equivalently Are conjugate with respect to if Let be a sequence of n mutualy conjugate directions and Since Then and IPIM, IST, José Bioucas,

18 Conjugate gradient method as an iterative method Computing the solution of is equivalent to minimize 2- Define to as the projection error of onto the direction 1- minimize along conjugate directions directions IPIM, IST, José Bioucas,

19 Conjugate gradient and steepest descent paths steepest descent conjugate gradient IPIM, IST, José Bioucas,

20 The resulting algorithm ( denotes the negative of the gradient) IPIM, IST, José Bioucas,

21 Some remarks about the CG method Convergenge [2] IPIM, IST, José Bioucas,

22 Comparison: CG and First/Second Order Stationary Iterative Methods st order nd order 10 0 CG IPIM, IST, José Bioucas,

23 Preconditioned conjugate gradient (PCG) method Let be a s.p.d matrix such that The eigenvalues of are more clustered than those of CG solves the system system faster than the Note: PCG can be written as a small modification of CG: The complexity of each PCG iteration is that of CG plus the computation of IPIM, IST, José Bioucas,

24 Constrained Tikhonov regularization/gaussian priors where is a closed convex set Projection onto a convex set is non-expansive IPIM, IST, José Bioucas,

25 Projected iterations is a closed convex set Let be a contraction mapping Assume that sequence generated by solution of the unconstrained problem converges to the Define the operator: is a contraction mapping for any starting element, the sequence of sucessive approximations is convergent and its limit is the unique fixed point of the unique fixed point of is the solution of the constrained optimization problem IPIM, IST, José Bioucas,