LSTRS 1.2: MATLAB Software for Large-Scale Trust-Regions Subproblems and Regularization

Transcription

1 LSTRS 1.2: MATLAB Software for Large-Scale Trust-Regions Subproblems and Regularization Marielba Rojas Informatics and Mathematical Modelling Technical University of Denmark Computational Methods with Applications Harrachov, Czech Republic August 19-25, 2007

2 Joint work with Sandra A. Santos, Campinas, Brazil Danny C. Sorensen, Rice, USA Special thanks to Wake Forest, CERFACS, and T.U. Delft.

3 Outline The Trust-Region Subproblem LSTRS - The basic idea LSTRS - The Algorithm LSTRS - The Software Comparisons Applications

4 Basic Idea

5 Trust-Region Subproblem min s.t. 1 2 xt Hx + g T x x H IR n n,h=h T,nlarge g IR n,g 0 >0

6 Regularization Problem min s.t. 1 Ax b 2 2 x A IR m n,m nlarge, from ill-posed problem b IR m, containing noise, and A T b 0 >0

7 Trust-Region Subproblem Characterization of solutions. Gay 1981, Sorensen x with x is a solution of TRS with Lagrange multiplier λ, if and only if (i) (H λ I)x = g. (ii) H λ I positive semidefinite. (iii) λ 0. (iv) λ ( x )=0.

8 TRS as Parameterized Eigenvalue Problem Consider the bordered matrix B α = α g gt H Then Eigenvalues of H interlace eigenvalues of B α αsuch that TRS equivalent to 1 min 2y T B α y s.t. y T y 1+ 2 e T 1 y=1

9 LSTRS Note α g gt H 1 x = λ 1 x α λ = g T x (H λi)x = g Let H = Qdiag(δ 1,δ 2,...,δ n ) Q T and γ i = Q T g, i =1,2,...,n Suppose x IR n such that (H λi)x = g. Define φ(λ) = g T x= n i=1 γ 2 i δ i λ Then φ (λ) =x T x. Idea: Adjust α such that α ˆλ = φ(ˆλ) withφ (ˆλ)= 2.

10 LSTRS Compute rational interpolant φ and adjust α such that α ˆλ = φ(ˆλ) with φ (ˆλ) = 2. Obtain interpolation points by solving large-scale eigenvalue problems for smallest eigenvalue of B α. Solve eigenvalue problems with efficient method such as ARPACK (matrix-free, fixed storage).

11 LSTRS in pictures - the standard case (H λi)x = g, α λ = φ(λ) g T x, φ (λ)=x T x. 5 4 φ(λ) 3 φ(λ), α λ 2 1 α * λ φ(λ * ) + 2 λ λ * λ

12 LSTRS in pictures - the (near) hard case (H λi)x = g, α λ = φ(λ) g T x, φ (λ)=x T x φ(λ) φ(λ), α λ α k+1 λ φ(λ) λ k λ k λ 1 k λ

13 LSTRS in pictures - hard case in ill-posed problems

14 Algorithm

15 LSTRS - The Algorithm Input: H IR n n, symmetric; g IR n ; >0; tolerances (ε,ε ν,ε HC,ε α,ε Int ). Output: x, solution to TRS and Lagrange multiplier λ. 1. Initialization 1.1 Compute δ U δ 1, initialize α U, initialize α Compute eigenpairs {λ 1 (α 0 ), (ν 1,u T 1 )T }, {λ i (α 0 ),(ν 2,u T 2 )T } of B α0 1.3 Initialize α L, set k =0 2. repeat 2.1 Adjust α k (might need to compute eigenpairs) 2.2 if g ν 1 >ε ν 1 ν12 then set λ k = λ 1 (α k ) and x k = u 1 ν 1, and update α L or α U. else set λ k = λ i (α k ), x k = u 2 ν 2, and α U = α k 2.3 Compute α k+1 by 1-point (k =0) or 2-point interpolation scheme 2.4 Safeguard α k+1 and set k = k Compute eigenpairs {λ 1 (α k ), (ν 1,u T 1 )T }, {λ i (α k ),(ν 2,u T 2 )T } of B αk until convergence

16 LSTRS - The Algorithm Update α k Adjust α k by rational interpolation until eigenvector with desired structure is obtained Safeguard α k using safeguarding interval Tolerances Convergence (Stopping Criteria) H (Hessian Matrix) Eigensolver

17 ε ε HC LSTRS - The Algorithm Tolerances The desired relative accuracy in the norm of the trust-region solution. A boundary solution x satisfies x ε The desired accuracy of a quasi-optimal solution in the hard case. A quasi-optimal solution ˆx satisfies ψ(x ) ψ(ˆx) ε HC ψ(x ), where ψ(x) = 1 2 xt Hx + g T x,andx isthetruesolution. ε α The minimum relative length of the safeguarding interval for α. ε Int ε ν The interval is too small when α U α L ε α max{ α L, α U } Used to declare that the smallest eigenvalue of B α is positive in the test for an interior solution: λ 1 (α) is considered positive if λ 1 (α) > ε Int The minimum relative size of an eigenvector component. The component ν is small when ν ε ν u g

18 LSTRS - The Algorithm Stopping Criteria 1. Boundary Solution - ε 2. Interior Solution - ε Int 3. Quasi-Optimal Solution (Near Hard Case!) -ε HC 4. Safeguarding interval cannot be further decreased - ε α 5. Maximum number of iterations reached

19 Software

20 LSTRS - The Software Version: 1.2 System: MATLAB 6.0 or higher

21 LSTRS - The Software Front-end routine: LSTRS Iteration: Update of α k : Adjustment of α k : Safeguarding of α k : Eigenproblems: Stopping Criteria: Output: lstrs lstrs method upd param0, upd paramk, interpol1, interpol2, inter point adjust alpha safe alpha1, safe alphak, upd alpha safe b epairs, eig gateway, eigs lstrs, eigs lstrs gateway, tcheigs lstrs gateway convergence, boundary sol, interior sol, quasioptimal sol output

22 LSTRS - The Software General Call [x,lambda,info,moreinfo] =... lstrs(h,g,delta,epsilon,eigensolver,lopts,hpar,eigensolverpar) Simplest Call [x,lambda,info,moreinfo] = lstrs(h,g,delta);

23 LSTRS - The Software Input Parameters char: double: struct: function-handle: H, eigensolver H, g,delta epsilon, lopts, Hpar, eigensolverpar H, eigensolver Output Parameters x: solution vector lambda: info: moreinfo: Lagrange multiplier (Tikhonov parameter) exit condition other exit conditions, matrix-vector products, etc.

24 LSTRS Software: Key Features Two options for Hessian Matrix: H (explicitly) Matrix-Vector Multiplication Routine Several options for Eigensolver: eig eigs lstrs (modified version of eigs that returns more information): this is MATLAB s interface to ARPACK (Implicitly Restarted Arnoldi Method) tcheigs lstrs +Tchebyshev Spectral Transformation user-provided Fixed Storage

25 % % File: simple.m % A simple problem where the Hessian is the Identity matrix. % H = eye(50); g = ones(50,1); mu = -3; % chosen arbitrarily xexact = -ones(50,1)/(1-mu); Delta = norm(xexact); % % The simplest possible calls to lstrs. Default values are used. % [x,lambda,info,moreinfo] = lstrs(h,g,delta); [x,lambda,info,moreinfo] = lstrs(@mv,g,delta);

26 % % File: mv.m % A simple matrix-vector multiplication routine % that computes the Identity matrix times a vector v % function [w] = mv(v,varargin) w=v;

27 >> simple < MATLAB> Problem: no name available. Dimension: 50. Delta: e+00 Eigensolver: eigs lstrs gateway LSTRS iteration: 0 x : e-01, lambda: e+00 x -Delta /Delta: e-01 LSTRS iteration: 1 x : e+00, lambda: e+00 x -Delta /Delta: e-15 Number of LSTRS Iterations: 2 Number of calls to eigensolver: 2 Number of MV products: 18 ( x -Delta)/Delta: e-15 lambda: e+00 g + (H-lambda*I)x / g = e-15 The vector x is a Boundary Solution

28 % % File: regularization.m % Computes a regularized solution to problem phillips from % the Regularization Tools Package by P.C. Hansen % [A,b,xexact] = phillips(300); atamvpar.a = A; g=-a *b; Delta = norm(xexact); lopts.name = phillips ; lopts.plot = y ; lopts.message level = 2; lopts.correction = n ; lopts.interior = n ; epar.k = 2; epar.p = 7; % for a total of 7 vectors (default) [x,lambda,info,moreinfo] =... lstrs(@atamv,g,delta,[ ],@tcheigs lstrs gateway,lopts,atamvpar,epar);

29 % % File: atamv.m % A matrix-vector multiplication routine % that computes A *A*v % The matrix A must be a field of the structure atamvpar % function [w] = atamv(v,atamvpar) w = atamvpar.a*v; w = (w *atamvpar.a) ;

30 < M A T L A B > >> regularization Problem: phillips. Dimension: 300. Delta: e+00 Eigensolver: tcheigs lstrs gateway LSTRS iteration: 0 x : e-01, lambda: e+01 x -Delta /Delta: e-01 LSTRS iteration: 1 x : e+00, lambda: e+01 x -Delta /Delta: e-01 LSTRS iteration: 2 x : e+00, lambda: e-01 x -Delta /Delta: e-02 LSTRS iteration: 3 x : e+00, lambda: e-03 x -Delta /Delta: e-04 Number of LSTRS Iterations: 4 Number of calls to eigensolver: 5 Number of MV products: 342 ( x -Delta)/Delta: e-15 lambda: e-03 g + (H-lambda* I)x / g = e-05 The vector x is a Quasi-optimal Solution

31 LSTRS Solution Exact Solution

32 Comparisons

33 Comparisons SSM - Sequential Subspace Method. Hager SDP - Semidefinite Programming approach. Rendl and Wolkowicz 1997, Fortin and Wolkowicz GLTR - Generalized Lanczos Trust Region method. Gould, Lucidi, Roma, and Toint 1999.

34 Average results for the 2-D Laplacian, n = Easy Case. METHOD MVP STORAGE (H λi)x+g g LSTRS SSM SSM d SDP GLTR

35 Average results for the 2-D Laplacian, n = Hard Case. METHOD MVP STORAGE (H λi)x+g g LSTRS SSM SSM d SDP GLTR

36 Inverse Heat Equation, n = Mildly Ill-Posed. METHOD MVP STORAGE (H λi)x+g g x x IP x IP LSTRS SSM SSM d SDP

37 Inverse Heat Equation, n = Severely Ill-Posed. METHOD MVP STORAGE (H λi)x+g g x x IP x IP LSTRS SSM SSM d SDP

38 Applications

39 y x 45 Bathymetry of the Sea of Galilee. Dimension: Vectors: 5. Matvecs: 206.

40 True image Blurred and noisy image Dimension: Cost: 7 Vectors 3 LSTRS iterations 201 Matvecs LSTRS restoration

41 REFERENCES M. Rojas, S. A. Santos and D. C. Sorensen. A New Matrix-Free Algorithm for the Large-Scale Trust-Region Subproblem. SIAM Journal on Optimization, 11(3): , M. Rojas, S. A. Santos and D. C. Sorensen. Algorithm xxx: LSTRS: MATLAB Software for Large-Scale Trust-Region Subproblems and Regularization. ACM Transactions on Mathematical Software, toappear. mr