Experiments with iterative computation of search directions within interior methods for constrained optimization

Transcription

1 Experiments with iterative computation of search directions within interior methods for constrained optimization Santiago Akle and Michael Saunders Institute for Computational Mathematics and Engineering (ICME) Stanford University Twelfth Copper Mountain Conference on Iterative Methods Copper Mountain, Colorado March 25 30, 2012 Akle and Saunders, ICME, Stanford Copper Mountain /31

2 Outline 1 PDCO 2 Iterative solvers 3 colnorms, QR 4 Partial Cholesky or QR 5 Numerical results Akle and Saunders, ICME, Stanford Copper Mountain /31

3 Abstract Our primal-dual interior-point optimizer PDCO has found many applications for optimization problems of the form min ϕ(x) st Ax = b, l x u, in which ϕ(x) is convex and A is a sparse matrix or a linear operator. We focus on the latter case and the need for iterative methods to compute dual search directions from linear systems of the form AD 2 A T y = r, D diagonal and positive definite. Although the systems are positive definite, they do not need to be solved accurately and there is reason to use MINRES rather than CG (see PhD thesis of David Fong (2011)). When the original problem is regularized, the systems can be converted to least-squares problems and there is similar reason to use LSMR rather than LSQR. Also, D becomes increasingly ill-conditioned as the interior method proceeds and there is need for some kind of preconditioning, such as the partial Cholesky approach of Bellavia, Gondzio and Morini (2011). We present numerical results on matters such as these. Akle and Saunders, ICME, Stanford Copper Mountain /31

4 PDCO Primal-Dual Interior Method minimize c T x x subject to Ax = b, x 0, PDCO is a Matlab solver for such problems A may be a sparse matrix or a linear operator Akle and Saunders, ICME, Stanford Copper Mountain /31

5 PDCO Primal-Dual Interior Method minimize c T x + 1 x, r 2 γx r 2 subject to Ax + δr = b, x 0, γ and δ 10 4 for linear programs δ = 1 for nonnegative least-squares PDCO is a Matlab solver for such problems A may be a sparse matrix or a linear operator Akle and Saunders, ICME, Stanford Copper Mountain /31

6 Primal-Dual Interior Method PDCO solves a sequence of nonlinear equations Ax + δ 2 y = b A T y + z = c + γ 2 x Xz = µe X = diag(x) µ 0 Newton s method: A δ 2 I γ 2 I A T I Z X x y = z r 1 r 2 r 3 Akle and Saunders, ICME, Stanford Copper Mountain /31

7 PDCO search direction Define D 2 = (X 1 Z + γ 2 I ) 1 Posdef diagonal with big and small elements Solve either ( AD 2 A T + δ 2 I ) y = AD 2 r 4 + r 1 or ( min DA T δi ) y ( ) Dr4 2 r 1 /δ D changes each PDCO iteration Increasingly ill-conditioned Akle and Saunders, ICME, Stanford Copper Mountain /31

8 Iterative solvers spd systems least squares Akle and Saunders, ICME, Stanford Copper Mountain /31

9 Iterative solvers for PDCO ( AD 2 A T + δ 2 I ) y = AD 2 r 4 + r 1 CG or MINRES ( min DA T δi ) y ( ) Dr4 2 r 1 /δ LSQR or LSMR Akle and Saunders, ICME, Stanford Copper Mountain /31

10 Iterative solvers for PDCO ( AD 2 A T + δ 2 I ) y = AD 2 r 4 + r 1 CG or MINRES ( min DA T δi ) y ( ) Dr4 2 r 1 /δ LSQR or LSMR Comparisons: Fong and S 2011a,b Fong thesis 2011 Akle and Saunders, ICME, Stanford Copper Mountain /31

11 Backward errors for CG and MINRES on spd Ax = b (A + E k )x k = b r k = b Ax k E k = r kx T k x k 2 E k = r k x k Akle and Saunders, ICME, Stanford Copper Mountain /31

12 Backward errors for CG and MINRES on spd Ax = b (A + E k )x k = b r k = b Ax k E k = r kx T k x k 2 E k = r k x k We know r k for MINRES (but not for CG) x k for CG (Steihaug 1983) x k for MINRES (Fong 2011) Akle and Saunders, ICME, Stanford Copper Mountain /31

13 Backward errors for CG and MINRES on spd Ax = b (A + E k )x k = b r k = b Ax k E k = r kx T k x k 2 E k = r k x k We know r k for MINRES (but not for CG) x k for CG (Steihaug 1983) x k for MINRES (Fong 2011) Plot log 10 E k for CG and MINRES Data: Tim Davis s sparse matrix collection Real spd examples Ax = b that include b Akle and Saunders, ICME, Stanford Copper Mountain /31

14 Backward errors for CG and MINRES when A Name:Simon_raefsky4, Dim:19779x19779, nnz: , id=7 CG MINRES 0 1 Name:Cannizzo_sts4098, Dim:4098x4098, nnz:72356, id=13 CG MINRES log( r / x ) 8 log( r / x ) iteration count x iteration count Name:Schenk_AFE_af_shell8, Dim:504855x504855, nnz: , id=11 1 CG MINRES 2 Name:BenElechi_BenElechi1, Dim:245874x245874, nnz: , id=22 0 CG MINRES log( r / x ) log( r / x ) iteration count iteration count x 10 4 Akle and Saunders, ICME, Stanford Copper Mountain /31

15 Backward errors for LSQR and LSMR Stewart (1977): min (A + E k )x k b E k = r kr T k A r k 2 r k = b Ax k E k = AT r k r k Akle and Saunders, ICME, Stanford Copper Mountain /31

16 Backward errors for LSQR and LSMR Stewart (1977): min (A + E k )x k b E k = r kr T k A r k 2 r k = b Ax k E k = AT r k r k LSQR CG on A T Ax = A T b LSMR MINRES on A T Ax = A T b Hence A T r k for LSMR (but not for LSQR) Akle and Saunders, ICME, Stanford Copper Mountain /31

17 Measure of convergence r k = b Ax k r k ˆr, A T r k 0 min Ax b Akle and Saunders, ICME, Stanford Copper Mountain /31

18 Measure of convergence r k = b Ax k r k ˆr, A T r k 0 min Ax b LSQR r k LSQR log A T r k 1.01 Name:lp fit1p, Dim:1677x627, nnz:9868, id=625 0 Name:lp fit1p, Dim:1677x627, nnz:9868, id= r log A T r iteration count iteration count Akle and Saunders, ICME, Stanford Copper Mountain /31

19 min Ax b Measure of convergence r k = b Ax k r k ˆr, A T r k 0 r k LSQR LSMR log A T r k 1 Name:lp fit1p, Dim:1677x627, nnz:9868, id=625 LSQR LSMR 0 Name:lp fit1p, Dim:1677x627, nnz:9868, id=625 LSQR LSMR r log A T r iteration count iteration count Akle and Saunders, ICME, Stanford Copper Mountain /31

20 Two linear algebra tools colnorms.m Householder QR Akle and Saunders, ICME, Stanford Copper Mountain /31

21 colnorms.m C = = [c 1,..., c n ] (m n) Estimates all c j from p products C T v p n Each v = randn(m,1) If A = C T C, we estimate diag(a) from colnorms(c) 2 Akle and Saunders, ICME, Stanford Copper Mountain /31

22 Householder QR W = = Q ( ) R = ( Y Z ) ( ) R = YR (n k) 0 0 Q = product of( Householder ) transformations ( ) I O Y = Q Z = Q 0 I Q, Y, Z are fast operators if k is small Choose W to be approximate eigenvectors (say) Householder QR orthogonalizes W, represents full Q Akle and Saunders, ICME, Stanford Copper Mountain /31

23 Partial Cholesky or partial QR Akle and Saunders, ICME, Stanford Copper Mountain /31

24 Two ideas for spd Hx = b Part direct, part iterative (hybrid!) Schur complement CG Axelsson 1994 Partial Cholesky preconditioning Bellavia, Gondzio, Morini 2011 Akle and Saunders, ICME, Stanford Copper Mountain /31

25 Two ideas for spd Hx = b Part direct, part iterative (hybrid!) Schur complement CG Axelsson 1994 Partial Cholesky preconditioning Bellavia, Gondzio, Morini 2011 Transform first: Q T HQy = Q T b Q = Householder QR on W W = 10, 20, 50,... approximate eigenvectors of H Two-level (subspace splitting) Schur complement CG Hanke and Vogel 1999 Subspace preconditioned LSQR Jacobsen, Hansen, and S 2003 = partial QR equivalent of partial Cholesky Akle and Saunders, ICME, Stanford Copper Mountain /31

26 Partial Cholesky preconditioning for spd Hx = b Q = [Y Z], Y is n k for small k BGM 2011: Q = permutation (approx diagonal pivoting) Our experiments: Q is from Householder QR on W k eigenvectors Q T HQ = M = ( ) ( ) ( L1 I L T 1 L T ) 2 L 2 I S I ( ) ( ) ( L1 I L T 1 L T ) 2 L 2 I S I S = diag(s) or colnorms gives S diag(s) Akle and Saunders, ICME, Stanford Copper Mountain /31

27 Numerical Results PDCO on LPs, satellite image LP degen LP fit2p Satellite Akle and Saunders, ICME, Stanford Copper Mountain /31

28 MINRES on ( AD 2 A T + δ 2 I ) y = AD 2 r 4 + r 1 at PDCO iteration 1, middle, end Think of systems as Hx = b QHQ T y = Q T b Preconditioners constructed from k = 50 cols of partial Cholesky Q = permutation for diagonal pivoting or Householder QR on k approx eigenvectors Akle and Saunders, ICME, Stanford Copper Mountain /31

29 MINRES on ( AD 2 A T + δ 2 I ) y = AD 2 r 4 + r 1 at PDCO iteration 1, middle, end Think of systems as Hx = b QHQ T y = Q T b Preconditioners constructed from k = 50 cols of partial Cholesky Q = permutation for diagonal pivoting or Householder QR on k approx eigenvectors Schur complement S = C T C approximated by diag(s) or colnorms(c) Akle and Saunders, ICME, Stanford Copper Mountain /31

30 LP degen3, PDCO itn 1 Akle and Saunders, ICME, Stanford Copper Mountain /31

31 LP degen3, PDCO itn 15 Akle and Saunders, ICME, Stanford Copper Mountain /31

32 LP degen3, PDCO itn 32 Akle and Saunders, ICME, Stanford Copper Mountain /31

33 LP fit2p, PDCO itn 1 Akle and Saunders, ICME, Stanford Copper Mountain /31

34 LP fit2p, PDCO itn 12 Akle and Saunders, ICME, Stanford Copper Mountain /31

35 LP fit2p, PDCO itn 26 Akle and Saunders, ICME, Stanford Copper Mountain /31

36 Satellite, PDCO itn 1 Akle and Saunders, ICME, Stanford Copper Mountain /31

37 Satellite, PDCO itn 9 Akle and Saunders, ICME, Stanford Copper Mountain /31

38 Satellite, PDCO itn 17 Akle and Saunders, ICME, Stanford Copper Mountain /31

39 References O. Axelsson (1994), Iterative Solution Methods, Cambridge Univ Press M. Hanke and C. R. Vogel (1999), Two-level preconditioners for regularized inverse problems, Numer Math 83, M. Jacobsen, P. C. Hansen, and M. A. Saunders (2003), Subspace preconditioned LSQR for discrete ill-posed problems, BIT 43:5, S. Bellavia, J. Gondzio, and B. Morini (Jul 2011), New matrix-free preconditioner for sparse least-squares problems, Report ERGO , University of Edinburgh D. C.-L. Fong and M. A. Saunders (Aug 2011), LSMR: An iterative algorithm for sparse least-squares problems, SISC 33, D. C.-L. Fong and M. A. Saunders (Dec 2011), CG versus MINRES: An empirical comparison, for SQU Journal for Science D. C.-L. Fong (Dec 2011), Minimum-Residual Methods for Sparse Least-Squares Using Golub-Kahan Bidiagonalization, PhD thesis, ICME, Stanford University Akle and Saunders, ICME, Stanford Copper Mountain /31

40 Special thanks to Sven Leyffer Conferences really do promote action and creativity! Akle and Saunders, ICME, Stanford Copper Mountain /31