UPDATING CONSTRAINT PRECONDITIONERS FOR KKT SYSTEMS IN QUADRATIC PROGRAMMING VIA LOW-RANK CORRECTIONS

Transcription

1 UPDATING CONSTRAINT PRECONDITIONERS FOR KKT SYSTEMS IN QUADRATIC PROGRAMMING VIA LOW-RANK CORRECTIONS STEFANIA BELLAVIA, VALENTINA DE SIMONE, DANIELA DI SERAFINO, AND BENEDETTA MORINI Abstract This wor focuses on the iterative solution of sequences of KKT linear systems arising in interior point methods applied to large convex quadratic programming problems This tas is the computational core of the interior point procedure and an efficient preconditioning strategy is crucial for the efficiency of the overall method Constraint preconditioners are very effective in this context; nevertheless, their computation may be very expensive for large-scale problems and resorting to approximations of them may be convenient Here we propose a procedure for building inexact constraint preconditioners by updating a seed constraint preconditioner computed for a KKT matrix at a previous interior point iteration These updates are obtained through low-ran corrections of the Schur complement of the (1,1) bloc of the seed preconditioner The updated preconditioners are analyzed both theoretically and computationally The results obtained show that our updating procedure, coupled with an adaptive strategy for determining whether to reinitialize or update the preconditioner, can enhance the performance of interior point methods on large problems Key words KKT systems, constraint preconditioners, matrix updates, convex quadratic programming, interior point methods AMS subject classifications 65F08, 65F10, 90C20, 90C51 1 Introduction Second-order methods for constrained and unconstrained optimization require, at each iteration, the solution of a system of linear equations For large-scale problems it is common to solve each system by an iterative method coupled with a suitable preconditioner Since the computation of a preconditoner for each linear system can be very expensive, in recent years there has been a growing interest in reducing the cost for preconditioning by sharing some computational effort through subsequent linear systems Updating preconditioner framewors for sequences of linear systems have a common feature: based on information generated during the solution of a linear system in the sequence, a preconditioner for a subsequent system is generated An efficient updating procedure is expected to build a preconditioner which is less effective in terms of linear iterations than the one computed from scratch, but more convenient in terms of cost for the overall linear algebra phase The approaches existing in literature can be broadly classified as: limited-memory quasi-newton preconditioners for symmetric positive definite and nonsymmetric matrices (see, eg, [10, 36, 41]), recycled Krylov information preconditioners for symmetric and nonsymmetric matrices (see, eg, [18, 30, 32, 38]), updates of factorized preconditioners for symmetric positive definite and nonsymmetric matrices (see, eg, [3, 4, 5, 6, 7, 17, 25, 40]) In this paper we study the problem of preconditioning sequences of KKT systems Wor partially supported by INdAM-GNCS, under the 2013 Projects Strategie risolutive per sistemi lineari di tipo KKT con uso di informazioni strutturali and Metodi numerici e software per l ottimizzazione su larga scala con applicazioni all image processing Dipartimento di Ingegneria Industriale, Università degli Studi di Firenze, viale Morgagni 40, Firenze, Italy, stefaniabellavia@unifiit, benedettamorini@unifiit Dipartimento di Matematica e Fisica, Seconda Università degli Studi di Napoli, viale A Lincoln 5, Caserta, Italy, valentinadesimone@unina2it, danieladiserafino@unina2it Istituto di Calcolo e Reti ad Alte Prestazioni, CNR, via P Castellino 111, Napoli, Italy 1

2 arising in the solution of the convex quadratic programing (QP) problem (11) minimize 1 2 xt Qx + c T x, subject to A 1 x s = b 1, A 2 x = b 2, x + v = u, (x, s, v) 0 by Interior Point (IP) methods [34, 46] Here Q R n n is symmetric positive semidefinite, and A 1 R m1 n, A 2 R m2 n, with m = m 1 + m 2 n Note that s and v are slac variables, used to transform the inequality constraints A 1 x b 1 and x u into equality constraints The updating strategy proposed here concerns constraint preconditioners (CPs) [11, 13, 19, 22, 31, 37, 42] Using the factorization of a seed CP computed for some KKT matrix of the sequence, the factorization of an approximate CP is built for subsequent systems The resulting preconditioner is a special case of inexact CP [12, 26, 39, 42, 44] and is intended to reduce the computational cost while preserving effectiveness To the best of our nowledge, this is the first attempt to study, both theoretically and numerically, a preconditioner updating technique for sequences of KKT systems arising from (11) A computational analysis of the reuse of CPs in consecutive IP steps, which can be regarded as a limit case of preconditioner updating, has been presented in [15] Concerning preconditioner updates for sequences of systems arising from IP methods, we are aware of the wor in [1, 45], where linear programming problems are considered In this case, the KKT linear systems are reduced to the so-called normal equation form; some systems of the sequence are solved by the Cholesy factorization, while the remaining ones are solved by the Conjugate Gradient method preconditioned with a low-ran correction of the last computed Cholesy factor Motivated by [1, 45], in this wor we adapt the low-ran corrections given therein to our problem In our approach the updated preconditioner is an inexact CP where the Schur complement of the (1,1) bloc is replaced by a low-ran modification of the corresponding Schur complement in the seed preconditioner The validity of the proposed procedure is supported by a spectral analysis of the preconditioned matrix and by numerical results illustrating its performance The paper is organized as follows In Section 2 we provide preliminaries on CPs and new spectral analysis results for general inexact CPs In Section 3 we present our updating procedure and specialize the spectral analysis conducted in the previous section to the updated preconditioners In Section 4 we discuss implementation issues of the updating procedure and present numerical results obtained by solving sequences of linear systems arising in the solution of convex QP problems These results show that our updating technique is able to reduce the computational cost for solving the overall sequence whenever the updating strategy is performed in conjunction with adaptive strategies for determining whether to recompute the seed preconditioner or update the current one In the following, denotes the vector or matrix 2-norm and, for any symmetric matrix A, λ min (A) and λ max (A) denote its minimum and maximum eigenvalues Finally, for any complex number α, R(α) and I(α) denote the real and imaginary parts of α, respectively 2 Inexact Constraints Preconditioners In this section we discuss the features of the KKT matrices arising in the solution of problem (11) by IP methods and present spectral properties of both CPs and inexact CPs This analysis will be used 2

3 to develop our ] updating strategy Throughout the paper we assume that the matrix A = [ A1 A 2 R m n is full ran The application of an IP method to problem (11) gives rise to a sequence of symmetric indefinite matrices differing by a diagonal, possibly indefinite, matrix In fact, at the th IP iteration, the KKT matrix taes the form A = [ Q + Θ (1) A T A Θ (2) where Θ (1) R n n is diagonal positive definite and Θ (2) R m m is diagonal positive semidefinite In particular, Θ (1) Θ (2) = ] = X 1 W + V 1 T, [ ] Y 1 S 0, 0 0 where (x, w ), (s, y ), and (v, t ) are the pairs of complementary variables of problem (11) evaluated at the current iteration, and X, W, S, Y, V and T are the corresponding diagonal matrices according to the standard IP notation If the QP problem has no linear inequality constraints then Θ (2) is the zero matrix; otherwise Θ (2) admits positive diagonal entries corresponding to slac variables for linear inequality constraints To simplify the notation, in the rest of the paper we drop the iteration index from A, Θ (1) and Θ (2) Hence, A = A (21), and the CP for A is given by (22) [ G A T P ex = A Θ (2) ], where G is an approximation to Q + Θ (1) Clearly, the application of P ex requires its factorization In order to define P ex, one possibility is to specify G and then explicitly factorize the preconditioner; alternatively, implicit factorizations can be used, where P ex is derived from specially chosen factors M and C in P ex = MCM T (see, eg, [23, 24]) We follow the former approach and choose G as the diagonal matrix with the same diagonal entries as Q + Θ (1), ie, (23) G = diag(q + Θ (1) ) The resulting matrix P 1 ex A has an eigenvalue at 1 with multiplicity 2m p, with p = ran(θ (2) ), and n m + p real positive eigenvalues such that the better G approximates Q + Θ (1) the more clustered around 1 they are [22, 37] The factorization of P ex can be obtained by computing a Cholesy-lie factorization of the negative Schur complement, S, of G in A, (24) S = AG 1 A T + Θ (2), 3

4 and using the bloc decomposition [ I P ex = n 0 (25) AG 1 I m ] [ G 0 0 S ] [ In G 1 A T 0 I m where I r is the identity matrix of dimension r, for any integer r In problems where a large part of the computational cost for solving the linear system depends on the computation of a Cholesy-lie factorization of S, this matrix may be replaced by a computationally cheaper approximation of it [26, 39, 42] This yields the following inexact CP: [ P inex = where S inex is the approximation of S I n 0 ] [ G 0 ] [ In G 1 A T AG 1 I m 0 S inex 0 I m 21 Analysis of the eigenvalues Due to the indefiniteness of P 1 inex A and the specific form of P inex, the preconditioned linear system must be solved by either nonsymmetric solvers, such as GMRES [43] and QMR [27], or by the simplified version of the QMR method [28]; such a simplified solver, named SQMR, is capable of exploiting the symmetry of A and P inex and is used in our numerical experiments Although the convergence of many Krylov solvers is not fully characterized by the spectral properties of the coefficient matrix, in many practical cases it depends to a large extent on the eigenvalue distribution For this reason, exploiting the spectral A made in the general context of saddle point problems [8, 9, 44], we provide bounds on the eigenvalues which will guide our preconditioner updates In order to carry out our analysis we write the eigenvalue problem for the matrix analysis of P 1 inex P 1 inex A in the form [ Q + Θ (1) (26) A T A Θ (2) ] [ x y ] [ x = λp inex y Starting from results in [8], in the next theorem we give bounds on the eigenvalues which highlight the dependence of the spectrum of P 1 inexa on that of S 1 inex S Theorem 21 Let λ and [x T, y T ] T be an eigenvalue of problem (26) and a corresponding eigenvector Let ] ], ], (27) X = G 1 2 (Q + Θ (1) )G 1 2 and suppose that 2I n X is positive definite Let (28) (29) if Θ (2) 0, and (210) (211) λ = λ max (S 1 inex S) max{2 λ min(x), 1}, λ = λ min (S 1 inex S) min{2 λ max(x), 1}, λ = λ max (S 1 inex S) (2 λ min(x)), λ = λ min (S 1 inex S) (2 λ max(x)), otherwise Then, P 1 inexa has at most 2m eigenvalues with nonzero imaginary part, counting conjugates Furthermore, if λ has nonzero imaginary part, then (212) 1 2 (λ min(x) + λ) R(λ) 1 2 (λ max(x) + λ), 4

5 otherwise (213) (214) min{λ min (X), λ} λ max{λ max (X), λ}, for y 0, λ min (X) λ λ max (X), for y = 0 Finally, the imaginary part of λ satisfies (215) I(λ) λ max (S 1 inex AG 1 A T ) I n X Proof Let S inex = RR T be the Cholesy factorization of S inex with R lower triangular With a little algebra, the eigenvalue problem (26) becomes [ ] [ ] [ ] [ ] X Y T u In 0 u = λ, Y Z v 0 I m v where X is the matrix in (27) and (216) Y = R 1 AG 1 2 (In X), Z = R 1 AG 1 2 (2In X)G 1 2 A T R T + R 1 Θ (2) R T, [ ] [ 1 ] [ ] u G 2 G = 1/2 A T x v 0 R T y (see [8]) From [8, Proposition 23] it follows that P 1 inexa has at most 2m eigenvalues with nonzero imaginary part, counting conjugates Furthermore, [8, Proposition 212] shows that, when Y is full ran and Z is positive semidefinite, if λ has nonzero imaginary part, then (217) otherwise 1 2 (λ min(x) + λ min (Z)) R(λ) 1 2 (λ max(x) + λ max (Z)), (218) (219) min{λ min (X), λ min (Z)} λ max{λ max (X), λ max (Z)}, for v 0, λ min (X) λ λ max (X), for v = 0 In addition, the imaginary part of λ satisfies (220) I(λ) Y The same results hold if Y is ran deficient with Z positive definite on er(y T ) By assumptions, Z is positive definite and hence inequalities (217) (220) hold Furthermore, since v = R T y and R is nonsingular, (214) is equivalent to (219) It remains to prove (212), (213) and (215); we proceed by bounding λ min (Z), λ max (Z) and Y We consider first the case Θ (2) 0 In order to provide an upper bound on λ max (Z), we note that Z = R 1 [ AG 1 2 (Θ (2) ) 1 2 ] [ ] [ 1 ] 2I n X 0 G 2 A T 0 I m (Θ (2) R T ) 1 2 5

6 [ G 1 2 A Let V U be the ran-retaining factorization of T (Θ (2) ) 1 2 ], where V R (n+m) m has orthogonal columns and U R m m is upper triangular and nonsingular Then, S = U T U and [ ] Z = R 1 U T V T 2In X 0 (221) V UR T 0 I m Letting N = R 1 U T, we have λ max (Z) = Z N 2 max{2 λ min (X), 1} Since N T N is similar to S 1 inex S, the upper bounds in (212) and (213), with λ given in (28), follow from the upper bounds in (217) and (218) In order to provide a lower bound on λ min (Z), we observe that (222) 1 λ min (Z) = Z 1 ( [ ] ) 1 N 1 2 V T 2In X 0 V 0 I m By noting that N T N 1 is similar to S 1 S inex and hence N = λ min (S 1 inex S), and by applying the Courant-Fischer minimax characterization (see, eg, [33, Theorem 812]) to the rightmost term in (222), we have that the lower bounds in (212) and (213), with λ given in (29), follow from the lower bounds in (217) and (218) When Θ (2) = 0, inequalities (212) (213) can be derived by assuming that V U is the ran-retaining factorization of G 1 2 A T, with V R n m having orthogonal columns and U R m m upper triangular and nonsingular In this case equality (221) becomes Z = R 1 U T V T (2I n X)V UR T and the thesis follows by reasoning as above Finally, from (216) it follows that (223) Y R 1 AG 1 2 In X = λ max (R 1 AG 1 A T R T ) I n X Inequality (215) follows from (220) and (223) by noting that R 1 AG 1 A T R T and S 1 inex AG 1 A T are similar Remar 21 Note that the assumption on 2I n X in the theorem can be fulfilled by a proper scaling of X enforcing its eigenvalues to be smaller than 2 Furthermore, if Q is diagonal, then X = I n and from (215) it follows that all the eigenvalues of A are real In this case P 1 inexa has at least n unit eigenvalues, with n associated independent eigenvectors of the form [x T, 0 T ] T, and the remaining eigenvalues lie in the interval [λ min (S 1 inex S), λ max(s 1 inex S)] (see also [26]) P 1 inex 6

7 3 Building an inexact constraint preconditioner by updates In this section we design a strategy for updating the CP built for some seed matrix of the KKT sequence The update is based on low-ran corrections and generates inexact constraint preconditioners for subsequent systems Let us consider a KKT matrix A seed generated at some iteration r of the IP procedure, (31) A seed = [ Q + Θ (1) seed A T A Θ (2) seed where Θ (1) seed Rn n is diagonal positive definite and Θ (2) seed Rm m is diagonal positive semidefinite The corresponding seed CP has the form [ H A T ] (32) P seed = A Θ (2), seed where H = diag(q + Θ (1) seed ) Assume that a bloc factorization of P seed has been obtained by computing the Cholesy-lie factorization of the negative Schur complement, S seed, of H in P seed, (33) S seed = AH 1 A T + Θ (2) seed = LDLT, where L is unit lower triangular and D is diagonal positive definite Let A in (21) be a subsequent matrix of the KKT sequence, P ex in (25) the corresponding CP, with G defined as in (23), and S the Schur complement (24) We replace S with a suitable update of S seed, named S upd, obtaining the inexact preconditioner (34) [ P upd = I n 0 ] [ G 0 ] [ In G 1 A T AG 1 I m 0 S upd 0 I m In the remainder of this section we show that specific choices of S upd provide easily computable bounds on the eigenvalues of S 1 upds; we use these bounds, together with the spectral analysis in Section 2, for constructing practical inexact preconditioners by updating techniques For ease of presentation, we consider the cases Θ (2) = 0 and Θ (2) 0 separately 31 Updated preconditioners for Θ (2) = 0 We consider a low-ran update/downdate of S seed = AH 1 A T = LDL T that generates a matrix S upd of the form ], ] (35) S upd = AJ 1 A T = L upd D upd L T upd, where J is a suitably chosen diagonal positive definite matrix, L upd is unit lower triangular and D upd is diagonal positive definite A guideline for choosing J is provided by Theorem 21 and by a result in [1], reported next for completeness Lemma 31 Let B R m n be full ran and let E, F R n n be symmetric and positive definite Then, any eigenvalue λ of (BEB T ) 1 BF B T satisfies (36) λ min (E 1 F ) λ((beb T ) 1 BF B T ) λ max (E 1 F ) 7

8 For any diagonal matrix W IR n, let γ(w ) = (γ 1 (W ),, γ n (W )) T be the vector with entries given by the diagonal entries of W G 1 sorted in nondecreasing order, ie, (37) W ii W ii min = γ 1 (W ) γ 2 (W ) γ n (W ) = max 1 i n G ii 1 i n G ii By Lemma 31, the eigenvalues of S 1 upds can be bounded by using the diagonal entries of JG 1, ie, (38) γ 1 (J) λ(s 1 upd S) γ n(j); then, Theorem 21 implies the following result Corollary 32 Let A, P upd and S upd be the matrices in (21), (34) and (35), respectively, and λ an eigenvalue of P 1 upd A Let γ 1(J) and γ n (J) be defined according to (37) and let X be the matrix in (27) If 2I n X is positive definite, then (212) (214) hold with (39) (310) λ γ n (J) (2 λ min (X)), λ γ 1 (J) (2 λ max (X)) Furthermore, (311) I(λ) γ n (J) I n X The ey issue is to define J such that a good tradeoff can be achieved between the effectiveness of the bounds provided by Corollary 32 and the cost for building the factors L upd and D upd in (35) On the basis of this consideration, following [1] we define S upd as a low-ran correction of S seed of the form S upd = AH 1 A T + Ā KĀT = A ( H 1 + K ) A T, where K IR n n is a diagonal matrix with only q < n nonzero entries on the diagonal, K IR q q is the principal submatrix of K containing these nonzero entries, Ā IRm q is made of the corresponding q columns of A, and J 1 = H 1 + K accounts for major changes from H to G In order to choose K, we consider the vector γ(h), defined according to (37), and the vector of integers l = (l 1,, l n ) T such that l i gives the position of H ii /G ii in γ(h), ie, and define the set γ li (H) = H ii G ii, (312) Γ = {i : n q l i n and γ li (H) > µ γ } {i : 1 l i q 2 and γ li (H) < ν γ }, where µ γ 1, ν γ (0, 1], and q 1 and q 2 are nonnegative integers such that q 1 +q 2 = q Then we define the diagonal matrix K by setting (313) K ii = { G 1 ii H 1 ii, if i Γ, 0, otherwise, 8

9 Algorithm 31 (Building S upd via low-ran update) [L upd,d upd ] = lr update (H, G, A, L, D, q 1, q 2, µ γ, ν γ ) 1: Build the vector γ(h), containing the diagonal entries of HG 1 sorted in nondecreasing order, and the vector l, containing the positions of the diagonal entries of HG 1 in γ(h) 2: Build the set Γ in (312) and the vector C Γ IR q that contains the indices i Γ in increasing order 3: Build the matrix K by defining its diagonal entries as and set Ā = A(:, C Γ) K ss = G 1 jj H 1 jj, j = C Γ(s), s = 1,, q, 4: Compute the factorization L upd D upd L T upd of S upd = S seed + Ā KĀT by updating/downdating the factors L and D and choose K as the principal submatrix of K having as diagonal entries the values K ii with index i Γ Finally, we define Ā as the matrix consisting of the columns of A corresponding to the indices in Γ, ordered as they are in A The matrix S upd is positive definite and S upd S seed is a low-ran matrix if the cardinality of Γ is small, ie, q n In this case the Cholesy-lie factorization of S upd can be conveniently computed by updating or downdating the factorization LDL T of S seed Specifically, an update must be performed if H ii > G ii, and a downdate if H ii < G ii This tas can be accomplished by either using efficient procedures for updating and downdating the Cholesy factorization [21], or by the Shermann- Morrison-Woodbury formula (see, eg, [33, Section 213]) Clearly, once S upd has been factorized, the factorization of P upd is readily available The procedure described so far is summarized in Algorithm 31 and is called lr update It taes in input the matrices H, G and A, the factors L and D of S seed, and the scalars q 1, q 2, µ γ, and ν γ, and returns in output the factors L upd and D upd of S upd Note that we borrow the Matlab notation The previous choice of K implies that the matrix J has the following diagonal entries: { Gii, if i Γ, J ii = H ii, otherwise, and hence (314) (315) γ 1 (J) = min{1, min γ l i (H)} = min{1, γ q2+1(h)}, i/ Γ γ n (J) = max{1, max γ l i (H)} = max{1, γ n q1 (H)} i/ Γ Then, from Corollary 32 it follows that, among the possible choices of J, the one considered here is expected to provide good eigenvalue bounds as long as γ q2 (H) and γ n q1+1(h) are well separated from γ q2+1(h) and γ n q1 (H), respectively We observe that by setting µ γ = ν γ = 1 we allow to include in Γ indices corresponding to values γ li (H) close to 1, which may not change much the bounds provided by (311) 9

10 1 x 10 3 spectrum of A 1 x spectrum of P ex A Im(λ) 0 Im(λ) e+7 4e+7 6e+7 8e+7 10e+7 Re(λ) Re(λ) 1 1 spectrum of P upd A (q1 = q 2 = 25) 05 Im(λ) e Re(λ) Fig 31 Spectra of the matrices A, P 1 ex A, and P 1 upd A with q 1 = q 2 = 25, at the 10th iteration of an IP method applied to problem CVXQP1 (n = 1000, m = 500) The updated preconditioner is built from the Schur complement of the exact preconditioner at the 6th IP iteration and by (212) (214) with (39) (310) Therefore, the use of larger (smaller) values for µ γ (ν γ ) may be effective anyway, while saving computational cost (see Section 4 for further details) Furthermore, a consequence of this low-ran correction strategy is that S S upd has q zero eigenvalues; thus P 1 inex A has 2q unit eigenvalues with geometric multiplicity q [44, Theorem 33] We note that in the limit case q = 0 the set Γ is empty; hence S upd = S seed and γ li (H) = Q ii + (Θ (1) seed ) ii Q ii + Θ (1) ii The element γ li (H) is expected to be close to 1 if (Θ (1) ) ii (Θ (1) seed ) ii is small, while it may significantly differ from 1 if Θ (1) ii tends to zero or infinity, as it happens when the IP iterate approaches an optimal solution where strict complementarity holds We conclude this section showing the spectra of the matrices A, Pex 1 A and P 1 upd A, where A has been obtained by applying an IP solver to problem CVXQP1 from the CUTEst collection [35], with dimensions n = 1000 and m = 500 (see Figure 31) The IP solver, described in Section 4, has been run using P ex, and A is the KKT matrix at the 10th IP iteration The updated preconditioner P upd has been built by updating the matrix A seed obtained at the 6th IP iteration, using q 1 = q 2 = 25 In this case 10

11 γ q2+1 = 185e-3 and γ n q1 = 273e+0 We see that, unlie P ex, P upd moves some eigenvalues from the real to the complex field Nevertheless, P upd tends to cluster the eigenvalues of A around 1, and γ q2+1 and γ n q1 provide approximate bounds on the real and imaginary parts of the eigenvalues of the preconditioned matrix, according to (311) (310) and (314) (315) Of course, this clustering is less effective than the one performed by P ex, but it is useful in several cases, as shown in Section 4 32 Updated preconditioners for Θ (2) 0 The updating strategy described in the previous section can be generalized to the case Θ (2) 0 To this end, we note that the sparsity pattern of Θ (2) does not change throughout the IP iterations and the set L = {i : Θ (2) ii 0} has cardinality equal to the number m 1 of linear (2) inequality constraints in the QP problem Let Θ seed and Θ (2) be the m 1 m 1 diagonal submatrices containing the nonzero diagonal entries of Θ (2) seed and Θ(2), respectively, and let Ĩm be the rectangular matrix consisting of the columns of I m with indices in L Then, we have S seed = AH 1 A T + Θ (2) seed = à H 1 à T, S = AG 1 A T + Θ (2) = à G 1 à T, where à = [ A Ĩm [ ] H 1 0, H 1 = 0 Θ(2) seed ] [ G 1, G 1 0 = 0 Θ(2) ] Analogously, letting (316) we have S upd = AJ 1 A T + Θ (2) upd, S upd = à J 1 à T, where J 1 = [ J Θ(2) upd ], (2) and Θ upd is the m 1 m 1 diagonal submatrix of Θ (2) upd containing its nonzero diagonal entries Thus, we can choose J using the same arguments as in the previous section With a little abuse of notation, let γ( J) = (γ 1 ( J),, γ n+m1 ( J)) be the vector with elements equal to the diagonal entries of J G 1 sorted in nondecreasing order Then, by Lemma 31, the eigenvalues of S 1 upd S satisfy (317) γ 1 ( J) λ(s 1 upd S) γ n+m 1 ( J), and the following result holds Corollary 33 Let A, P upd and S upd be the matrices in (21), (34) and (316), respectively, and λ an eigenvalue of P 1 upd A Let γ 1( J) and γ n+m1 ( J) be the smallest and the largest element of γ( J) and let X be the matrix in (27) If 2I n X is positive definite, then (212) (214) hold with (318) (319) λ γ n+m1 ( J) max{2 λ min (X), 1}, λ γ 1 ( J) min{2 λ max (X), 1} 11

12 Furthermore, (320) I(λ) γ n+m1 ( J) I n X Proof Inequalities (318) and (319) follow directly from Theorem 21 and (317) Since Θ (2) is positive semidefinite, for any vector w R n we have w T S 1 2 upd AG 1 A T S 1 2 upd w wt S 1 2 upd (AG 1 A T + Θ (2) )S upd w Then, by matrix similarity, (321) λ max (S 1 upd AG 1 A T ) λ max (S 1 upd S) and (320) follows by using Theorem 21 and (317) On the basis of the previous results, the generalization of the updating procedure to the case Θ (2) 0 is straightforward If S seed = LDL T, the factorization of P upd can be computed by invoing procedure lr update with input data H, G and à in place of H, G and A 4 Numerical results We tested the effectiveness of our updating procedure by solving sequences of KKT systems arising in the solution of convex QP problems where m n and A is full ran To this end, we implemented lr update within PRQP, a Fortran 90 solver for convex QP problems based on a primal-dual inexact Potential Reduction IP method [13, 16, 19] For comparison purpose, we implemented the preconditioner P ex too, in the form specified in (25) We used the CHOLMOD library [21] to compute the sparse LDL T factorization of S seed and S, and to perform the low-ran updates and downdates required by S upd For the solution of the KKT systems, we developed an implementation of the left-preconditioned SQMR method without loo-ahead [28], taing into account the bloc structure of the system matrices and of the exact and updated preconditioners All the new code was written in Fortran 90, with interfaces to the functions of CHOLMOD, written in C We note that only one matrix-vector product per iteration is performed in our SQMR implementation, except in the last few iterations, where an additional matrix-vector product per iteration is computed to use the residual instead of the preconditioned BCG-residual in the stopping criterion, as in the QMRPACK code [29] This eeps the computational cost per iteration comparable with that of the Conjugate Gradient method We also observe that, although no theoretical convergence estimates are available for SQMR, this method has shown good performance in all our experiments PRQP was run on several test problems, either taen from the CUTEst collection [35] or obtained by modifying CUTEst problems, as explained later in this section The starting point was chosen as explained in [20] and the IP iterations were stopped when the relative duality gap and suitable measures of the primal and dual infeasibilities became lower than 10 7 and 10 8, respectively (see [14] for the details) The zero vector was used as starting guess in SQMR An adaptive criterion was applied to stop the iterations [14], which relates the accuracy in the solution of the KKT system to the quality of the current IP iterate, in the spirit of inexact IP methods [2] A maximum number of 1000 SQMR iterations was considered too, but it was never reached in our runs 12

13 Concerning the choice of q, some comments are in order As the value of q increases, the updated preconditioner is expected to improve its effectiveness, reducing the number of linear iterations On the other hand, its computational cost is also expected to increase, because of the growing cost of the low-ran modification In order to reduce the time for the solution of the overall KKT sequence, the updating strategy must realize a tradeoff between effectiveness and cost, and our experience has shown that q must be much smaller than the dimension of the Schur complement On the basis of these considerations, we also thin that heuristic rules for choosing q dynamically, such as the one proposed in [45], may have limited impact of our procedure because only a small range of values of q is affordable However, we postpone a systematic study of this issue to future wor In the experiments we set q = 50, 100 and q 1 = q 2 = q/2, which is much smaller than the dimension of the Schur complement in our test problems We also considered the limit case q = 0, corresponding to S upd = S seed in the updated preconditioner P upd Preliminary experiments with larger values of q, ie, q = 150, 200, did not lead to any performance improvement and therefore we decided to discard these values Furthermore, since we had not obtained practical benefits by including in Γ indices corresponding to values of γ li (H) close to 1, we set µ γ = 10 and ν γ = 01 When the number of elements γ li (H) > µ γ or the number of elements γ li (H) < ν γ was less than q/2, we chose q 1 and q 2 to get the largest possible value of q 1 + q 2 The same comments hold for γ li ( H) For simplicity, in the following the notation γ li (H) is used also to indicate γ li ( H), as it will be clear from the context On the basis of numerical experiments, we decided to refresh the preconditioner, ie, to build P ex instead of P upd, when the time for computing P upd and solving the linear system exceeded 90% of the time for building the last exact preconditioner and solving the corresponding system When for a specific system of the sequence this situation occured, the next system of the sequence was solved using the preconditioner P ex We also set a maximum number, max, of consecutive preconditioner updates, after which the refresh was performed anyway This strategy aims at avoiding possible situations in which the time saved by updating the Schur complement, instead of refactorizing it, is offset by an excessive increase in the number of SQMR iterations, due to deterioration of the quality of the preconditioner In the experiments discussed here max = 5 was used for all the test problems We observe that we did not apply any scaling to the matrix X in (27), although our supporting theory lies on the assumption that the eigenvalues of X are smaller than 2 Nevertheless, the results generally appear to be in agreement with the theory It is also worth noting that since each KKT system is built from the approximation of the optimal solution computed at the previous IP iteration, different preconditioners produce changes in the sequence of KKT systems On the other hand, testing our updating technique inside an IP solver allows to better evaluate its impact on the performance of the overall optimization method, providing a more general evaluation of our approach In our experiments we did not observe strong differences in the behaviour of the IP method by using P ex or P upd with different values of q, and the number of IP iterations was generally unaffected by the choice of the preconditioner (a small variation of the IP iterations was observed in few cases, as shown in the tables reported in the next pages) We performed the numerical experiments on an Intel Core 2 Duo E7300 processor with cloc frequency of 266 GHz, 4 GB of RAM and 3 MB of cache memory, running Debian GNU/Linux 607 (ernel version amd64) All the software 13

14 was compiled using the the GNU C and Fortran compilers (version 445) Since the performance of the updating strategy was expected to depend on the cost of the factorization of the Schur complement, we selected test problems requiring different factorization costs As a first test set, we considered some problems taen from the CUTEst collection [35] or obtained by modifying problems available in this collection, as explained next Most of the large CUTEst convex QP problems with inequality constraints (corresponding to Θ (2) 0) were not useful for our experiments, because of the extremely low cost of the factorization of their Schur complements Therefore, we also modified two CUTEst QP problems with linear constraints Ax = b and non-negligible factorization costs, by changing Ax = b into Ax b The problems of this test set are listed in Table 41, along with their dimensions and the number of nonzero entries of their Schur complements The modified problems are identified by appending -M to their original names In the first three problems Θ (2) = 0, while in the remaining ones Θ (2) 0 For all the problems the Schur complements are very sparse; furthermore, they are banded for STCQP2, MOSARQP1 and QPBAND (diagonal for the latter problem) A comparison among the exact and updated preconditioners on this set of problems is presented in Table 41 For each preconditioner we report the total number of PRQP iterations (IP its), the total number of SQMR iterations (its) and the overall computation time, in seconds, needed to solve the KKT sequence The results shows that using the updating strategy is not beneficial on these problems Since the exact factorization of the Schur complement is not significantly more expensive than SQMR, the time saved by applying the updating strategy is not enough to offset the time required by the larger number of SQMR iterations resulting from the use of an approximate CP Nevertheless, for STCQP2 and MOSARQP1, the updating strategy shows the same performance as the exact preconditioner We also see that in many cases the number of SQMR iterations increases with q, which seems contrary to our expectations This behaviour has been explained with a deeper analysis of the execution of PRQP on the selected test problems The convergence histories of the IP method enlighten that the IP iterations at which the refresh taes place vary with q; this does not allow a fair comparison among the updating rules using different values of the parameter q This behaviour is ascribed to the fact that the computation of the exact preconditioner is not expensive and therefore the refresh strategy is very sensitive to the choice of q In particular, in our experiments the choice q = 0 often leads to recomputing the exact preconditioners before the number of SQMR iterations increases too much, thus reducing the iteration count with respect to larger values of q We also note that in some cases the number of SQMR iterations is practically constant as q varies, because either the number of elements γ li (H) [ν γ, µ γ ] is much smaller than q, or the values γ li (H) excluded by the updating strategy are not well separated from γ q2 (H) and γ n q1+1(h) Despite these first unfavourable results, since the updating strategy does not excessively increase the number of SQMR iterations, we can still expect a significant time reduction on problems with Schur complements requiring large factorization times In order to investigate this issue, we built a second set of test problems with less sparse Schur complements, by modifying the problems in Table 41 as follows In problems CVXQP1, CVXQP1-M, CVXQP3 and CVXQP3-M, we added four nonzero entries per row in the matrix A, while in problem QPBAND we added two nonzero entries per row In problem MOSARQP1, we introduced nonzeros in the positions (i, n) of the constraint matrix A, where i is such that mod(i, 10) = 1 These new 14

15 Problem CVXQP1 CVXQP3 STCQP2 CVXQP1-M CVXQP3-M MOSARQP1 QPBAND n m nnz(s) Pex Pupd (q = 0) Pupd (q=50) Pupd (q=100) IP its its time IP its its time IP its its time IP its its time e e e e e e e e e e e e e e e e e e e e e e e e e e e e+1 Table 41 Comparison between Pex and Pupd on the first set of test problems 15

16 problems are identified by appending -D to the names of the problems they come from, as listed in Table 42 ( D stands for denser ) Finally, starting from problems CVXQP3 and CVXQP3-M we generated two further problems, named CVXQP3-D2 and CVXQP3-M-D2, respectively They were obtained by adding only one nonzero entry per row in the matrix A The densities of the resulting Schur complements are between the Schur complements densities of the corresponding original and -D versions The results in Table 42 show that when the Schur complement is denser, the updating procedure provides a significant reduction in the overall computation time, because the increase in the SQMR iterations is largely offset by the time saving obtained by updating the factors of the Schur complement instead of recomputing them For the problems under consideration the reduction ranges from 21%, for CVXQP3-M-D2 with q = 0, to 80%, for STCQP2-D with all the three values of q Comparing the behaviour of the updating strategy on CVXQP3-M-D and CVXQP3- M-D2, we see that the percentage of time saved with the updating strategy drops from 53-54% to 21-24% when going from CVXQP3-M-D to CVXQP3-M-D2 (the latter has a sparser Schur complement) A similar behaviour can be observed by comparing the results obtained on CVXQP3-D and CVXQP3-D2 In this case the best time reduction for CVXQP3-D amounts to 68% (q = 0), while the best one for CVXQP3- D is 47% (q = 50) Furthermore, the time reduction also holds when the number of IP iterations corresponding to the updating strategy is greater than the number of IP iterations obtained with the exact preconditioner (see CVXQP3-D2) Surprisingly, also the reverse may happen, ie, the updating procedure may slightly reduce the number of IP iterations (see CVXQP1-M-D and MOSARQP1-D) We further note that the number of iterations obtained with P upd generally decreases as q increases; thus, for the second set of problems, updating the Schur complement by low-ran information appears to be beneficial in terms of iterations There are also some cases where the number of SQMR iterations is practically constant as q varies (see STCQP2-D and QPBAND-D) In these cases, as for the corresponding problems in the first test set, we verified that either the number of elements γ li (H) with indices in Γ is very small or even zero, or those values of γ li (H) are not well separated from the remaing ones, thus maing the low-ran modification ineffective For similar reasons the reduction of the number of iterations from q = 50 to q = 100 is generally less significant than from q = 0 to q = 50 Finally, the best results in terms of execution time are mostly obtained with q > 0 To provide more insight into the behaviour of the updated preconditioners, in Tables we show some details concerning the solution of the sequences of KKT systems arising from four problems, ie, CVXQP3, CVXQP3-D, MOSARQP1 and MOSARQP1-D For each IP iteration we report the number, its, of SQMR iterations, as well as the time, T prec, for building the preconditioner, the time, T solve, for solving the linear system, and their sum, T sum The last row contains the total number of SQMR iterations and the total times, over all IP iterations, while the rows in bold correspond to the IP iterations at which the preconditioner is refreshed These tables clearly support the previous observation that the updating strategy is efficient when the computation of P ex is expensive, as it is for CVXQP3-D and MOSARQP1-D Conversely, when the time for building P ex is modest, recomputing P ex is a natural choice It also appears that the refresh strategy plays a significant role in achieving efficiency, since it prevents the preconditioner from excessive deterioration Finally, when the time for computing P ex is not dominant, the refresh tends to occur more 16

17 Problem CVXQP1-D CVXQP3-D CVXQP3-D2 STCQP2-D CVXQP1-M-D CVXQP3-M-D CVXQP3-M-D2 MOSARQP1-D QPBAND-D n m nnz(s) Pex Pupd (q = 0) Pupd (q=50) Pupd (q=100) IP its its time IP its its time IP its its time IP its its time e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e+2 Table 42 Comparison between Pex and Pupd on the second set of test problems 17

18 P ex P upd (q=50) IP it its T prec T solve T sum its T prec T solve T sum e-2 250e-1 338e e-2 249e-1 337e e-2 917e-2 170e e-2 170e-1 192e e-2 735e-2 144e e-2 217e-1 239e e-2 616e-2 140e e-2 293e-1 312e e-2 616e-2 140e e-2 585e-2 133e e-2 590e-2 134e e-3 998e-2 105e e-2 839e-2 162e e-2 184e-1 198e e-2 800e-2 155e e-2 796e-2 154e e-2 101e-1 175e e-2 120e-1 131e e-2 981e-2 177e e-2 201e-1 216e e-2 119e-1 198e e-2 119e-1 193e e-2 131e-1 205e e-2 142e-1 152e e-2 126e-1 204e e-2 334e-1 353e e-2 125e-1 204e e-2 124e-1 199e e-2 125e-1 196e e-2 159e-1 170e e-2 126e-1 204e e-2 272e-1 284e e-2 147e-1 225e e-2 145e-1 220e e-2 144e-1 219e e-2 199e-1 210e e-2 144e-1 223e e-2 143e-1 213e e-2 141e-1 216e e-2 201e-1 212e e-2 165e-1 243e e-2 163e-1 237e e-2 143e-1 222e e-2 231e-1 241e e-2 268e-1 343e e-2 265e-1 340e e-2 271e-1 345e e-2 397e-1 408e e+0 536e+0 804e e+0 753e+0 886e+0 Table 43 CVXQP3: details for P ex and P upd with q = 50 frequently, since a small increase in the number of iterations obtained with P upd may easily raise the execution time over 90% of the time corresponding to the last application of the exact preconditioner 5 Conclusion We have proposed a preconditioner updating procedure for the solution of sequences of KKT systems arising in IP methods for convex QP problems The preconditioners built by this procedure belong to the class of inexact CPs and are obtained by updating a given seed CP The updates are performed through low-ran corrections of the Schur complement of the (1,1) bloc in the seed preconditioner and generate factorized preconditioners The rule for identifying the low-ran corrections is based on new bounds on the eigenvalues of the preconditioned matrix The numerical experiments show that our updated preconditioners, combined with a suitable preconditioner refreshing, can be rather successful More precisely, the higher the cost of the Schur complement factorization, the more advantageous the updating procedure becomes Finally, we believe that the updating strategy proposed here paves the way to the definition of preconditioner updating procedures for sequences of KKT systems where the Hessian and constraint matrices change from one iteration to the next 18

19 P ex P upd (q=50) IP it its T fact T solve T sum its T prec T solve T sum e+0 119e+0 637e e+0 118e+0 642e e+0 487e-1 565e e-1 554e-1 111e e+0 340e-1 550e e-1 601e-1 115e e+0 224e-1 535e e-1 529e-1 104e e+0 227e-1 537e e-1 137e+0 192e e+0 337e-1 550e e-1 179e+0 239e e+0 415e-1 557e e+0 417e-1 566e e+0 493e-1 565e e-1 589e-1 745e e+0 561e-1 570e e-1 839e-1 111e e+0 558e-1 574e e-1 154e+0 202e e+0 633e-1 578e e-1 290e+0 339e e+0 668e-1 583e e-1 509e+0 556e e+0 740e-1 588e e+0 744e-1 599e e+0 811e-1 597e e-1 116e+0 136e e+0 924e-1 608e e-1 232e+0 279e e+0 139e+0 651e e-1 317e+0 364e e+0 176e+0 703e e-1 583e+0 629e e+1 118e+1 995e e+1 306e+1 526e+1 Table 44 CVXQP3-D: details for P ex and P upd with q = 50 P ex P upd (q=100) IP it its T prec T solve T sum its T prec T solve T sum e-1 295e-2 241e e-1 295e-2 239e e-1 453e-2 254e e-2 929e-2 128e e-1 484e-2 258e e-2 154e-1 222e e-1 655e-2 275e e-1 615e-2 266e e-1 629e-2 276e e-3 277e-1 286e e-1 821e-2 291e e-1 730e-2 274e e-1 784e-2 292e e-2 256e-1 306e e-1 776e-2 287e e-1 730e-2 278e e-1 751e-2 288e e-2 292e-1 363e e-1 920e-2 301e e-1 867e-2 287e e-1 893e-2 306e e-2 235e-1 258e e-1 109e-1 323e e-2 912e-1 981e e-1 895e-2 303e e-1 842e-2 293e e-1 107e-1 320e e-2 322e-1 369e e-1 107e-1 320e e-1 101e-1 302e e-1 104e-1 313e e-2 335e-1 363e e+0 127e+0 465e e+0 339e+0 521e+0 Table 45 MOSARQP1: details for P ex and P upd with q = 100 Acnowledgments We are indebted to Tim Davis for his valuable help in interfacing PRQP with CHOLMOD We also express our thans to Miroslav T uma for maing available a Fortran 90 implementation of SQMR used in preliminary numerical experiments Finally, we wish to than the anonymous referees for their useful comments, which helped us to improve the quality of this wor 19

20 P ex P upd (q=100) IP it its T prec T solve T sum its T prec T solve T sum e+0 599e-2 201e e+0 593e-2 196e e+0 592e-2 198e e-2 134e-1 182e e+0 585e-2 197e e-2 246e-1 336e e+0 902e-2 202e e-1 436e-1 117e e+0 901e-2 200e e-1 631e-1 136e e+0 118e-1 203e e-1 156e+0 207e e+0 119e-1 203e e+0 113e-1 200e e+0 151e-1 205e e-2 439e-1 490e e+0 148e-1 205e e-2 574e-1 625e e+0 147e-1 206e e-1 792e-1 117e e+0 150e-1 205e e-1 155e+0 227e e+0 146e-1 205e e+0 142e-1 203e e+0 147e-1 206e e-2 164e-1 255e e+0 148e-1 208e e-1 269e-1 372e e+0 119e-1 203e e-1 486e-1 589e e+0 146e-1 206e e-1 132e+0 203e e+0 145e-1 208e e+0 194e-1 207e e+0 203e-1 213e e-2 457e-1 508e e+0 207e-1 211e e-1 130e+0 202e e+0 175e-1 208e e+0 197e-1 208e e+0 206e-1 211e e-1 569e-1 790e e+0 203e-1 212e e-1 919e-1 142e e+0 203e-1 213e e+0 202e-1 212e e+1 344e+0 494e e+1 126e+1 278e+1 Table 46 MOSARQP1-D: details for P ex and P upd with q =

21 REFERENCES [1] V Baryamureeba, T Steihaug, Y Zhang, Properties of a Class of Preconditioners for Weighted Least Squares Problems, Technical Report No 170, Department of Informatics, University of Bergen, and Technical Report No TR99 16, Department of Computational and Applied Mathematics, Rice University, Houston, 1999 [2] S Bellavia, Inexact Interior-Point Method, Journal of Optimization Theory and Applications, 96 (1998), pp [3] S Bellavia, D Bertaccini, B Morini, Nonsymmetric preconditioner updates in Newton- Krylov methods for nonlinear systems, SIAM Journal on Scientific Computing, 33 (2011), pp [4] S Bellavia, V De Simone, D di Serafino, B Morini, Efficient preconditioner updates for shifted linear systems, SIAM Journal on Scientific Computing, 33 (2011), pp [5] S Bellavia, V De Simone, D di Serafino, B Morini, A preconditioning framewor for sequences of diagonally modified linear systems arising in optimization, SIAM Journal on Numerical Analysis, 50 (2012), pp [6] S Bellavia, B Morini, M Porcelli, New updates of incomplete LU factorizations and applications to large nonlinear systems, Optimization Methods and Software, 29 (2014), pp [7] M Benzi, D Bertaccini, Approximate inverse preconditioning for shifted linear systems, BIT, 43 (2003), pp [8] M Benzi, V Simoncini, On the eigenvalues of a class of saddle point matrices, Numerische Mathemati, 103 (2006), pp [9] L Bergamaschi, Eigenvalue distribution of constraint-preconditioned symmetric saddle point matrices, Numerical Linear Algebra with Applications, 19 (2012), pp [10] L Bergamaschi, R Bru, A Martinez, M Putti, Quasi-Newton preconditioners for the inexact Newton method, Electronic Transactions on Numerical Analysis, 23 (2006) pp [11] L Bergamaschi, J Gondzio, G Zilli, Preconditioning Indefinite Systems in Interior Point Methods for Optimization, Computational Optimization and Applications, 28 (2004), pp [12] L Bergamaschi, J Gondzio, M Venturin, G Zilli, Inexact Constraint Preconditioners for Linear Systems Arising in Interior Point Methods, Computational Optimization and Applications, 36 (2007), pp [13] S Cafieri, M D Apuzzo, V De Simone, D di Serafino, On the iterative solution of KKT systems in potential reduction software for large-scale quadratic problems, Computational Optimization and Applications, 38 (2007), pp [14] S Cafieri, M D Apuzzo, V De Simone, D di Serafino, Stopping criteria for inner iterations in inexact potential reduction methods: a computational study, Computational Optimization and Applications, 36 (2007), pp [15] S Cafieri, M D Apuzzo, V De Simone, D di Serafino, On the Use of an Approximate Constraint Preconditioner in a Potential Reduction Algorithm for Quadratic Programming, in Applied and Industrial Mathematics in Italy II, V Cutello, G Fotia and L Puccio eds, Series on Advances in Mathematics for Applied Sciences, 75, World Scientific, 2007, pp [16] S Cafieri, M D Apuzzo, V De Simone, D di Serafino, G Toraldo, Convergence analysis of an inexact potential reduction method for convex quadratic programming, Journal of Optimization Theory and Applications, 135 (2007), pp [17] C Calgaro, JP Chehab, Y Saad, Incremental incomplete ILU factorizations with applications, Numerical Linear Algebra with Applications, 17 (2010), pp [18] B Carpentieri, IS Duff, L Giraud, A class of spectural two-level preconditioners, SIAM Journal on Scientific Computing, 25 (2003), pp [19] M D Apuzzo, V De Simone, D di Serafino, On mutual impact of numerical linear algebra and large-scale optimization with focus on interior point methods, Computational Optimization and Applications, 45 (2010), pp [20] M D Apuzzo, V De Simone, D di Serafino, Starting-Point Strategies for an Infeasible Potential Reduction Method, Optimization Letters, 4 (2010), pp [21] TA Davis, WW Hager, Dynamic supernodes in sparse Cholesy update/downdate and triangular solves, ACM Transactions on Mathematical Software, 35 (2009), article 27 [22] HS Dollar, Constraint-style preconditioners for regularized saddle-point problems, SIAM Journal on Matrix Analysis and Applications, 29 (2007), pp [23] HS Dollar, NIH Gould, WHA Schilders, AJ Wathen, Implicit-factorization preconditioning and iterative solvers for regularized saddle-point systems, SIAM Journal on Matrix 21