Comput Optim App (27) 36: 249 27 DOI.7/s589-6-94-x Using constraint preconditioners with reguarized sadde-point probems H.S. Doar N.I.M. Goud W.H.A. Schiders A.J. Wathen Pubished onine: 22 February 27 Springer Science+Business Media, LLC 27 Abstract The probem of finding good preconditioners for the numerica soution of a certain important cass of indefinite inear systems is considered. These systems are of a 2 by 2 bock (KKT) structure in which the (2,2) bock (denoted by C) is assumed to be nonzero. In Constraint preconditioning for indefinite inear systems, SIAM J. Matrix Ana. App. 2 (2), Keer, Goud and Wathen introduced the idea of using constraint preconditioners that have a specific 2 by 2 bock structure for the case of C being zero. We sha give resuts concerning the spectrum and form of the eigenvectors when a preconditioner of the form considered by Keer, Goud and Wathen is used but the system we wish to sove may have C. In particuar, the resuts presented here H.S. Doar ( ) N.I.M. Goud Computationa Science and Engineering Department, Rutherford Appeton Laboratory, Chiton, Oxfordshire, OX QX, Engand, UK e-mai: s.doar@r.ac.uk N.I.M. Goud e-mai: n.i.m.goud@r.ac.uk N.I.M. Goud A.J. Wathen Numerica Anaysis Group, Oxford University Computing Laboratory, Wofson Buiding, Parks Road, Oxford, OX 3QD, UK e-mai: nick.goud@comab.ox.ac.uk A.J. Wathen e-mai: andy.wathen@comab.ox.ac.uk W.H.A. Schiders Design Methods and Soutions, NXP Semiconductors, High Tech Campus 48, 5656 AE Eindhoven, The Netherands e-mai: wi.schiders@nxp.com W.H.A. Schiders Department of Mathematics and Computer Science, Technische Universiteit Eindhoven, PO Box 53, 56 MB Eindhoven, The Netherands
25 H.S. Doar et a. indicate custering of eigenvaues and, hence, faster convergence of Kryov subspace iterative methods when the entries of C are sma; such a situations arise naturay in interior point methods for optimization and we present resuts for such probems which vaidate our concusions. Keywords Preconditioning Indefinite inear systems Kryov subspace methods Introduction The soution of systems of the form A B T B C }{{} A C ] ] x y ] c = d }{{} b (.) where A R n n, C R m m are symmetric and B R m n, is often required in optimization and other various fieds, Sect... We sha assume that <m n and B is of fu rank. Various preconditioners which take the genera form ] G B T P C = (.2) B C where G R n n is some symmetric matrix, have been considered (for exampe, see 3 5, 8, 8, 23].) When C =, (.2) is commony known as a constraint preconditioner 2, 6, 7, 9]. In practice C is often positive semi-definite (and frequenty diagona). As we wi observe in Sect.., in interior point methods for constrained optimization a sequence of such probems are soved with the entries in C generay becoming sma as the optimization iteration progresses. That is, the reguarization is successivey reduced as the iterates get coser to the minimum. For the Stokes probem, the entries of C are generay sma since they scae with the underying mesh size and so reduce for finer grids. This motivates us to ook at the spectra properties of P A C, where ] G B T P =, (.3) B but C in(.), Sect. 2. We wi anayze both the cases of C having fu rank and C being rank deficient. We note that when there are equaity constraints in the noninear programming probem, the corresponding diagona of C wi be identicay zero, and thus C wi be (triviay) rank deficient. The obvious advantage in being abe to use such a constraint preconditioner is as foows: if B remains constant in each system of the form (.), and we choose G in our preconditioner to remain constant, then the preconditioner P wi be unchanged. Any factorizations required to carry out the preconditioning steps in a Kryov subspace iteration wi ony need to be done once and then used during each execution of the chosen Kryov subspace iteration, instead of carrying out the factorizations at the beginning of each execution.
Using constraint preconditioners with reguarized sadde-point probems 25 For symmetric (and in genera norma) matrix systems, the convergence of an appicabe iterative method is determined by the distribution of the eigenvaues of the coefficient matrix. It is often desirabe for the number of distinct eigenvaues to be sma so that the rate of convergence is rapid. For non-norma systems the convergence is not so readiy described, see 4, page 6].. Appications requiring the soution of reguarized sadde-point probems In this section we indicate two appication areas that require the soution of a reguarized sadde-point probems. A comprehensive ist of further appications can be found in 2]. Exampe. (Noninear Programming) Consider the convex noninear optimization probem minimize f(x) such that c(x), (.4) where x R n, and f : R n R and c : R n R m are convex and twice differentiabe. Prima dua interior point methods 24] for this probem aim to track soutions to the (perturbed) optimaity conditions f(x)= B T (x)y and Yc(x)= μe (.5) where y are Lagrange mutipiers (dua variabes), e is the vector of ones, B(x) = c(x) and Y = diag{y,y 2,...,y m }, as the positive scaar parameter μ is decreased to zero. The Newton correction ( x, y) to the soution estimate (x, y) of (.5) satisfy the equation 3]: A(x,y) B T ] ] (x) x f(x)+ B = T ] (x)y Y B(x) C(x) y Yc(x)+ μe where A(x,y) = xx f(x) m i= y i xx c i (x) and C(x) = diag{c (x), c 2 (x),..., c m (x)}. It is common to eiminate the variabes y from the Newton system. Since this may introduce unwarranted i conditioning, it is often better ] to isoate the effects of poor conditioning by partitioning the constraints so that the vaues of those indexed by I are arge whie those indexed by A are sma, and instead to sove A + B T I CI Y ] ] IB I BA T x B A C A YA = y A f + B T A y A + μb T I C I c A + μy A e where, for brevity, we have dropped the dependence on x and y. The matrix C A YA is symmetric and positive definite; as the iterates approach optimaity, the entries of this matrix become sma. The entries of BI T C I Y IB I aso become sma when cose to optimaity. e ]
252 H.S. Doar et a. Exampe.2 (Stokes) Mixed finite eement (and other) discretizations of the Stokes equations 2 u + p = f in Ω u = inω, for the fuid veocity u and pressure p in the domain Ω R 2 or R 3 yieds inear systems in the sadde-point form (.) (for derivation and the foowing properties of this exampe see 7]). The symmetric bock A arises from the diffusion terms 2 u and B T represents the discrete gradient operator whist B represents its adoint, the (negative) divergence. When (inf-sup) stabe mixed finite eement spaces are empoyed, C =, however for equa order and other spaces which are not inherenty stabe, stabiized formuations yied symmetric and positive semi-definite matrices C which typicay have a arge-dimensiona kerne for exampe for the famous Q P eement which has piecewise biinear veocities and piecewise constant pressures in 2-dimensions, C typicay has a kerne of dimension m/4. 2 Preconditioning A C by P Suppose that we precondition A C by P, where P is defined in (.3). The decision to investigate this form of preconditioner is motivated in Sect.. We sha use the foowing assumptions in our theorems: A B R m n (m n) has fu rank, A2 C has rank p> and is factored as EDE T, where E R m p and has orthonorma coumns, and D R p p is non-singuar, A3 If p<m, then F R m (m p) is such that its coumns form a basis for the nuspace of C and N R n (n m+p) is such that its coumns form a basis of the nuspace of F T B, A4 If p = m, then N = I R n n. Theorem 2. Assume that A A4 hod, then the matrix P A C has: at east 2(m p) eigenvaues at, its non-unit eigenvaues defined by the finite (and non-unit) eigenvaues of the quadratic eigenvaue probem = λ 2 N T B T ED E T BNw n λn T (G + 2B T ED E T B)Nw n + N T (A + B T ED E T B)Nw n. Proof We sha consider the cases of p = m and <p<mseparatey. Case p = m. The generaized eigenvaue probem takes the form ] ] ] ] A B T x G B T x = λ. (2.) B C y B y
Using constraint preconditioners with reguarized sadde-point probems 253 Expanding this out we obtain Ax + B T y = λgx + λb T y, (2.2) Bx Cy = λbx. (2.3) From (2.3) we deduce that either λ = and y =, or λ. If the former hods, then (2.2) impies that x must satisfy Ax = Gx. Thus, the associated eigenvectors wi take the form x T T ] T, where x satisfies Ax = Gx. There is no guarantee that such an eigenvector wi exist, and therefore no guarantee that there are any unit eigenvaues. If λ, then Eq. (2.3) and the non-singuarity of C gives y = ( λ)c Bx, x. By substituting this into (2.2) and rearranging we obtain the quadratic eigenvaue probem (λ 2 B T C B λ(g + 2B T C B)+ A + B T C B)x =. (2.4) The non-unit eigenvaues of (2.) are therefore defined by the finite (non-unit) eigenvaues of (2.4). Now, assumption A2 impies that C = ED E T, and, hence, etting w n = x we compete our proof for the case p = m. Case <p<m.anyy R m can be written as y = Ey e + Fy f. Substituting this into (2.) and premutipying the resuting generaized eigenvaue probem by I E T, F T we obtain A B T E B T F x G B T E B T F x E B D y e = λ E B y e. (2.5) F T B F T B y f Noting that the (3,3) bock has dimension (m p) (m p) and is a zero matrix in both coefficient matrices, we can appy Theorem 2. from 6] to obtain: P A C has an eigenvaue at with mutipicity 2(m p), y f
254 H.S. Doar et a. the remaining n m + 2p eigenvaues are defined by the generaized eigenvaue probem N T A B T ] E E T Nw B D n = λn T G B T ] E E T Nw B n (2.6) where N is an (n + p) (n m + 2p) basis for the nuspace of F T B ]. One choice for N is ] N N =. I Substituting this into (2.6) we obtain the generaized eigenvaue probem N T AN N T B T E E T BN D ] ] wn = λ w n2 N T GN N T B T ] ] E wn E T. (2.7) BN w n2 This generaized eigenvaue probem resembes that of (2.) in the first case considered in this proof. Therefore, the non-unit eigenvaues of P A C are equa to the finite (and non-unit) eigenvaues of the quadratic eigenvaue probem = λ 2 N T B T ED E T BNw n λn T (G + 2B T ED E T B)Nw n + N T (A + B T ED E T B)Nw n. (2.8) Since N T B T ED E T BN has a nuspace of dimension n m, this quadratic eigenvaue probem has 2(n m + p) (n m) = n m + 2p finite eigenvaues 22]. The foowing numerica exampes iustrate how the rank of C dictates a ower bound on the number of unit eigenvaues. In particuar, Exampe 2.2 demonstrates that there is no guarantee that the preconditioned matrix has unit eigenvaues when C is nonsinguar. Exampe 2.2 (C nonsinguar) Consider the matrices 2 A C =, P = 2, so that m = p = and n = 2. The preconditioned matrix P A C has eigenvaues at 2,2 2 and 2+ 2. The corresponding eigenvectors are ] T, ( 2 )] T and ( 2 + )] T respectivey. The preconditioned system P A C has a nonunit eigenvaues, but this does not go against Theorem 2. because m p =. With our choices of A C and P, and setting D =] and E =] (C = EDE T ), the quadratic eigenvaue probem (2.8)is ( ] λ 2 λ ] 4 + 2 ]) ] 2 u =. u 2
Using constraint preconditioners with reguarized sadde-point probems 255 This quadratic eigenvaue probem has three finite eigenvaues which are λ = 2, λ = 2 2 and λ = 2 + 2. Exampe 2.3 (C semidefinite) Consider the matrices 2 A C =, P = 2, so that m = 2, n = 2 and p =. The preconditioned matrix P A C has two unit eigenvaues and a further two at λ = 2 2 and λ = 2 + 2. There is ust one ineary independent eigenvector associated with the unit eigenvector; specificay this is ] T. For the non-unit eigenvaues, the eigenvectors are ( 2 )] T and ( 2 + )] T respectivey. Since 2(m p) = 2, we correcty expected there to be at east two unit eigenvaues, Theorem 2.. The remaining eigenvaues wi be defined by the finite eigenvaues of the quadratic eigenvaue probem (2.8): ( ] ] ]) ] λ 2 2 u λ + = 4 2 u 2 where D =] and E =] T are used as factors of C. This quadratic eigenvaue probem has three finite eigenvaues which are λ = 2 2 and λ = 2 + 2; the corresponding eigenvectors have u =. 2. Anaysis of the quadratic eigenvaue probem We note that the quadratic eigenvaue probem (2.8) can have negative and compex eigenvaues, see 22]. The foowing theorem gives sufficient conditions for genera quadratic eigenvaue probems to have rea and positive eigenvaues. Theorem 2.4 Consider the quadratic eigenvaue probem (λ 2 K λl + M)x =, (2.9) where M,L R n n are symmetric positive definite, and K R n n is symmetric positive semidefinite. Define γ(m,l,k)to be γ(m,l,k)= min{(x T Lx) 2 4(x T Mx)(x T Kx): x 2 = }. If γ(m,l,k)>, then the eigenvaues λ are rea and positive, and there are n ineary independent eigenvectors associated with the n argest (n smaest) eigenvaues. Proof From 22, Sect. ] we know that under our assumptions the quadratic eigenvaue probem (μ 2 M + μl + K)x =
256 H.S. Doar et a. has rea and negative eigenvaues. Suppose we divide this equation by μ 2 and set λ = /μ. The quadratic eigenvaue probem (2.9) is obtained, and since μ is rea and negative, λ is rea and positive. We woud ike to be abe to use the above theorem to show that, under suitabe assumptions, a the eigenvaues of P A C are rea and positive. Let D = N T B T ED E T BN (2.) where D and E are as defined in assumption A2, and N is as defined in assumption A3. If we assume that N T AN + D is positive definite, then we may write N T AN + D = R T R for some nonsinguar matrix R. If we premutpy-mutipy the quadratic eigenvaue probem (2.8)byR T and substitute in z = Rw n, then we find that it is simiar to the quadratic eigenvaue probem ( λ 2 R T DR λr T (N T GN + 2 D)R + I ) z =. Thus, if we assume that N T AN + D, N T GN + 2 D are positive definite and D is positive semi-definite, and can show that γ ( I,R T (N T GN + 2 D)R,R T DR ) >, where γ(,, ) is as defined in Theorem 2.4, then we can appy the above theorem to show that (2.8) has rea and positive eigenvaues. Let us assume that z 2 =, then ( z T R T ( N T GN + 2 D ) R z ) 2 4z T zz T R T DR z = ( z T R T N T GNR z + 2z T R T DR z ) 2 4z T R T DR z = (z T R T N T GNR z) 2 + 4z T R T DR z ( z T R T N T GNR z + z T R T DR z ) = (wn T N T GNw n ) 2 + 4wn T Dw ( n w T n N T GNw n + wn T Dw n ) (2.) where = z 2 = Rw n 2 = w n N T AN+ D. Ceary, we can guarantee that (2.) is positive if that is w T n N T GNw n + w T n Dw n > for a w n such that w n N T AN+ D =, w T n N T GNw n + w T n Dw n w T n (N T AN + D)w n Rearranging we find that we require > wt n (N T AN + D)w n w T n (N T AN + D)w n for a w n. w T n N T GNw n >w T n N T ANw n
Using constraint preconditioners with reguarized sadde-point probems 257 for a w n. Thus we need ony scae any positive definite G such that wn T N T GNw n /(wn T N T Nw n )> A 2 2 for a Nw n to guarantee that (2.) is positive for a w n such that w n N T AN+ D =. For exampe, we coud choose G = αi, where α> A 2 2. Using the above in conunction with Theorem 2. we obtain the foowing resut: Theorem 2.5 Suppose that A A4 hod and D is as defined in (2.). Further, assume that A+ D and G+2 D are symmetric positive definite, D is symmetric positive semidefinite and min { (z T Gz) 2 + 4(z T Dz)(z T Gz + z T Dz ) : z A+ D = } >, (2.2) then a the eigenvaues of P A C are rea and positive. The matrix P A C aso has m p + i + ineary independent eigenvectors. There are. m p eigenvectors of the form T yf T ]T that correspond to the case λ =, 2. i( i n) eigenvectors of the form w T T yf T ]T arising from Aw = Gw for which the i vectors w are ineary independent, and λ =, and 3. ( n m+2p) eigenvectors of the form T wn T wt n2 yt f ]T corresponding to the eigenvaues of P A C not equa to, where the components w n arise from the quadratic eigenvaue probem = λ 2 N T B T ED E T BNw n λn T (G + 2B T ED E T B)Nw n + N T (A + B T ED EB)Nw n, with λ, and w n2 = ( λ)d E T BNw n. Proof It remains for us to prove the form of the eigenvectors and that they are ineary independent. We wi consider the case p = m and <p<mseparatey. Case p = m. From the proof of Theorem 2., when λ = the eigenvectors must take the form x T T ] T, where Ax = σgx for which the i vectors x are ineary independent, σ =. Hence, any eigenvectors corresponding to a unit eigenvaue fa into the second statement of the theorem and there are i( i n) such eigenvectors which are ineary independent. The proof of Theorem 2. aso shows that the eigenvectors corresponding to λ take the form x T y T ] T, where x corresponds to the quadratic eigenvaue probem (2.4) and y = ( λ)c Bx = ( λ)d EBNx (since we can set D = C and E = I ). Ceary, there are at most n + m such eigenvectors. By our assumptions, a of the vectors x defined by the quadratic eigenvaue probem (2.4) are ineary independent. Aso, if x is associated with two eigenvaues, then these eigenvaues must be distinct 22]. By setting w n = x and w n2 = y we obtain ( n + m) eigenvectors of the form given in statement 3 of the proof.
258 H.S. Doar et a. It remains for us to prove that the i + eigenvectors defined above are ineary independent. Hence, we need to show that ] a () x ()... x () (2) i.. x... +... x (2) ] a (2).. y (2)... y (2) =. (2.3) a () i impies that the vectors a () and a (2) are zero vectors. Mutipying (2.3)byP A C, and recaing that in the previous equation the first matrix arises from λ = ( =,...,i)and the second matrix from λ ( =,...,)gives x ()... x () i... ] a ().. a () i + Subtracting (2.3) from (2.4) we obtain (2) x... x (2) y (2)... y (2) (2) x... x (2) y (2)... y (2) ] ] a (2) λ (2) a(2). λ (2) a (2) =.. (2.4) (λ (2) )a (2). =... (2.5) (λ (2) )a (2) Some of the eigenvectors x defined by the quadratic eigenvaue probem (2.4) wi be associated with two (non-unit) eigenvaues; et us assume that there are k such eigenvectors. By our assumptions, these eigenvaues must be distinct. Without oss of generaity, assume that x (2) = x (2) k+ for =,...,k. The vectors x (2) ( = k +,...,) are ineary independent and λ (2) ( = 2k +,...,), which gives rise to a (2) = for = 2k +,...,. Equation (2.5) becomes (2) x... x (2) k x (2)... x (2) k y (2)... y (2) k y (2) k+... y (2) 2k The vectors x (2) ] ( =,...,k)are ineary independent. Hence (λ (2) )a (2). =... (2.6) (λ (2) )a (2) 2k (λ (2) )a (2) x (2) + (λ (2) +k )a(2) +k x(2) =, =,...,k, and a (2) = a (2) λ (2) +k +k λ (2), =,...,k. Now y (2) = ( λ (2) )C Bx (2) for =,...,2k. Hence, we require (λ (2) ) 2 a (2) C Bx (2) + (λ (2) +k )2 a (2) +k C Bx (2) =, =,...,k.
Using constraint preconditioners with reguarized sadde-point probems 259 Substituting in a (2) = a (2) +k ( λ(2) +k )/( λ(2) ) and rearranging gives (λ (2) )a (2) = (λ (2) +k )a(2) +k for =,...,k. Since these eigenvaues are non-unit and λ (2) λ (2) +k for =,...,k, we concude that a(2) We aso have inear independence of x () ( =,...,i). = ( =,...,). ( =,...,i), which impies that a () = Case <p<m. From the proof of Theorem 2., the generaized eigenvaue probem can be expressed as A B T E B T F x E B D y e = λ F T B y f G E T B F T B B T E B T F x. (2.7) The first part of the proof for this case foows simiary to that of Theorem 2.3 in 6]. Let M N]R T ] T be an orthogona factorization of F T B ], where R R (m p) (m p) is upper trianguar, M R (n+p) (m p), and N R (n+p) (n m+2p) is a basis for the nuspace of F T B ]. Premutipying (2.7) by the nonsinguar and square matrix M T N T, I y e y f substituting in x y e y f = M N I ] m w w n, y f and expanding out gives M T Â Mw m + M T Â Nw n + Ry f = λ M T Ĝ Mw m + M T ] Ĝ Nw n + Ry f, (2.8) N T Â Mw m + N T Â Nw n = λ N T Ĝ Mw m + N T ] Ĝ Nw n, (2.9) R T w m = λr T w m (2.2) where A B Â = T ] E E T B D and Ĝ = G B T ] E E T. B From (2.2), it may be deduced that either λ = orw m =. In the former case, (2.8) and (2.9) may be simpified to Q T A B T ] E E T Qw = Q T G B T ] E B D E T Qw (2.2) B
26 H.S. Doar et a. where Q =M N] and w =wm T wt n ]. Since Q is orthogona, the genera eigenvaue probem (2.2) is equivaent to considering A B T E E T B D ] w w 2 ] = σ G B T ] ] E w E T B w 2 (2.22) where w T wt 2 ]T if and ony if σ =, and w R n, w 2 R p. As in the first case of this proof, nonsinguarity of D and σ = impies that w 2 =. There are m p ineary independent eigenvectors T T u T f ]T corresponding to w =, and a further i( i n) ineary independent eigenvectors corresponding to w and σ =. Now suppose that λ, in which case w m =. Equations (2.8) and (2.9) yied N T A B T E E T B D M T A B T E E T B D ] Nw n = λn T G B T E E T B ] Nw n + Ry f = λ M T G B T E E T B ] Nw n, (2.23) ] Nw n + Ry f ]. (2.24) The generaized eigenvaue probem (2.24) defines n m + 2p eigenvaues, where ( n m) of these are not equa to and for which two cases have to be distinguished. If w n =, then (2.23) and λ impy that y f =. In this case no extra eigenvaues arise. Suppose that w n, then, from the proof of Theorem 2., the eigenvaues are equivaenty defined by (2.8) and w n = w n ( λ)d E T BNw n Hence, the ( n m + 2) eigenvectors corresponding to the non-unit eigenvaues of P A C take the form T w T n wt n2 yt f ]T. Proof of the inear independence of these eigenvectors foows simiary to the case of p = m. Observing that the coefficient matrices in (2.5) are of the form of those considered by Goud, Hribar and Noceda 2], we coud appy a proected preconditioned conugate gradient method to sove (.) if a the eigenvaues of P A C are rea and positive and we have a decomposition of C as in A2. Theorem 2.5 therefore gives conditions which aow us to use such a method. Doar gives a variant of this method in which no decomposition of C is required, see 6, Sect. 5.5]. The derivation of such a method bears cose resembance to that of a nuspace method. The nuspace N is required in the derivation but, as in 2], we can rewrite the agorithm in such a manner that there is no need for N to be known expicity. ]. 3 Convergence In the context of this paper, the convergence of an iterative method under preconditioning is not ony infuenced by the spectra properties of the coefficient matrix, but
Using constraint preconditioners with reguarized sadde-point probems 26 aso by the reationship between m, n and p. We can determine an upper bound on the number of iterations of an appropriate Kryov subspace method by considering minimum poynomias of the coefficient matrix. Definition 3. Let A R (n+m) (n+m). The monic poynomia f of minimum degree such that f(a) = is caed the minimum poynomia of A. Kryov subspace theory states that iteration with any method with an optimaity property, e.g. GMRES, wi terminate when the degree of the minimum poynomia is attained, 2]. In particuar, the degree of the minimum poynomia is equa to the dimension of the corresponding Kryov subspace (for genera b), 2, Proposition 6.]. Theorem 3.2 Suppose that the assumptions of Theorem 2.5 hod. The dimension of the Kryov subspace K(P A C,b)is at most min{n m + 2p + 2,n+ m}. Proof Suppose that <p<m. As in the proof to Theorem 2., the generaized eigenvaue probem can be written as A B T E B T F x E B D y e = λ F T B y f G E T B F T B B T E B T F x. (3.) Hence, the preconditioned matrix P A C can be written as ] P Θ Â C =, (3.2) Θ 2 I where the precise forms of Θ R (n+p) (n+p) and Θ 2 R (m p) (n+p) are irreevant. From the earier eigenvaue derivation, it is evident that the characteristic poynomia of the preconditioned inear system (3.2)is n m+2p (P A C I) 2(m p) i= (P A C λ i I). In order to prove the upper bound on the Kryov subspace dimension, we need to show that the order of the minimum poynomia is ess than or equa to min{n m + 2p + 2,n+ m}. Expanding the poynomia (P A C I) n m+2p i= (P A C λ i I) of degree n m + 2p +, we obtain (Θ I) ] n m+2p i= (Θ λ i I) n m+2p. Θ 2 i= (Θ λ i I) Since the assumptions of Theorem 2.5 hod, Θ has a fu set of ineary independent eigenvectors and is diagonaizabe. Hence, (Θ I) n m+2p i= (Θ λ i I)=. y e y f
262 H.S. Doar et a. We therefore obtain n m+2p (P A C I) i= (P A C λ i I)= Θ 2 n m+2p i= (Θ λ i I) ]. (3.3) If Θ 2 n m+2p i= (Θ λ i I) =, then the order of the minimum poynomia of P A C is ess than or equa to min{n m + 2p +,n+ m}. IfΘ 2 n m+2p i= (Θ λ i I)=, then the dimension of K(P A C,c)is at most min{n m + 2p + 2,n+ m} since mutipication of (3.3) by another factor (P A C I) gives the zero matrix. If p = m, then triviay K(P A C,b) has dimension at most min{n m + 2p + 2,n+ m}. 3. Custering of eigenvaues when C is sma When using interior-point methods to sove optimization probems, the matrix C is generay diagona and of fu rank. In this case, Theorem 3.2 woud suggest that there is itte advantage of using a constraint preconditioner of the form P over any other preconditioner. However, in interior-point methods the entries of C aso become sma as we get cose to optimaity and, hence, C is sma. In the foowing we sha assume that the norm considered is the 2 norm, but the resuts can be generaized to other norms. Theorem 3.3 Let ζ>, δ, ε and δ 2 + 4ζ(δ ε) then the roots of the quadratic equation λ 2 ζ λ(δ + 2ζ)+ ε + ζ = satisfy λ = + δ 2ζ ± μ, Proof The roots of the quadratic equation satisfy If δ ε ζ, then ( ) δ 2 + δ ε 2ζ ζ μ { } δ 2max 2ζ, δ ε ζ λ = δ + 2ζ ± (δ + 2ζ) 2 4ζ(ε+ ζ) 2ζ = + δ δ 2ζ ± 2 + 4ζ(δ ε) 2ζ ( = + δ ) δ 2 2ζ ± + δ ε. 2ζ ζ 2max { ( δ 2ζ ) } 2, δ ε = { 2max ζ δ 2ζ, } δ ε. ζ
Using constraint preconditioners with reguarized sadde-point probems 263 If δ ε ζ, then the assumption δ 2 + 4ζ(δ ε) impies that ( ) δ 2 ε δ. 2ζ ζ Hence, ( ) δ 2 + δ ε δ 2ζ ζ 2ζ < { } δ 2max 2ζ, ε δ. ζ Remark 3.4 The important point to notice is that if ζ δ and ζ ε, then λ in Theorem 3.3. Theorem 3.5 Assume that the assumptions of Theorem 2.5 hod, then the eigenvaues λ of (2.8) subect to E T BNu, wi satisfy λ =O ( max { C, G A C }) for sma vaues of C. Proof Suppose that C = EDE T is a reduced singuar vaue decomposition of C, where the coumns of E R m p are orthogona and D R p p is diagona with entries d that are non-negative and in non-increasing order. In the foowing,. =. 2, so that C = D =d. Premutipying the quadratic eigenvaue probem (2.8) byu T gives = λ 2 u T Du λ(u T N T GNu + 2u T Du) + (u T N T ANu + u T Du). (3.4) Assume that v = E T BNu and v =, where u is an eigenvector of the above quadratic eigenvaue probem, then u T Du = v T D v = v2 d + v2 2 d 2 + + v2 m d m vt v d = C. Hence, C. ut Du Let ζ = u T Du, δ = u T N T GNu and ε = u T N T ANu, then (3.4) becomes λ 2 ζ λ(δ + 2ζ)+ ε + ζ =.
264 H.S. Doar et a. From Theorem 3.3, λ must satisfy λ = + δ 2ζ ± μ, μ { } δ 2max 2ζ, δ ε. ζ Now δ c N T GN, ε c N T AN, where c is an upper bound on u and u are eigenvectors of (2.8) subect to E T BNu =. Hence, the eigenvaues of (2.8) subect to E T BNu satisfy λ =O ( max { C, G A C }) for sma vaues of C. The resuts of this theorem are not very surprising, but basic eigenvaue perturbation theorems such as Theorem 7.7.2 in ] in conunction with Theorem 2.3 of 6] are weaker than what we have estabished. Specificay, the structure of our coefficient matrix and preconditioner means that we are sti guaranteed to have 2(m p) unit eigenvaues, whereas the more genera eigenvaue perturbation theorems woud ony impy that these eigenvaues wi be cose to. Exampe 3.6 (C with sma entries) Suppose that A C and P are as in Exampe 2.2, but C = a ] for some positive rea number a. Setting D = a ] and E =] (C = EDE T ), the quadratic eigenvaue probem (2.8) is ( ] λ 2 a λ ] 2 + 2 a 2 + + a ]) xy This quadratic eigenvaue probem has three finite eigenvaues: λ = 2, λ = + a ± a + a. x z ] =. For arge vaues of a, λ + a ± a 2 ; the eigenvaues wi be cose to. This custering of part of the spectrum of P A C wi often transate into a speeding up of the convergence of a seected Kryov subspace method,, Sect..3]. 3.2 Numerica exampes We wi carry out severa numerica tests to verify that, in practice, our theoretica resuts transate to a speeding up in the convergence of a seected Kryov subspace method as the entries of C converge towards. Exampe 3.7 The CUTEr test set 3] provides a set of quadratic programming probems. We sha use the probem CVXQP2_M in the foowing two exampes. This probem has n = and m = 25. Barrier penaty terms (in this case α, where α is defined beow) are added to the diagona of A to simuate systems that might arise during an iteration of an interior-point method for such probems. We sha set
Using constraint preconditioners with reguarized sadde-point probems 265 G = diag(a) (ignoring the additiona penaty terms), and C = αi, where α is a positive, rea parameter that we wi change. A tests were performed on a dua Inte Xeon 3.2 GHz machine with hyperthreading and 2 GByte of RAM. It was running Fedora Core 2 (Linux kerne 2.6.8) with MATLAB 7.. We sove the resuting inear systems with unrestarted GMRES ], the Proected Preconditioned Conugate Gradient (PPCG) method 6, Agorithm 5.5.2] and the Simpified Quasi-Minima Residua (SQMR) method 9]. We terminate the iteration when the vaue of residua is reduced by at east a factor of 8 and aways use P and P C as eft preconditioners. We emphasize that for the PPCG method knowedge of the eigenvaues is a you need to describe convergence whereas Greenbaum, Pták and Strakoš show that this is not generay the case with GMRES 5]. In Fig. we compare the performance (in terms of iteration count) between using a preconditioner of the form P and one of the form P C,Eqs.(.3) and (.2) respectivey for the three different iterative methods. Athough the SQMR method doesn t have an optimaity property as was assumed in Sect. 3,asα becomes smaer, we hope that the difference between the number of iterations required by the two preconditioners decreases. We observe that, for this exampe, once α 4 there is itte benefit in reproducing C in the preconditioner in any of the iterative methods tested. However, the SQMR method requires around 9 iterations when α, whist PPCG and GMRES require ust 5 iterations to reach the desired toerance. We woud expect the PPCG and GMRES methods to take around 5 iterations because the preconditioned system has 5 unit eigenvaues and a further 5 custered about one when α ; the remaining 5 eigenvaues ie away from the unit eigenvaues. The SQMR method does not satisfy an optimaity condition and, in this and the foowing exampe, this resuts in substantiay more than 5 iterations being required to reach the desired toerance when α. In this exampe, when α and the preconditioned system P A C has additiona eigenvaues custered around above those 2m p guaranteed to ie at. However, as α decreases, this eigenvaues move away from which resuts in the number of iterations to increase. Exampe 3.8 In this exampe we again use the CVXQP2_M probem from the CUTEr test set. The ony difference to the above exampe is that we sha set C = α diag(,...,,,...,), where rank(c) = m/2. In Fig. 2 we compare the performance (in terms of iteration count) between using a preconditioner of the form P and one of the form P C,Eqs.(.3) and (.2) respectivey for our chosen iterative methods. We observe that if α, then fewer iterations are required in Fig. 2 than in Fig. to reach the required toerance this is as we woud expect because of there now being a guarantee of at east 25 unit eigenvaues in the preconditioned system compared to the possibiity of none. However, as α approaches, the number of eigenvaues custered around wi converge to be the same as in Exampe 3.7. We observe from Figs. and 2 that the number of iterations to reach the required toerance is, as expected, converging to be the same as α. MATLAB code for SQMR can be obtained from the MATLAB Centra Fie Exchange at http://www. mathworks.fr/matabcentra/.
266 H.S. Doar et a. Fig. Comparison of number of iterations required when either (a) P or (b) P C are used as preconditioners for C = αi with GMRES, PPCG and SQMR on the CVXQP2_M probem
Using constraint preconditioners with reguarized sadde-point probems 267 Fig. 2 Comparison of number of iterations required when either (a) P or (b) P C are used as preconditioners for C = α diag(,...,,,...,), where rank C = m/2, with GMRES, PPCG and SQMR on the CVXQP2_M probem
268 H.S. Doar et a. Exampe 3.9 AUG2DQP is another test probem from the CUTEr test set. This probem has n = 328 and m = 6. Barrier penaty terms (in this case α, where α is defined beow) are added to the diagona of A to simuate systems that might arise during an iteration of an interior-point method for such probems. We sha set G = diag(a) (ignoring the additiona penaty terms), and C = αi, where α is a positive, rea parameter that we wi change. In Fig. 3 we observe that once α 4 there is itte benefit in reproducing C in the preconditioner for the PPCG method. Simiary, when C = α diag(,...,,,...,), where rank(c) = m/2, there is itte benefit in reproducing C in the preconditioner for the PPCG method when α 4,Fig.4. These exampes suggest that during premutpy-asymptotic iterations of an interior point method for a noninear programming probem, we may need to use a preconditioner of the form P C, but as the method proceeds there wi be a point at which we wi be abe to swap to using a preconditioner of the form P. From this point Fig. 3 Number of PPCG iterations when either (a) P or (b) P C are used as preconditioners for C = αi on the AUG2DQP probem Fig. 4 Number of PPCG iterations when either (a) P or (b) P C are used as preconditioners for C = α diag(,...,,,...,), where rank C = m/2,onthe AUG2DQP probem
Using constraint preconditioners with reguarized sadde-point probems 269 onwards, we be abe to use the same preconditioner during each iterative sove of the resuting sequence of sadde-point probems. 4 Concusion and further research In this paper, we have investigated a cass of preconditioners for indefinite inear systems that incorporate the (,2) and (2,) bocks of the origina matrix. These bocks are often associated with constraints. We have shown that if C has rank p>, then the preconditioned system has at east 2(m p) unit eigenvaues, regardess of the structure of G. In addition, we have shown that if the entries of C are very sma, then we wi expect an additiona 2p eigenvaues to be custered around and, hence, for the number of iterations required by our chosen Kryov subspace method to be dramaticay reduced. These ater resuts are of particuar reevance to interior point methods for optimization. The practica impications of the anaysis of this paper in the context of soving noninear programming probems wi be the subect of a foow-up paper. We wi investigate the point at which the user shoud switch from using a preconditioner of the form P C to that of P during an interior point method, and how the sub-matrix G in the preconditioner shoud be chosen. Acknowedgements The authors woud ike to thank the referees for their hepfu comments. References. Axesson, O., Barker, V.A.: Finite Eement Soution of Boundary Vaue Probems. Theory and Computation, Cassics in Appied Mathematics, vo. 35. SIAM, Phiadephia (2). Reprint of the 984 origina 2. Benzi, M., Goub, G.H., Liesen, J.: Numerica soution of sadde point probems. Acta Numer. 4, 37 (25) 3. Bergamaschi, L., Gondzio, J., Zii, G.: Preconditioning indefinite systems in interior point methods for optimization. Comput. Optim. App. 28, 49 7 (24) 4. Cafieri, S., D Apuzzo, M., De Simone, V., di Serafino, D.: On the iterative soution of KKT systems in potentia reduction software for arge-scae quadratic probems. Technica Report 9/24, Dept. of Mathematics, Second University of Napes (December 24). Comput. Optim. App. (to appear) 5. Doar, H.S.: Constraint-stye preconditioners for reguarized sadde-point probems. Technica Report 3/26, Dept. of Mathematics, University of Reading (March 26). SIAM J. Matrix Ana. App. (to appear) 6. Doar, H.S.: Iterative inear agebra for constrained optimization. Thesis of Doctor of Phiosophy. Oxford University (25) 7. Eman, H.C., Sivester, D.J., Wathen, A.J.: Finite Eements and Fast Iterative Sovers: with Appications in Incompressibe Fuid Dynamics. Oxford University Press, Oxford (25) 8. Forsgren, A., Gi, P.E., Griffin, J.D.: Iterative soution of augmented systems arising in interior methods. Technica Report NA-5-3, University of Caifornia, San Diego (August 25) 9. Freund, R.W., Nachtiga, N.M.: A new Kryov-subspace method for symmetric indefinite inear systems. In: Ames, W.F. (ed.) Proceedings of the 4th IMACS Word Congress on Computationa and Appied Mathematics, pp. 253 256. IMACS (994). Goub, G.H., Van Loan, C.F.: Matrix Computations, 3rd edn., Johns Hopkins Studies in the Mathematica Sciences. Johns Hopkins University Press, Batimore (996). Goud, N.I.M.: On the accurate determination of search directions for simpe differentiabe penaty functions. IMA J. Numer. Ana. 6, 357 372 (986)
27 H.S. Doar et a. 2. Goud, N.I.M., Hribar, M.E., Noceda, J.: On the soution of equaity constrained quadratic programming probems arising in optimization. SIAM J. Sci. Comput. 23, 376 395 (2) 3. Goud, N.I.M., Orban, D., Toint, P.L.: CUTEr and SifDec: a constrained and unconstrained testing environment, revisited. ACM Trans. Math. Software 29, 373 394 (23) 4. Greenbaum, A.: Iterative Methods for Soving Linear Systems. Frontiers in Appied Mathematics, vo. 7. SIAM, Phiadephia (997) 5. Greenbaum, A., Pták, V., Strakoš, Z.: Any nonincreasing convergence curve is possibe for GMRES. SIAM J. Matrix Ana. App. 7, 465 469 (996) 6. Keer, C., Goud, N.I.M., Wathen, A.J.: Constraint preconditioning for indefinite inear systems. SIAM J. Matrix Ana. App. 2, 3 37 (2) 7. Lukšan, L., Vček, J.: Indefinitey preconditioned inexact Newton method for arge sparse equaity constrained non-inear programming probems. Numer. Linear Agebra App. 5, 29 247 (998) 8. Perugia, I., Simoncini, V.: Bock-diagona and indefinite symmetric preconditioners for mixed finite eement formuations. Numer. Linear Agebra App. 7, 585 66 (2) 9. Rozožník, M., Simoncini, V.: Kryov subspace methods for sadde point probems with indefinite preconditioning. SIAM J. Matrix Ana. App. 24, 368 39 (22) 2. Saad, Y.: Iterative Methods for Sparse Linear Systems, 2nd edn. SIAM, Phiadephia (23) 2. Saad, Y., Schutz, M.H.: GMRES: a generaized minima residua agorithm for soving nonsymmetric inear systems. SIAM J. Sci. Stat. Comput. 7, 856 869 (986) 22. Tisseur, F., Meerbergen, K.: The quadratic eigenvaue probem. SIAM Rev. 43, 235 286 (2) 23. Toh, K.-C., Phoon, K.-K., Chan, S.-H.: Bock preconditioners for symmetric indefinite inear systems. Int. J. Numer. Methods Eng. 6, 36 38 (24) 24. Wright, S.J.: Prima Dua Interior-Point Methods. SIAM, Phiadephia (997)