Primal and dual active-set methods for convex quadratic programming

Transcription

1 Math. Program., Ser. A 216) 159: DOI 1.17/s FULL LENGTH PAPER Prima and dua active-set methods for convex quadratic programming Anders Forsgren 1 Phiip E. Gi 2 Eizabeth Wong 2 Received: 3 Apri 215 / Accepted: 13 November 215 / Pubished onine: 16 December 215 Springer-Verag Berin Heideberg and Mathematica Optimization Society 215 Abstract Computationa methods are proposed for soving a convex quadratic program QP). Active-set methods are defined for a particuar prima and dua formuation of a QP with genera equaity constraints and simpe ower bounds on the variabes. In the first part of the paper, two methods are proposed, one prima and one dua. These methods generate a sequence of iterates that are feasibe with respect to the equaity constraints associated with the optimaity conditions of the prima dua form. The prima method maintains feasibiity of the prima inequaities whie driving the infeasibiities of the dua inequaities to zero. The dua method maintains feasibiity of the dua inequaities whie moving to satisfy the prima inequaities. In each of these methods, the search directions satisfy a KKT system of equations formed from Hessian and constraint components associated with an appropriate coumn basis. The composition of the basis is specified by an active-set strategy that guarantees the nonsinguarity of each set of KKT equations. Each of the proposed methods is a conventiona active-set method in the sense that an initia prima- or dua-feasibe point is required. In the A. Forsgren Research supported by the Swedish Research Counci VR). P. E. Gi, E. Wong Research supported in part by Northrop Grumman Aerospace Systems, and Nationa Science Foundation Grants DMS and DMS B Eizabeth Wong ewong@ucsd.edu Anders Forsgren andersf@kth.se Phiip E. Gi pgi@ucsd.edu 1 Optimization and Systems Theory, Department of Mathematics, KTH Roya Institute of Technoogy, 1 44 Stockhom, Sweden 2 Department of Mathematics, University of Caifornia, San Diego, La Joa, CA , USA

2 47 A. Forsgren et a. second part of the paper, it is shown how the quadratic program may be soved as a couped pair of prima and dua quadratic programs created from the origina by simutaneousy shifting the simpe-bound constraints and adding a penaty term to the objective function. Any conventiona coumn basis may be made optima for such a prima dua pair of shifted-penaized probems. The shifts are then updated using the soution of either the prima or the dua shifted probem. An obvious appication of this approach is to sove a shifted dua QP to define an initia feasibe point for the prima or vice versa). The computationa performance of each of the proposed methods is evauated on a set of convex probems from the CUTEst test coection. Keywords Quadratic programming Active-set methods Convex quadratic programming Prima active-set methods Dua active-set methods Mathematics Subject Cassification 9C2 1 Introduction We consider the formuation and anaysis of active-set methods for a convex quadratic program QP) of the form 1 minimize x R n, y R m 2 x T Hx yt My + c T x subject to Ax + My = b, x, 1) where A, b, c, H and M are constant, with H and M symmetric positive semidefinite. In order to simpify the theoretica discussion, the inequaities of 1) invove nonnegativity constraints ony. However, the methods to be described are easiy extended to treat a forms of inear constraints. Numerica resuts are given for probems with constraints in the form x L x x U and b L Ax b U, for fixed vectors x L, x U, b L and b U.) If M =, the QP 1) is a conventiona convex quadratic program with constraints defined in standard form. A reguarized quadratic program may be obtained by defining M = μi for some sma positive parameter μ. For appications that require the soution of a reguarized QP see, e.g., [1,32,6].) Active-set methods for quadratic programming probems of the form 1) sove a sequence of inear equations that invove the y-variabes and a subset of the x-variabes. Each set of equations constitutes the optimaity conditions associated with an equaityconstrained quadratic subprobem. The goa is to predict the optima active set, i.e., the set of constraints that are satisfied with equaity, at the soution of the probem. A conventiona active-set method has two phases. In the first phase, a feasibe point is found whie ignoring the objective function; in the second phase, the objective is minimized whie feasibiity is maintained. A usefu feature of active-set methods is that they are we-suited for warm starts, where a good estimate of the optima active set is used to start the agorithm. This is particuary usefu in appications where a sequence of quadratic programs is soved, e.g., in a sequentia quadratic programming method or in an ODE- or PDE-constrained probem with mesh refinement. Other appications of active-set methods for quadratic programming incude mixed-

3 Prima and dua active-set methods for convex quadratic integer noninear programming, portfoio anaysis, structura anaysis, and optima contro. In Sect. 2, the prima and dua forms of a convex quadratic program with constraints in standard form are generaized to incude genera ower bounds on both the prima and dua variabes. These probems constitute a prima dua pair that incudes probem 1) and its associated dua as a specia case. In Sects. 3 and 4, an active-set method is proposed for each of the prima and dua forms associated with the generaized probem of Sect. 2. Both of these methods provide a sequence of iterates that are feasibe with respect to the equaity constraints associated with the optimaity conditions of the prima dua probem pair. The prima method maintains feasibiity of the prima inequaities whie driving the infeasibiities of the dua inequaities to zero. By contrast, the dua method maintains feasibiity of the dua inequaities whie moving to satisfy the prima inequaities. In each of these methods, the search directions satisfy a KKT system of equations formed from Hessian and constraint components associated with an appropriate coumn basis. The composition of the basis is specified by an active-set strategy that guarantees the nonsinguarity of each set of KKT equations. The methods formuated in Sects. 3 4 define conventiona active-set methods in the sense that an initia feasibe point is required. In Sect. 5, a method is proposed that soves a pair of couped quadratic programs created from the origina by simutaneousy shifting the simpe-bound constraints and adding a penaty term to the objective function. Any conventiona coumn basis can be made optima for such a prima dua pair of shifted-penaized probems. The shifts are then updated using the soution of either the prima or the dua shifted probem. An obvious appication of this idea is to sove a shifted dua QP to define an initia feasibe point for the prima, or vice-versa. In addition to the obvious benefit of using the objective function whie getting feasibe, this approach provides an effective method for finding a dua-feasibe point when H is positive semidefinite and M =. Finding a dua-feasibe point is reativey straightforward for the stricty convex case, i.e., when H is positive definite. However, in the genera case, the dua constraints for the phase-one inear program invove entries from H as we as A, which compicates the formuation of the phase-one method consideraby. Finay, in Sect. 7 some numerica experiments are presented for a simpe Matab impementation of a couped prima dua method appied to a set of convex probems from the CUTEst test coection [45,47]. There are a number of aternative active-set methods avaiabe for soving a QP with constraints written in the format of probem 1). Broady speaking, these methods fa into three casses defined here in the order of increasing generaity: i) methods for stricty convex quadratic programming H symmetric positive definite) [2,34,41,55, 58]; ii) methods for convex quadratic programming H symmetric positive semidefinite) [8,4,51,52,59]; and iii) methods for genera quadratic programming no assumptions on H other than symmetry) [3,4,12,24,27,31,33,36,37,42 44,5,59]. Of the methods specificay designed for convex quadratic programming, ony the methods of Boand [8] and Wong [59, Chapter 4] are dua active-set methods. Some existing active-set quadratic programming sovers incude QPOPT [38], QPSchur [2], SQOPT [4], SQIC [33] and QPA part of the GALAHAD software ibrary) [46].

4 472 A. Forsgren et a. The prima active-set method proposed in Sect. 3 is motivated by the methods of Fetcher [24], Goud [42], and Gi and Wong [33], which may be viewed as methods that extend the properties of the simpex method to genera quadratic programming. At each iteration, a direction is computed that satisfies a nonsinguar system of inear equations based on an estimate of the active set at a soution. The equations may be written in symmetric form and invove both the prima and dua variabes. In this context, the purpose of the active-set strategy is not ony to obtain a good estimate of the optima active set, but aso to ensure that the systems of inear equations that must be soved at each iteration are nonsinguar. This strategy aows the appication of any convenient inear sover for the computation of the iterates. In this paper, these ideas are appied to convex quadratic programming. The resuting sequence of iterates is the same as that generated by an agorithm for genera QP, but the structure of the iteration is different, as is the structure of the inear equations that must be soved. Simiar ideas are used to formuate the new dua active-set method proposed in Sect. 4. The proposed prima, dua, and combined prima dua methods use a conventiona active-set approach in the sense that the constraints remain unchanged during the soution of a given QP. Aternative approaches that use a parametric active-set method have been proposed by Best [5,6], Ritter [56,57], Ferreau et a. [22], Potschka et a. [54], and impemented in the qpoases package by Ferreau et a. [23]. Prima methods based on the augmented Lagrangian method have been proposed by Debos and Gibert [18], Chiche and Gibert [15], and Gibert and Joannopouos [3]. The use of shifts for the bounds have been suggested by Cartis and Goud [13] in the context of interior methods for inear programming. Another cass of active-set methods that are convergent for stricty convex quadratic programs have been considered by Curtis et a. [16]. Notation and terminoogy Given vectors a and b with the same dimension, mina, b) is a vector with components mina i, b i ). The vectors e and e j denote, respectivey, the coumn vector of ones and the jth coumn of the identity matrix I. The dimensions of e, e i and I are defined by the context. Given vectors x and y, the coumn vector consisting of the components of x augmented by the components of y is denoted by x, y). 2 Background Athough the purpose of this paper is the soution of quadratic programs of the form 1), for reasons that wi become evident in Sect. 5, the anaysis wi focus on the properties of a pair of probems that may be interpreted as a prima dua pair of QPs associated with probem 1). It is assumed throughout that the matrix A M ) associated with the equaity constraints of probem 1) has fu row rank. This assumption can be made without oss of generaity, as shown in Proposition 12 of the Appendix. The paper invoves a number of other basic theoretica resuts that are subsidiary to the main presentation. The proofs of these resuts are reegated to the Appendix.

5 Prima and dua active-set methods for convex quadratic Formuation of the prima and dua probems For given constant vectors q and r, consider the pair of convex quadratic programs PQP q,r ) 1 minimize x,y 2 x T Hx yt My + c T x + r T x subject to Ax + My = b, x q, and DQP q,r ) maximize 1 x,y,z 2 x T Hx 2 1 yt My + b T y q T z subject to Hx + A T y + z = c, z r. The foowing resut gives joint optimaity conditions for the tripe x, y, z) such that x, y) is optima for PQP q,r ), and x, y, z) is optima for DQP q,r ). If q and r are zero, then PQP, ) and DQP, ) are the prima and dua probems associated with 1). For arbitrary q and r, PQP q,r ) and DQP q,r ) are essentiay the dua of each other, the difference is ony an additive constant in the vaue of the objective function. Proposition 1 Let q and r denote constant vectors in R n. If x, y, z) is a given tripe in R n R m R n, then x, y) is optima for PQP q,r ) and x, y, z) is optima for DQP q,r ) if and ony if Hx + c A T y z =, 2a) Ax + My b =, 2b) x + q, 2c) z + r, 2d) x + q) T z + r) =. 2e) In addition, the optima objective vaues satisfy optvapqp q,r ) optvadqp q,r ) = q T r. Finay, 2) has a soution if and ony if the sets { x, y, z) : Hx + A T y + z = c, z r } and { x, y) : Ax + My = b, x q } are both nonempty. Proof Let the vector of Lagrange mutipiers for the constraints Ax + My b = be denoted by ỹ. Without oss of generaity, the Lagrange mutipiers for the bounds x + q ofpqp q,r ) may be written in the form z + r, where r is the given fixed vector r. With these definitions, a Lagrangian function Lx, y, ỹ, z) associated with PQP q,r ) is given by Lx, y, ỹ, z) = 2 1 x T Hx + c + r) T x yt My ỹ T Ax + My b) z + r) T x + q).

6 474 A. Forsgren et a. Stationarity of the Lagrangian with respect to x and y impies that Hx + c + r A T ỹ z r = Hx + c A T ỹ z =, My M ỹ =. 3a) 3b) The optimaity conditions for PQP q,r ) are then given by: i) the feasibiity conditions 2b) and 2c); ii) the nonnegativity conditions 2d) for the mutipiers associated with the bounds x +q ; iii) the stationarity conditions 3); and iv) the compementarity conditions 2e). The vector y appears ony in the term My of 2b) and 3b). In addition, 3b) impies that My = M ỹ, in which case we may choose y = ỹ. This common vaue of y and ỹ must satisfy 3a), which is then equivaent to 2a). The optimaity conditions 2)forPQP q,r ) foow directy. With the substitution ỹ = y, the expression for the Lagrangian may be rearranged so that Lx, y, y, z) = 1 2 x T Hx 1 2 yt My+b T y q T z +Hx+c A T y z) T x q T r. 4) Taking into account 3) fory = ỹ, the dua objective is given by 4) as 2 1 x T Hx 1 2 yt My+b T y q T z q T r, and the dua constraints are Hx+c A T y z = and z+r. It foows that DQP q,r ) is equivaent to the dua of PQP q,r ), the ony difference is the constant term q T r in the objective, which is a consequence of the shift z + r in the dua variabes. Consequenty, strong duaity for convex quadratic programming impies optvapqp q,r ) optvadqp q,r ) = q T r. In addition, the variabes x, y and z satisfying 2) are feasibe for PQP q,r ) and DQP q,r ) with the difference in the objective function vaue being q T r.itfoowsthatx, y, z) is optima for DQP q,r )aswe as PQP q,r ). Finay, feasibiity of both PQP q,r ) and DQP q,r ) is both necessary and sufficient for the existence of optima soutions. 2.2 Optimaity conditions and the KKT equations The proposed methods are based on maintaining index sets B and N that define a partition of the index set I ={1, 2,, n}, i.e., I = B N with B N =. Foowing standard terminoogy, we refer to the subvectors x B and x N associated with an arbitrary x as the basic and nonbasic variabes, respectivey. The crucia feature of B is that it defines a unique soution x, y, z) to the equations Hx + c A T y z =, x N + q N =, Ax + My b =, z B + r B =. 5) For the symmetric Hessian H, the matrices and H NN denote the subset of rows and coumns of H associated with the sets B and N, respectivey. The unsymmetric matrix of components h ij with i B and j N wi be denoted by H BN. Simiary, A B and A N denote the matrices of coumns of A associated with B and N respectivey.

7 Prima and dua active-set methods for convex quadratic With this notation, the Eq. 5) may be written in partitioned form as x B + H BN x N + c B A T y z =, x B B N + q N =, H T x + H BN B NN x N + c N A T y z =, z N N B + r B =, A B x B + A N x N + My b =. Eiminating x N and z B from these equations using the equaities x N + q N z B + r B = yieds the symmetric equations HBB A T ) ) ) B xb HBN q = N c B r B M y A N q N + b A B = and 6) for x B and y. It foows that 5) has a unique soution if and ony if 6) has a unique soution. Therefore, if B is chosen to ensure that 5) has a unique soution, it must foow from 6) that the matrix K B such that HBB A K B = T ) B 7) A B M is nonsinguar. Once x B and y have been computed, the z N -variabes are given by z N = H T x H q + c BN B NN N N AT N y. 8) As in Gi and Wong [33], any set B such that K B is nonsinguar is referred to as a second-order consistent basis. Methods that impose restrictions on the eigenvaues of K B are known as inertia-controing methods. For a description of inertia-controing methods for genera quadratic programming, see, e.g., [33,37].) The two methods proposed in this paper, one prima, one dua, generate a sequence of iterates that satisfy the Eq. 5) for some partition B and N. If the conditions 5) are satisfied, the additiona requirement for fufiing the optimaity conditions of Proposition 1 are x B +q B and z N +r N. The prima method of Sect. 3 imposes the restriction that x B + q B, which impies that the sequence of iterates is prima feasibe. In this case the method terminates when z B + r B is satisfied. Conversey, the dua method of Sect. 4 imposes dua feasibiity by means of the bounds z N +r N and terminates when x B + q B. In both methods, an iteration starts and ends with a second-order consistent basis, and comprises one or more subiterations. In each subiteration an index and index sets B and N are known such that B {} N ={1, 2,, n}. This partition defines a search direction x, y, z) that satisfies the identities H x A T y z =, x N =, A x + M y =, z B =. 9) As / B and / N, these conditions impy that neither x nor z are restricted to be zero. The conditions x N = and z B = impy that 9) may be expressed in the partitioned-matrix form

8 476 A. Forsgren et a. h h T B a T x 1 h B A T x B B h N H T BN A T N I y z =, a A B M z N where h denotes the th diagona of H, and the coumn vectors h B and h N denote the coumn vectors of eements h i and h j with i B, and j N, respectivey. It foows that x, x B, y and z satisfy the homogeneous equations h h T B a T x 1 h B A T x B B y =, 1a) a A B M z and z N is given by z N = h N x + H T BN x B AT N y. 1b) The properties of these equations are estabished in the next subsection. 2.3 The inear agebra framework This section estabishes the inear agebra framework that serves to emphasize the underying symmetry between the prima and dua methods. It is shown that the search direction for the prima and the dua method is a nonzero soution of the homogeneous equations 1a), i.e., every direction is a nontrivia nu vector of the matrix of 1a). In particuar, it is shown that the nu-space of 1a) has dimension one, which impies that the soution of 1a) is unique up to a scaar mutipe. The ength of the direction is then competey determined by fixing either x = 1or z = 1. The choice of which component to fix depends on whether or not the corresponding component in a nu vector of 1a) is nonzero. The conditions are stated precisey in Propositions 3 and 4 beow. The first resut shows that the components x and z of any direction x, y, z) satisfying the identities 9) must be such that x z. Proposition 2 If the vector x, y, z) satisfies the identities H x A T y z =, A x + M y =, then x T z = x T H x + y T M y. Moreover, given an index and index sets B and N such that B {} N ={1, 2,,n} with x N = and z B =, then x z = x T H x + y T M y.

9 Prima and dua active-set methods for convex quadratic Proof Premutipying the first identity by x T and the second by y T gives x T H x x T A T y x T z =, and y T A x + y T M y =. Eiminating the term x T A T y gives x T H x + y T M y = x T z. By definition, H and M are symmetric positive semidefinite, which gives x T z. In particuar, if B {} N ={1, 2,, n}, with x N = and z B =, it must hod that x T z = x z. The set of vectors x, x B, y, z, z N ) satisfying the Eq. 1) is competey characterized by the properties of the matrices K B and K such that HBB A K B = T B A B M ) h h T B a T and K = h B A T B. 11) a A B M The properties are summarized by the resuts of the foowing two propositions. Proposition 3 Assume that K B is nonsinguar. Let x be a given nonnegative scaar. 1. If x =, then the ony soution of 1) is zero, i.e., x B =, y =, z = and z N =. 2. If x >, then the quantities x B, y, z and z N of 1) are unique and satisfy the equations HBB A T ) ) ) B xb hb = x A B M y a, z = h x + h T x B B at y, z N = h N x + H T x BN B AT y. N 12) Moreover, either i) K is nonsinguar and z >, or ii) K is singuar and z =, in which case it hods that y =, z N =, and the mutipicity of the zero eigenvaue of K is one, with corresponding eigenvector x, x B, ). Proof Proposition 2 impies that z if x >, which impies that the statement of the proposition incudes a possibe vaues of z. The second and third bocks of the Eq. 1a) impy that ) hb HBB A x a + T ) ) B xb =. 13) A B M y ) As K B is nonsinguar by assumption, the vectors x B and y must constitute the unique soution of 13) for a given vaue of x. Furthermore, given x B and y,the quantities z and z N of 12) are aso uniquey defined. The specific vaue x =,

10 478 A. Forsgren et a. gives x B = and y =, so that z = and z N =. It foows that x must be nonzero for at east one of the vectors x B, y, z or z N to be nonzero. Next it is shown that if x >, then either 2i) or 2ii) must hod. For 2i), it is necessary to show that if x > and K is nonsinguar, then z >. If K is nonsinguar, the homogeneous equations 1a) may be written in the form h h T B a T x 1 h B A T x B B = z, 14) a A B M y which impies that x, x B and y are unique for a given vaue of z. In particuar, if z = then x =, which woud contradict the assumption that x >. If foows that z must be nonzero. Finay, Proposition 2 impies that if z is nonzero and x >, then z > as required. For the first part of 2ii), it must be shown that if K is singuar, then z =. If K is singuar, it must have a nontrivia nu vector p, p B, u). Moreover, every nu vector must have a nonzero p, because otherwise p B, u) woud be a nontrivia nu vector of K B, which contradicts the assumption that K B is nonsinguar. A fixed vaue of p uniquey defines p B and u, which indicates that the mutipicity of the zero eigenvaue must be one. A simpe substitution shows that p, p B, u, v ) is a nontrivia soution of the homogeneous equation 1a) such that v =. As the subspace of vectors satisfying 1a) is of dimension one, it foows that every soution is unique up to a scaar mutipe. Given the properties of the known soution p, p B, u, ), it foows that every soution x, x B, y, z )of1a) is an eigenvector associated with the zero eigenvaue of K, with z =. For the second part of 2ii), if z =, the homogeneous equations 1a) become h h T B a T x h B A T x B B =. 15) a A B M y As K is singuar in 15), Proposition 5 of the Appendix impies that h h T ) B a T h B x =, and A x T B y =. 16) B M a A B The nonsinguarity of K B impies that A B M ) has fu row rank, in which case the second equation of 16) gives y =. It foows that every eigenvector of K associated with the zero eigenvaue has the form x, x B, ). It remains to show that z N =. If Proposition 6 of the Appendix is appied to the first equation of 16), then it must hod that h h T ) B h B x =. x B h N H T BN

11 Prima and dua active-set methods for convex quadratic It foows from the definition of z N in 12) that z N = h N x + H T BN x B AT N y =, which competes the proof. Proposition 4 Assume that K is nonsinguar. Let z be a given nonnegative scaar. 1. If z =, then the ony soution of 1) is zero, i.e., x =, x B =, y = and z N =. 2. If z >, then the quantities x, x B, y and z N of 1) are unique and satisfy the equations h h T B a T h B A T x 1 x B B = z, 17a) a A B M y z N = H N x + H T BN x B AT N y. 17b) Moreover, either i) K B is nonsinguar and x >, or ii) K B is singuar and x =, in which case, it hods that x B = and the mutipicity of the zero eigenvaue of K B is one, with corresponding eigenvector, y). Proof In Proposition 2 it is estabished that x if z >, which impies that the statement of the proposition incudes a possibe vaues of x. It foows from 1a) that x, x B, and y must satisfy the equations h h T B a T h B A T B a A B M x z x B =. 18) y Under the given assumption that K is nonsinguar, the vectors x, x B and y are uniquey determined by 18) for a fixed vaue of z. In addition, once x, x B and y are defined, z N is uniquey determined by 17b). It foows that if z =, then x =, x B =, y = and z N =. It remains to show that if z >, then either 2i) or 2ii) must hod. If K B is singuar, then Proposition 5 of the Appendix impies that there must exist u and v such that HBB A B ) u = ) and ) A T B v =. M ) Proposition 6 of the Appendix impies that the vector u must aso satisfy h T u =. B If u is nonzero, then, u, ) is a nontrivia nu vector for K, which contradicts the assumption that K is nonsinguar. It foows that A T ) B has fu row rank and the singuarity of K B must be caused by dependent rows in A B M ). The nonsinguarity of K impies that a A B M ) has fu row rank and there must exist a vector v

12 48 A. Forsgren et a. such that v T a =, v T A B = and v T M =. If v is scaed so that v T a = z, then,, v) must be a soution of 18). It foows that x =, v = y, and, y) is an eigenvector of K B associated with a zero eigenvaue. The nonsinguarity of K impies that v is unique given the vaue of the scaar z, and hence the zero eigenvaue has mutipicity one. Conversey, x = impies that x B, y) is a nu vector for K B. However, if K B is nonsinguar, then the vector is zero, contradicting 17a). It foows that K B must be singuar. 3 A prima active-set method for convex QP In this section a prima-feasibe method for convex QP is formuated. Each iteration begins and ends with a point x, y, z) that satisfies the conditions Hx + c A T y z =, x N + q N =, x B + q B, Ax + My b =, z B + r B =, 19) for appropriate second-order consistent bases. The purpose of the iterations is to drive x, y, z) to optimaity by driving the dua variabes to feasibiity i.e., by driving the negative components of z N + r N to zero). Methods for finding B and N at the initia point are discussed in Sect. 5. An iteration consists of a group of one or more consecutive subiterations during which a specific dua variabe is made feasibe. The first subiteration is caed the base subiteration. In some cases ony the base subiteration is performed, but, in genera, additiona intermediate subiterations are required. At the start of the base subiteration, an index in the nonbasic set N is identified such that z + r <. The idea is to remove the index from N i.e., N N \{}) and attempt to increase the vaue of z + r by taking a step aong a prima-feasibe direction x, x B, y, z ). The remova of from N impies that B {} N ={1, 2,, n} with B second-order consistent. This impies that K B is nonsinguar and the unique) search direction may be computed as in 12) with x = 1. If z >, the step α = z + r )/ z gives z + α z + r =. Otherwise, z =, and there is no finite vaue of α that wi drive z + α z + r to its bound, and α is defined to be +. Proposition 11 of the Appendix impies that the case z = corresponds to the prima objective function being inear and decreasing aong the search direction. Even if z is positive, it is not aways possibe to take the step α and remain prima feasibe. A positive step in the direction x, x B, y, z ) must increase x from its bound, but may decrease some of the basic variabes. This makes it necessary to imit the step to ensure that the prima variabes remain feasibe. The argest step ength that maintains prima feasibiity is given by α max = min i: x i < x i + q i x i.

13 Prima and dua active-set methods for convex quadratic If α max is finite, this vaue gives x k + α max x k + q k =, where k is the index k = argmin i: xi < x i + q i )/ x i ). The overa step ength is then α = min α,α max ). An infinite vaue of α indicates that the prima probem PQP q,r ) is unbounded, or, equivaenty, that the dua probem DQP q,r ) is infeasibe. In this case, the agorithm is terminated. If the step α = α is taken, then z + α z + r =, the subiterations are terminated with no intermediate subiterations and B B {}. Otherwise, α = α max, and the basic and nonbasic sets are updated as B B\{k} and N N {k} giving a new partition B {} N ={1, 2,, n}. In order to show that the equations associated with the new partition are we-defined, it is necessary to show that aowing z k to move does not give a singuar K. Proposition 9 of the Appendix shows that the submatrix K associated with the updated B and N is nonsinguar for the cases z > and z =. Because the remova of k from B does not ater the nonsinguarity of K,itis possibe to add to B and thereby define a unique soution of the system 5). However, if z + r <, additiona intermediate subiterations are required to drive z + r to zero. In each of these subiterations, the search direction is computed by choosing z = 1 in Proposition 4. The step ength α is given by α = z + r )/ z as in the base subiteration above, but now α is aways finite because z = 1. Simiar to the base subiteration, if no constraint is added, then z + α z + r =. Otherwise, the index of another bocking variabe k is moved from B to N. Proposition 9 impies that the updated matrix K is nonsinguar at the end of an intermediate subiteration. As a consequence, the intermediate subiterations may be repeated unti z + r is driven to zero. At the end of the base subiteration or after the intermediate subiterations are competed, it must hod that z + r = and the fina K is nonsinguar. This impies that a new iteration may be initiated with the new basic set B {} defining a nonsinguar K B. The prima active-set method is summarized in Agorithm 1 beow. The convergence properties of Agorithm 1 are estabished in Sect. 5, which concerns a genera prima agorithm that incudes Agorithm 1 as a specia case. 4 A dua active-set method for convex QP Each iteration of the dua active-set method begins and ends with a point x, y, z) that satisfies the conditions Hx + c A T y z =, x N + q N =, Ax + My b =, z B + r B =, z N + r N, 2) for appropriate second-order consistent bases. For the dua method, the purpose is to drive the prima variabes to feasibiity i.e., by driving the negative components of x + q to zero). An iteration begins with a base subiteration in which an index in the basic set B is identified such that x + q <. The corresponding dua variabe z may be increased from its current vaue z = r by removing the index from B, and defining

14 482 A. Forsgren et a. Agorithm 1 A prima active-set method for convex QP. Find x, y, z) satisfying conditions 19) for some second-order consistent basis B; whie : z + r < do N N \{}; prima_baseb, N,, x, y, z); [returns B, N, x, y, z] whie z + r < do prima_intermediateb, N,, x, y, z); [returns B, N, x, y, z] end whie B B {}; end whie function prima_baseb, N,, x, y, z) HBB A x 1; Sove T B A B M ) xb y ) ) hb = ; a z N h N x + H T BN x B AT N y; z h x + h T B x B at y; [ z ] α z + r )/ z ; [α + if z = ] α max min i + q i )/ x i ); k argmin x i + q i )/ x i ); i: x i < i: x i < α min ) α,α max ; if α =+ then stop; [DQP q,r ) is infeasibe] end if x x + α x ; x B x B + α x B ; y y + α y; z z + α z ; z N z N + α z N ; if z + r < then B B\{k}; N N {k}; end if return B, N, x, y, z; end function function prima_intermediateb, N,, x, y, z) h h T B a T z 1; Sove h B A T x x B B = 1 ; [ x ] a A B M y z N H N x + H T BN x B A T N y; α z + r ); α max min x i + q i )/ x i ); i: x i < k argmin x i + q i )/ x i ); i: x i < α min ) α,α max ; x x + α x ; x B x B + α x B ; y y + α y; z z + α z ; z N z N + α z N ; if z + r < then B B\{k}; N N {k}; end if return B, N, x, y, z; end function B B\{}. Once is removed from B, it hods that B {} N ={1, 2,, n}.the resuting matrix K of 11) is nonsinguar, and the unique direction x, x B, y) may be computed with z = 1 in Proposition 4. If x >, the step α = x + q )/ x gives x + α x + q =. Otherwise, x = and Proposition 11 of the Appendix impies that the dua objective function is inear and increasing aong x, y, z). In this case α =+.Asx + q is increased towards zero, some nonbasic dua variabes may decrease and the step must

15 Prima and dua active-set methods for convex quadratic be imited by α max = min i: zi < z i + r i ) z i ) to maintain feasibiity of the nonbasic dua variabes. This gives the step α = min ) α,α max.ifα =+, the dua probem is unbounded and the iteration is terminated. This is equivaent to the prima probem PQP q,r ) being infeasibe. If α = α, then x + α x + q =. Otherwise, it must hod that α = α max and N and B are redefined as N = N \{k} and B = B {k}, where k is the index k = argmin i: zi < z i + r i )/ z i ). The partition at the new point satisfies B {} N ={1, 2,, n}. Proposition 1 of the Appendix shows that the new K B is nonsinguar for both of the cases x > and x =. If x +q < at the new point, then at east one intermediate subiteration is necessary to drive x +q to zero. The nonsinguarity of K B impies that the search direction may be computed with x = 1 in Proposition 3. As in the base subiteration, the step ength is α = x + q )/ x, but in this case α can never be infinite because x = 1. If no constraint index is added to B, then x + α x + q =. Otherwise, the index k of a bocking variabe is moved from N to B. Proposition 1 of the Appendix impies that the updated K B is nonsinguar at the end of an intermediate subiteration. Once x + q is driven to zero, the index is moved to N and a new iteration is started. The dua active-set method is summarized in Agorithm 2 beow. Its convergence properties are discussed in Sect Combining prima and dua active-set methods The prima active-set method proposed in Sect. 3 maybeusedtosovepqp q,r )for a given initia second-order consistent basis satisfying the conditions 19). An appropriate initia point may be found by soving a conventiona phase-1 inear program. Aternativey, the dua active-set method of Sect. 4 may be used in conjunction with an appropriate phase-1 procedure to sove the quadratic program PQP q,r ) for a given initia second-order consistent basis satisfying the conditions 2). In this section a method is proposed that provides an aternative to the conventiona phase-1/phase-2 approach. It is shown that a pair of couped quadratic programs may be created from the origina by simutaneousy shifting the bound constraints. Any second-order consistent basis can be made optima for such a prima dua pair of shifted probems. The shifts are then updated using the soution of either the prima or the dua shifted probem. An obvious appication of this approach is to sove a shifted dua QP to define an initia feasibe point for the prima, or vice-versa. This strategy provides an aternative to the conventiona phase-1/phase-2 approach that utiizes the QP objective function whie finding a feasibe point. 5.1 Finding an initia second-order-consistent basis For the methods described in Sect. 5.2 beow, it is possibe to define a simpe procedure for finding the initia second-order consistent basis B such that the matrix K B of 7)is nonsinguar. The required basis may be obtained by finding a symmetric permutation Π of the fu KKT matrix K such that

16 484 A. Forsgren et a. Agorithm 2 A dua active-set method for convex QP. Find x, y, z) satisfying conditions 2) for some second-order consistent basis B; whie : x + q < do B B\{}; dua_baseb, N,, x, y, z); [Base subiteration] whie x + q < do dua_intermediateb, N,, x, y, z); [Intermediate subiteration] end whie N N {}; end whie function dua_baseb, N,, x, y, z) h h T B a T z 1; Sove h B A T x x B B = 1 ; [ x ] a A B M y z N h N x + H T BN x B AT N y; α x + q )/ x ; [α + if x = ] α max min z i + r i )/ z i ); k argmin z i + r i )/ z i ); i: z i < i: z i < α min ) α,α max ; if α =+ then stop; [PQP q,r ) is infeasibe] end if x x + α x ; x B x B + α x B ; y y + α y; z z + α z ; z N z N + α z N ; if x + q < then B B {k}; N N \{k}; end if return B, N, x, y, z; end function function dua_intermediateb, N,, x, y, z) HBB A x 1; Sove T ) ) ) B xb hb = ; A B M y a z h x + h T B x B at y; [ z ] z N h N x + H T BN x B AT N y; α x + q ); α max min z i + r i )/ z i ); i: z i < k argmin z i + r i )/ z i ); i: z i < α min ) α,α max ; x x + α x ; x B x B + α x B ; y y + α y; z z + α z ; z N z N + α z N ; if x + q < then B B {k}; N N \{k}; end if return B, N, x, y, z; end function Π T K Π = Π T H A A T M ) Π = A T B H BN A B M A N H T BN A T N H NN, 21) where the eading principa bock 2 2 submatrix is of the form 7). The fu row-rank assumption on A M ) ensures that the permutation 21) is we defined, see [28, Section 6]. In practice, the permutation may be determined using any method for

17 Prima and dua active-set methods for convex quadratic finding a symmetric indefinite factorization of K, see, e.g., [1,11,25]. Such methods use symmetric interchanges that impicity form the nonsinguar matrix K B by deferring singuar pivots. In this case, K B may be defined as any submatrix of the argest nonsinguar principa submatrix obtained by the factorization. There may be further permutations within Π that are not reevant to this discussion; for further detais, see, e.g., [2,21,28,29].) The permutation Π defines the initia B-N partition of the coumns of A, i.e., it defines an initia second-order consistent basis. 5.2 Initiaizing the shifts Given a second-order consistent basis, it is straightforward to create shifts q ), r ) ) and corresponding x, y, z) so that q ), r ) and x, y, z) are optima for PQP q ),r )) and DQP q ),r )). First, choose nonnegative vectors q) N and r ) B. Obvious choices are q ) N = and r ) B =.) Define z B = r ) B, x N = q ) N, and sove the nonsinguar KKT-system 6) to obtain x B and y, and compute z N from 8). Finay, et q ) B max{ x B, } and r ) N max{ z N, }. Then, it foows from Proposition 1 that x, y and z are optima for the probems PQP q ),r )) and DQP q ),r )), with q ) and r ). If q ) and r ) are zero, then x, y and z are optima for the origina probem. 5.3 Soving the origina probem by removing the shifts The origina probem may now be soved as a pair of shifted quadratic programs. Two aternative strategies are proposed. The first is a prima first strategy in which a shifted prima quadratic program is soved, foowed by a dua. The second is an anaogous dua first strategy. The prima-first strategy is summarized as foows. ) Find B, N, q ), r ), x, y, z, as described in Sects. 5.1 and ) Set q 1) = q ), r 1) =. Sove PQP q, ) using the prima active-set method. 2) Set q 2) =, r 2) =. Sove DQP, ) using the dua active-set method. In steps 1) and 2), the initia B N partition and initia vaues of x, y, and z are defined as the fina B N partition and fina vaues of x, y, and z from the preceding step. The dua-first strategy is defined in an anaogous way. ) Find B, N, q ), r ), x, y, z, as described in Sects. 5.1 and ) Set q 1) =, r 1) = r ). Sove DQP,r ) using the dua active-set method. 2) Set q 2) =, r 2) =. Sove PQP, ) using the prima active-set method. As in the prima-first strategy, the initia B N partition and initia vaues of x, y, and z for steps 1) and 2), are defined as the fina B N partition and fina vaues of x, y, and z from the preceding step. The strategies of soving two consecutive quadratic programs may be generaized to a sequence of more than two quadratic programs, where we aternate between prima and dua active-set methods, and eiminate the shifts in more than two steps.)

18 486 A. Forsgren et a. In order for these approaches to be we-defined, a simpe generaization of the prima and dua active-set methods of Agorithms 1 and 2 is required. 5.4 Reaxed initia conditions for the prima QP method For Agorithm 1, the initia vaues of B, N, q, r, x, y, and z must satisfy conditions 19). However, the choice of r = r 2) = in Step 2) of the dua-first strategy may give some negative components in the vector z B +r B. This possibiity may be handed by defining a simpe generaization of Agorithm 1 that aows initia points satisfying the conditions Hx + c A T y z =, x N + q N =, x B + q B, Ax + My b =, z B + r B, 22) instead of the conditions 19). In Agorithm 1, the index identified at the start of the prima base subiteration is seected from the set of nonbasic indices such that z j + r j <. In the generaized agorithm, the set of eigibe indices for is extended to incude indices associated with negative vaues of z B + r B. If the index is deeted from B, the associated matrix K is nonsinguar, and intermediate subiterations are executed unti the updated vaue satisfies z + r =. At this point, the index is returned B. The method is summarized in Agorithm 3. Agorithm 3 A prima active-set method for convex QP. Find x, y, z) satisfying conditions 22) for some second-order consistent basis B; whie : z + r < do if N then N N \{}; prima_baseb, N,, x, y, z); [returns B, N, x, y, z] ese B B\{}; end if whie z + r < do prima_intermediateb, N,, x, y, z); [returns B, N, x, y, z] end whie B B {}; end whie This section concudes with a convergence resut for the prima method of Agorithm 3. In particuar, it is shown that the agorithm is we-defined, and terminates in a finite number of iterations if PQP q,r )isnondegenerate. We define nondegeneracy to mean that a nonzero step in the x-variabes is taken at each iteration of Agorithm 3 that invoves a base subiteration. A sufficient condition on PQP q,r ) for this to hod is that the gradients of the equaity constraints and active bound constraints are ineary independent at each iterate. See, e.g., Fetcher [26] for further discussion of these issues. As the active-set strategy uses the same criteria for adding and deeting variabes as those used in the simpex method, standard pivot seection rues used to

19 Prima and dua active-set methods for convex quadratic avoid cycing in inear programming, such as exicographica ordering, east-index seection or perturbation may be appied directy to the method proposed here see, e.g., [7,14,17,49]). Theorem 1 Given a prima-feasibe point x, y, z ) satisfying conditions 22) for a second-order consistent basis B, then Agorithm 3 generates a sequence of secondorder consistent bases {B j } j>. Moreover, if probem PQP q,r ) is nondegenerate, then Agorithm 3 finds a soution of PQP q,r ) or determines that DQP q,r ) is infeasibe in a finite number of iterations. Proof Assume that x, y, z) satisfies the conditions 22) for the second-order consistent basis B. Propositions 3 and 4 impy that the KKT matrices associated with subsequent base and intermediate iterations are nonsinguar, in which case each basis is second-order consistent. Let B < denote the index set B < ={i B : z i + r i < }, and et r be the vector r i = r i, i / B <, and r i = z i, i B <. These definitions impy that r i = z i > z i + z i + r i = r i, for every i B <.Itfoowsthat r r, and the feasibe region of DQP q,r ) is a subset of the feasibe region of DQP q, r ). In addition, if r is repaced by r in 19), the ony difference is that z B + r B =, i.e., the initia point for 22) is a stationary point with respect to PQP q, r ). The first step of the proof is to show that after a finite number of iterations of Agorithm 3, one of three possibe events must occur: i) the cardinaity of the set B < is decreased by at east one; ii) a soution of probem PQP q,r ) is found; or iii) DQP q,r ) is decared infeasibe. The proof wi aso estabish that if i) does not occur, then either ii) or iii) must hod after a finite number of iterations. Assume that i) never occurs. This impies that the index seected in the base subiteration can never be an index in B < because at the end of such an iteration, it woud beong to B with z + r =, contradicting the assumption that the cardinaity of B < never decreases. For the same reason, it must hod that k / B < for every index k seected to be moved from B to N in any subiteration, because an index can ony be moved from N to B by being seected in the base subiteration. These arguments impy that z i = r i, with i B <, throughout the iterations. It foows that the iterates may be interpreted as being members of a sequence constructed for soving PQP q, r ) with a fixed r, where the initia stationary point is given, and each iteration gives a new stationary point. The nondegeneracy assumption impies that α x = for at east one subiteration. For the base subiteration, x >, and it foows from Proposition 4 that x = if and ony if x > for an intermediate subiteration. Therefore, Proposition 11 shows that the objective vaue of PQP q, r )is stricty decreasing for a subiteration where α x =. In addition, the objective vaue of PQP q, r ) is nonincreasing at each subiteration, so a strict overa improvement of the objective vaue of PQP q, r ) is obtained at each iteration. As there are ony a finite number of stationary points, Agorithm 3 either soves PQP q, r ) or concudes that DQP q, r ) is infeasibe after a finite number of iterations. If PQP q, r ) is soved, then z N + r N, because r j = r j for j N. Hence, Agorithm 3 can not proceed further by seecting an N, and the ony way to reduce the objective is to seect an in B such that z j + r j <. Under the assumption that i) does not occur, it must hod that no eigibe indices exist and B < =. However, in this case PQP q,r ) has been soved with r = r, and ii) must hod. If Agorithm 3 decares DQP q, r ) to be infeasibe, then

20 488 A. Forsgren et a. DQP q,r ) must aso be infeasibe because the feasibe region of DQP q,r ) is contained in the feasibe region of DQP q, r ). In this case DQP q,r ) is infeasibe and iii) occurs. Finay, if i) occurs, there is an iteration at which the cardinaity of B < decreases and an index is removed from B <. There may be more than one such index, but there is at east one moved from B < to B\B <, or one k moved from B < to N. In either case, the cardinaity of B < is decreased by at east one. After such an iteration, the argument given above may be repeated for the new set B < and new shift r. Appying this argument repeatedy gives the resut that the situation i) can occur ony a finite number of times. It foows that ii) or iii) must occur after a finite number of iterations, which is the required resut. 5.5 Reaxed initia conditions for the dua QP method Anaogous to the prima case, the choice of q = q 2) = in Step 2) of the prima-first strategy may give some negative components in the vector x N + q N. In this case, the conditions 2) on the initia vaues of B, N, q, r, x, y, and z are reaxed so that Hx + c A T y z =, x N + q N, Ax + My b =, z B + r B =, z N + r N. 23) Simiary, the set of eigibe indices may be extended to incude indices associated with negative vaues of x N + q N. If the index is from N, the associated matrix K B is nonsinguar, and intermediate subiterations are executed unti the updated vaue satisfies x + q =. At this point, the index is returned N. The method is summarized in Agorithm 4. Agorithm 4 A dua active-set method for convex QP. Find x, y, z) satisfying conditions 23) for some second-order consistent B; whie : x + q < do if B then B B\{}; dua_baseb, N,, x, y, z); ese N N \{}; end if whie x + q < do dua_intermediateb, N,, x, y, z); end whie N N {}; end whie [Base subiteration] [Intermediate subiteration] A convergence resut anaogous to Theorem 1 hods for the dua agorithm. In this case, the nondegeneracy assumption concerns the inear independence of the gradients of the equaity constraints and active bounds for DQP q,r ).

21 Prima and dua active-set methods for convex quadratic Theorem 2 Given a dua-feasibe point x, y, z ) satisfying conditions 23) for a second-order consistent basis B, then Agorithm 4 generates a sequence of secondorder consistent bases {B j } j>. Moreover, if probem DQP q,r ) is nondegenerate, then Agorithm 4 either soves DQP q,r ) or concudes that PQP q,r ) is infeasibe in a finite number of iterations. Proof The proof mirrors that of Theorem 1 for the prima method. 6 Practica issues As stated, the prima quadratic program has ower bound zero on the x-variabes. This is for notationa convenience. This form may be generaized in a straightforward manner to a form where the x-variabes has both ower and upper bounds on the prima variabes, i.e., b L x b U, where components of b L can be and components of b U can be +. Given prima shifts q L and q U, and dua shifts r L and r U,wehavethe prima dua pair and PQP q,r ) minimize 1 x,y 2 x T Hx yt My + c T x + r L r U ) T x subject to Ax + My = b, b L q L x b U + q U, DQP q,r ) maximize 1 x,y,z L,z U 2 x T Hx 2 1 yt My + b T y + b L q L ) T z L b U + q U ) T z U subject to Hx + A T y + z L z U = c, z L r L, z U r U. An infinite bound has neither a shift nor a corresponding dua variabe. For exampe, if the jth components of b L and b U are infinite, then the corresponding variabe x j is free. In the procedure given in Sect. 5.1 for finding the first second-order consistent basis B, it is assumed that variabes with indices not seected for B are initiaized at one of their bounds. As a free variabe has no finite bounds, any index j associated with a free variabe shoud be seected for B. However, this cannot be guaranteed in practice, and it is shown beow that the prima and dua QP methods may be extended to aow a free variabe to be fixed temporariy at some vaue. If the QP is defined in the genera probem format of Sect. 6, then any free variabe not seected for B has no upper or ower bound and must be temporariy fixed at some vaue x j = x j say). The treatment of such temporary bounds invoves some additiona modifications to the prima and dua methods of Sects. 5.4 and 5.5. Each temporary bound x j = x j defines an associated dua variabe z j with initia vaue z j. As the bound is temporary, it is treated as an equaity constraint, and the desired vaue of z j is zero. Initiay, an index j corresponding to a temporary bound is assigned a prima shift q j = and a dua shift r j = z j, making x j and z j feasibe for the shifted probem. In both the prima-first and dua-first approaches, the idea is to drive the z j -variabes associated with temporary bounds to zero in the prima and eave them unchanged in the dua.