5.5 Quadratic programming

Transcription

1 5.5 Quadratic programming Minimize a quadratic function subject to linear constraints: 1 min x t Qx + c t x 2 s.t. a t i x b i i I (P a t i x = b i i E x R n, where Q is an n n matrix, I and E are the sets of indices of the inequality and equality constraints. Without loss of generality: Q is symmetric (same value of the objective function with Q not symmetric and Q = 1 2 (Q + Qt. Difficulty depends on the nature of Q: if Q is positive semidefinite, (P is convex and relatively easy to solve, otherwise the problem can have a large number of local optima. Example: min{ x t x : 1 x i 1, i = 1,..., n} where all the 2 n vertices with 1, 1 coordinates are local minima. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

2 Illustrations of convex QPs: Quadratic Programs (QPs are the simplest NLP problems besides Linear Programs. Efficient QP algorithms are available. Many direct applications (for portfolio optimization see exercise 9.1 Example: Tranning linear Support Vector Machines (SVMs QPs play also an important role in efficient methods for more general NLP problems. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

3 QP with only equality constraints min{ 1 2 x t Qx + c t x : Ax = b } (1 where A is an m n matrix, with m n, of full rank m (there are no redundant constraints. Since only linear equations, constraint qualification assumption is satisfied at all feasible points and the KKT conditions simplify: m Qx + c + u i a i = 0 i=1 Ax = b. N.B.: The complementary slackness constraints are automatically satisfied. More or less direct solution of the linear system: ( ( Q A t x = A 0 u ( c b. If A is of full rank and Q is positive definite on the subspace {x R n matrix (on the left is non singular. : Ax = 0}, the Edoardo Amaldi (PoliMI Optimization Academic Year / 13

4 Null-space method Determine a matrix Z R n (n m whose columns span the null space {x R n : Ax = 0} of A. Z can be computed by (submatrix factorization of A (if A sparse by LU factorization. Given a feasible x 0, any other feasible solution can be expressed as for an appropriate vector w R n m. x = x 0 + Zw With simple algebraic manipulations we verify that problem (1 is equivalent to the unconstrained QP: min{ 1 2 w t (Z t QZw + (Qx 0 + c t Zw : w R n m }. If the reduced Hessian Z t QZ is positive definite, the unique optimal solution w can be obtained by solving the linear system: (Z t QZw = Z t (Qx 0 + c. There are other methods to solve (1 but null-space methods are widely used. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

5 QP with equality and inequality constraints Active-set methods 1 min x t Qx + c t x 2 s.t. a t i x b i i I (P a t i x = b i i E x R n where Q is an n n matrix, I and E are the set of indices of the inequality and equality constraints. Idea: Determine the subset I (x of indices that are active at an optimal solution x, by solving a sequence of QP problems with only equality constraints. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

6 Active-set method for convex QPs Initialization: Find and initial feasible solution x 0 and choose W 0 I (x 0 = {i I : a t i x 0 = b i} E subset of indices of the active constraints at x 0. Iteration k: Given the current feasible solution x k, determine a direction d k by solving the subproblem: min{ q(x k + d : a t i (x k + d = b i, i W k }, where W k is the current working set, with W k I (x k = {i I : a t i x k = b i} E. The subproblem is equivalent to: min{ q(x k + d : a t i d = 0, i W k }. (2 N.B.: If the reduced Hessian Z t QZ is positive definite (always true if Q is p.d., the subproblem (2 has a unique solution d k. Based on the type of solution d k of (2, we compute α k, we set x k+1 = x k + α k d k and we determine W k+1. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

7 If d k 0, we determine the longest possible step length that satisfy all the constraints not in W k : and we set x k+1 = x k + α k d k. α k = min{1, min i W k, a t i d k >0 b i a t i x k a t i d k }. (3 W k+1 is obtained by adding to W k the index of one of the constraints that become active at x k+1. If d k = 0, then x k is a minimum of the objective function over the subspace defined by W k and we set x k+1 = x k. We determine the multipliers u k i from the first order optimality conditions (2: Qx k + c + i W k u k i a i = 0. (4 If u k i 0 for each i W k I, then x k is a local optimum of the original QP problem. If ui k < 0 for at least one of the indices i W k I, we obtain W k+1 by deleting from W k the index i with the most negative ui k. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

8 Proposition: If Q is positive definite (o.f. is strictly convex, the metod (with anti-cycling rule finds an optimal solution of the QP within a finite number of iterations. N.B.: The number of working sets is finite. Example: min q(x 1, x 2 = (x (x s.t. x 1 + 2x x 1 + 2x x 1 2x x 1 0 x 2 0 where the constraints are numbered in the order, from 1 to 5. Figure: From J. Nocedal, S. Wright, Numerical optimization, First Edition, Springer 1999, p Edoardo Amaldi (PoliMI Optimization Academic Year / 13

9 Application of the active-set method to the example Iteration 0: Start from the initial solution x 0 = take W 0 = {3, 5}. ( 2 0. Constraints 3 and 5 are active at x 0 and we Since x 0 is a vertex (extreme point of the polyhedron of the feasible solutions, x 0 minimizes q(x w.r.t. W 0 (over {x R n : a t i x = b i, i W 0} and the optimal solution of the subproblem min{ q(x 0 + d : a t i d = 0, i W0 } is d 0 = 0. Thus x 1 = x 0 + α 0d 0 = x 0. By solving the KKT conditions (4 ( 2 q(x 0 = 5 = u 3 ( u 5 ( 0 1 we derive the values of the multipliers u 3 and u 5 associated to the active constraints, namely (u 3, u 5 = ( 2, 1. Since u 3 < u 5 < 0 we delete from the working set W 0 the third constraint, setting W 1 = {5}. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

10 Iteration 1: The optimal solution of the subproblem min{ q(x 1 + d : a t i d = 0, i W1 ( } is 1 d 1 =. 0 Since d 1 does not violate any constraint with ( indices not in W 1, the formula (3 provides 1 a step length α 1 = 1 and x 2 = x 1 + α 1d 1 =. 0 Since at x 2 no other constraints are active, we set W 2 = W 1 = {5}. Iteration 2: The optimal solution for the subproblem min{ q(x 2 + d : a t i d = 0, i W2 } is d 2 = 0. By solving the KKT conditions (4, that is ( ( 0 0 q(x 2 = = u we obtain u 5 = 5., Thus x 3 = x 2 and we set W 3 = W 2 \ {5} =. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

11 Iteration 3: The optimal solution of the unconstrained subproblem min{ q(x 3 + d : a t i d = 0, i W3 } is d 3 = ( Since d 3 violates constraints with indices 1 and 2 which ( are not in W 1, the formula (3 1 provides a step length α 3 = 0.6 and x 4 = x 3 + α 3d 3 =. 1.5 Since at x 4 only the constraint with index 1 becomes active, we set W 4 = {1}.. Iteration 4: The optimal solution of the subproblem min{ q(x 4 + d : a t i d = 0, i W4 ( } is 0.4 d 4 =. 0.2 ( 1.4 Since x 4 + d 4 = satisfies all the constraints with indices not in W 1.7 1, we take α 4 = 1, set x 5 = x 4 + d 4 and W 5 = W 4 = {1}. Edoardo Amaldi (PoliMI Optimization Academic Year / 13

12 Iteration 5: The optimal solution of the subproblem min{ q(x 5 + d : a t i d = 0, i W5 } is d 5 = 0. Since ( solving the KKT conditions (4 we obtain u 1 = , the feasible solution 1.4 x 5 = is optimal for the original problem. 1.7 Edoardo Amaldi (PoliMI Optimization Academic Year / 13

13 Non convex QP and solvers If the Hessian matrix Q has some negative eigenvalues, the active-set method for convex QP can be adapted by modifying the computation of d k and α k in certain situations. See J. Nocedal, S. Wright, Numerical optimization, First Edition, Springer 1999, p Since W k may change by just one index at every iteration, efficient QP solvers proceed by successive updates of the factors computed at the previous iteration. Several active-set-based solvers are available: LINDO, LSSOL, QPOPT, NAG Library, Matlab,... Edoardo Amaldi (PoliMI Optimization Academic Year / 13