MATH 6026 Dr. Pedro Vásquez UPRM P. Vásquez (UPRM) Conferencia 1 / 17
Quadratic programming uemath 6026 Equality constraints A general formulation of these problems is: min x 2R nq (x) = 1 2 x T Qx + x T c (1) subjec to Ax = b where A is the m n Jacobian of constraints (with m n) whose rows ai T, i 2 ε and b is the vector in R m. We assume that A has full row rank (rank m).so that the constraints are consistent. P. Vásquez (UPRM) Conferencia 2 / 17
First order necessary conditions The FONC for x to be a solution of (1) state that there is a vector λ such that the following system of equations is satis ed: Q A T x A 0 λ = c b The system can be rewritten in a form that is useful for computation by expressing x as x = x + p, where x is some estimate of the solution and p is the desired step, so the above system can be written: Q A T p A 0 λ = g h where h = Ax b, g = c + Qx, p = x x (2) (3) P. Vásquez (UPRM) Conferencia 3 / 17
The matrix in (2) is called the Karush-Kuhn-Tucker (KKT) matrix, and the following result gives conditions under which is non singular. We use Z as the n (n m) matrix whose columns are the basic null space of A, that is Z has full rank and satis es AZ = 0. Lemma Let A have full row rank, and assume that the reduced-hessian matrix Z T QZ is postive de nite. Then the KKT matrix Q A T K = (4) A 0 is nonsingular and hence there is a unique vector pair (x, λ ) satisfying (2). P. Vásquez (UPRM) Conferencia 4 / 17
Example P. Vásquez (UPRM) Conferencia 5 / 17
P. Vásquez (UPRM) Conferencia 6 / 17
Theorem Let A have full row rank and assume that the reduced-hessian matrix Z T QZ is positive de nite. Then the vector x satisfying (2) is the unique global solution of (1). P. Vásquez (UPRM) Conferencia 7 / 17
Direct solution of the KKT system One important observation is that if m 1, the KKT matrix is always inde nite. We de ne the inertia of a symmetric matrix K to be the triple that indicates the numbers n +, n, and n 0 of positive, negative and zero eigenvalues, respectively, that is, inertia(k ) = (n +, n, n 0 ) Theorem Let K be de ned by (4), and suppose that A has rank n. inertia(k ) = inertia(z T QZ ) + (n, m, 0). Then Therefore, if Z T QZ is positive de nite, inertia(k ) = (n, m, 0) P. Vásquez (UPRM) Conferencia 8 / 17
Factoring the full KKT system One option to solve (3) is to perform a triangular factorization on the full KKT matrix and then perform backward and forward substitution with the triangular factors. Because of inde niteness, we cannot use the Cholesky factorization. We would use Gaussian elimination with partial pivoting to obtain the L and U factors, but this approach has the disadvantage that it ignores symmetry.. The most e ective strategy in this case is to use a symmetricinde nitefactorization. For a general symmetric matrix K, the factorization has the form: P T KP = LBL T where P is a permutation matrix, L is unit lower triangular, and B is blocked-diagonal with either 1 1 or 2 2 blocks. P. Vásquez (UPRM) Conferencia 9 / 17
P. Vásquez (UPRM) Conferencia 10 / 17
Null- space method The null space method does not require nonsingularity of Q and therefore has wider applicability. It assumes only that the conditions of lemma, that A has full row rank and that Z T QZ is positive de nite. However requires knowledge of the null-space basis matrix Z Suppose that the vector p has a partition into two components: p = Yp Y + Zp Z where Z is the n (n m) null-space matrix, Y is any n n matrix such that [Y jz ] is nonsingular p Y is an m-vector p Z is an (n m)-vector Yx Y is a particular solution of Ax = b Zx Z is a displacement along the constraints P. Vásquez (UPRM) Conferencia 11 / 17
Then we obtain: (AY ) p Y = h Since A has rank m and [Y jz ] is n n nonsingular, the product A [Y jz ] = [AY j0] has rank m. Therefore, AY is a nonsingular m m matrix, and p Y is well determined, substituting : QYp Y QZp Z + A T λ = g and multiply by Z T : Z T QZ p Z = Z T QYp Y Z T g The system can be solved by performing a Cholesky factorization of the reduced Hessian matrix Z T QZ to determine p Z. We therefore compute the total step p = Yp Y + Zp Z. To obtain the lagrange multiplier, we use: which can be solved for λ. (AY ) T λ = Y T (g + Qp) P. Vásquez (UPRM) Conferencia 12 / 17
Example P. Vásquez (UPRM) Conferencia 13 / 17
P. Vásquez (UPRM) Conferencia 14 / 17
Iterative solution for the KKT system CG applied to the reduced system Assuminf that the solution of the QP is: x = Yx Y + Zx Z for some vectors x Z 2 R n m, x Y 2 R m, the constraints Ax = b yield: AYx Y = b whixh determines the vector x. To obtain x Z solves the unsconstrained reduced problem: where 1 min x 2 x Z T Z T QZx Z + xz T c Z, Z c Z = Z T QYx Y + Z T c The solution x Z satis es the linear system Z T QZx Z = c Z Since Z T QZ is positive de nite, we can apply the CG method: P. Vásquez (UPRM) Conferencia 15 / 17
Algorithm 16.1 (preconditioned CG for reduced systems Choose an initial point x Z ; Compute r Z = Z T QZx Z + c Z, g Z = WZZ 1 r Z, and d Z = repeat α r Z g Z /dz T Z T QZd Z ; x Z x Z + αd Z ; r + Z r Z + αz T QZd Z ; g + Z W ZZ 1 r + Z ; β r + T Z g + Z /rz T g Z ; d Z g + Z + βd Z ; g Z g + Z ; r Z r + Z until a termination test is satis ed. g Z P. Vásquez (UPRM) Conferencia 16 / 17
P. Vásquez (UPRM) Conferencia 17 / 17