Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming

Size: px

Start display at page:

Download "Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming"

Blaise Harper
6 years ago
Views:

1 School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio J.Gondzio@ed.ac.uk URL: Paris, January

2 Outline Part 1: IPM for QP quadratic forms duality in QP first order optimality conditions primal-dual framework Part 2: IPM for NLP NLP notation Lagrangian first order optimality conditions primal-dual framework Algebraic Modelling Languages Self-concordant barrier Paris, January

3 IPM for Convex QP Paris, January

4 Convex Quadratic Programs The quadratic function f(x) = x T Qx is convex if and only if the matrix Q is positive definite. In such case the quadratic programming problem is well defined. min c T x+ 1 2 xt Qx s.t. Ax = b, x 0, If there exists a feasible solution to it, then there exists an optimal solution. Paris, January

5 QP Background: Def. A matrix Q R n n is positive semidefinite if x T Qx 0 for any x 0. We write Q 0. Def. A matrix Q R n n is positive definite if x T Qx > 0 for any x 0. We write Q 0. Example: Consider quadratic functions f(x) = x T Qx with the following matrices: [ ] [ ] [ ] Q 1 = 0 2, Q 2 = 0 1, Q 3 = 4 3. Q 1 is positive definite (hence f 1 is convex). Q 2 and Q 3 are indefinite (f 2,f 3 are not convex). Paris, January

6 QP with IPMs Consider the convex quadratic programming problem. The primal min c T x+ 1 2 xt Qx s.t. Ax = b, x 0, and the dual max b T y 1 2 xt Qx s.t. A T y +s Qx = c, x,s 0. Apply the usual procedure: replace inequalities with log barriers; form the Lagrangian; write the first order optimality conditions; apply Newton method to them. Paris, January

7 QP with IPMs: Log Barriers Replace the primal QP min c T x+ 1 2 xt Qx s.t. Ax = b, x 0, with the primal barrier QP min c T x+ 1 2 xt Qx n s.t. Ax = b. j=1 lnx j Paris, January

8 QP with IPMs: Log Barriers Replace the dual QP with the dual barrier QP max b T y 2 1xT Qx s.t. A T y + s Qx = c, y free, s 0, max b T y 1 2 xt Qx+ n j=1 s.t. A T y +s Qx = c. lns j Paris, January

9 First Order Optimality Conditions Consider the primal barrier quadratic program min c T x+ 1 2 xt Qx µ n s.t. Ax = b, where µ 0 is a barrier parameter. Write out the Lagrangian j=1 lnx j L(x,y,µ) = c T x+ 1 2 xt Qx y T (Ax b) µ n j=1 lnx j, Paris, January

10 First Order Optimality Conditions (cont d) The conditions for a stationary point of the Lagrangian: L(x,y,µ) = c T x+ 1 n 2 xt Qx y T (Ax b) µ lnx j, are j=1 x L(x,y,µ) = c A T y µx 1 e+qx = 0 y L(x,y,µ) = Ax b = 0, where X 1 = diag{x 1 1,x 1 2,,x 1 n }. Let us denote s = µx 1 e, i.e. XSe = µe. The First Order Optimality Conditions are: Ax = b, A T y +s Qx = c, XSe = µe. Paris, January

11 Apply Newton Method to the FOC The first order optimality conditions for the barrier problem form a large system of nonlinear equations F(x,y,s) = 0, where F : R 2n+m R 2n+m is an application defined as follows: Ax b F(x,y,s) = A T y +s Qx c. XSe µe Actually, the first two terms of it are linear; only the last one, corresponding to the complementarity condition, is nonlinear. Note that F(x,y,s) = A 0 0 Q A T I. S 0 X Paris, January

12 Newton Method for the FOC (cont d) Thus, for a given point (x,y,s) we find the Newton direction ( x, y, s) by solving the system of linear equations: A 0 0 Q A T I S 0 X [ x y s ] = b Ax c A T y s+qx µe XSe. Paris, January

13 Interior-Point QP Algorithm Initialize k = 0, (x 0,y 0,s 0 ) F 0, µ 0 = 1 n (x0 ) T s 0, α 0 = Repeat until optimality k = k +1 µ k = σµ k 1, where σ (0,1) = Newton direction towards µ-center Ratio test: α P := max {α > 0 : x+α x 0}, α D := max {α > 0 : s+α s 0}. Make step: x k+1 = x k +α 0 α P x, y k+1 = y k +α 0 α D y, s k+1 = s k +α 0 α D s. Paris, January

14 From LP to QP QP problem min c T x+ 1 2 xt Qx s.t. Ax = b, x 0. First order conditions (for barrier problem) Ax = b, A T y +s Qx = c, XSe = µe. Paris, January

15 IPMs for Convex NLP Paris, January

16 Convex Nonlinear Optimization Consider the nonlinear optimization problem min f(x) s.t. g(x) 0, where x R n, and f : R n R and g : R n R m are convex, twice differentiable. Assumptions: f and g are convex If there exists a local minimum then it is a global one. f and g are twice differentiable We can use the second order Taylor approximations. Some additional (technical) conditions We need them to prove that the point which satisfies the first order optimality conditions is the optimum. We won t use them in this course. Paris, January

17 Taylor Expansion of f : R R Let f : R R. If all derivatives of f are continuously differentiable at x 0, then f (k) (x f(x) = 0 ) (x x k! 0 ) k, k=0 where f (k) (x 0 ) is the k-th derivative of f at x 0. The first order approximation of the function: f(x) = f(x 0 )+f (x 0 )(x x 0 )+r 2 (x x 0 ), where the remainder satisfies: r lim 2 (x x 0 ) = 0. x x 0 x x 0 Paris, January

18 Taylor Expansion (cont d) The second order approximation: f(x) = f(x 0 )+f (x 0 )(x x 0 ) where the remainder satisfies: f (x 0 )(x x 0 ) 2 +r 3 (x x 0 ), r lim 3 (x x 0 ) x x 0 (x x 0 ) 2 = 0. Paris, January

19 Derivatives of f : R n R The vector ( f(x)) T = is called the gradient of f at x. The matrix 2 f x 2 (x) 1 2 f(x)= ( f (x), f (x),..., f ) (x) x 1 x 2 x n 2 f x 2 x 1 (x) 2 f x 1 x 2 (x)... 2 f x 1 x n (x) 2 f x 2 (x) f x n x 1 (x) is called the Hessian of f at x. 2 f x n x 2 (x)... 2 f x 2 x n (x) 2 f x 2 (x) n Paris, January

20 Taylor Expansion of f : R n R Let f : R n R. If all derivatives of f are continuously differentiable at x 0, then f (k) (x f(x) = 0 ) (x x k! 0 ) k, k=0 where f (k) (x 0 ) is the k-th derivative of f at x 0. The first order approximation of the function: f(x) = f(x 0 )+ f(x 0 ) T (x x 0 )+r 2 (x x 0 ), where the remainder satisfies: r lim 2 (x x 0 ) x x 0 x x 0 = 0. Paris, January

21 Taylor Expansion (cont d) The second order approximation: of the function: f(x) = f(x 0 )+ f(x 0 ) T (x x 0 ) (x x 0) T 2 f(x 0 )(x x 0 )+r 3 (x x 0 ), where the remainder satisfies: r lim 3 (x x 0 ) x x 0 x x 0 2 = 0. Paris, January

22 Convexity: Reminder Property 1. For any collection {C i i I} of convex sets, the intersection i I C i is convex. Property 4. If C is a convexset and f : C R is convexfunction, the level sets {x C f(x) α} and {x C f(x) < α} are convex for all scalars α. Lemma 1: If g : R n R m is a convex function, then the set {x R n g(x) 0} is convex. Proof: Since every function g i : R n R,i = 1,2,...,m is convex, from Property 4, we conclude that every set X i = {x R n g i (x) 0} is convex. Next from Property 1, we deduce that the intersection m X = X i = {x R n g(x) 0} i=1 is convex, which completes the proof. Paris, January

23 Differentiable Convex Functions Property 8. Let C R n be a convex set and f : C R be twice continuously differentiable over C. (a)if 2 f(x)ispositivesemidefiniteforallx C,thenf isconvex. (b) If 2 f(x) is positive definite for all x C, then f is strictly convex. (c)iff isconvex,then 2 f(x)ispositivesemidefiniteforallx C. Let the second order approximation of the function be given: f(x) f(x 0 )+c T (x x 0 )+ 1 2 (x x 0) T Q(x x 0 ), where c = f(x 0 ) and Q = 2 f(x 0 ). From Property 8, it follows that when f is convex and twice differentiable, then Q exists and is a positive semidefinite matrix. Conclusion: Iff isconvexandtwicedifferentiable,thenoptimizationoff(x)can (locally) be replaced with the minimization of its quadratic model. Paris, January

24 Nonlinear Optimization with IPMs Nonlinear Optimization via QPs: Sequential Quadratic Programming (SQP). Repeat until optimality: approximate NLP (locally) with a QP; solve (approximately) the QP. Nonlinear Optimization with IPMs: works similarly to SQP scheme. However, the (local) QP approximations are not solved to optimality. Instead, only one step in the Newton direction corresponding to a given QP approximation is made and the new QP approximation is computed. Paris, January

25 Nonlinear Optimization with IPMs Derive an IPM for NLP: replace inequalities with log barriers; form the Lagrangian; write the first order optimality conditions; apply Newton method to them. Paris, January

26 NLP Notation Consider the nonlinear optimization problem min f(x) s.t. g(x) 0, where x R n, and f : R n R and g : R n R m are convex, twice differentiable. The vector-valued function g : R n R m has a derivative [ ] gi A(x) = g(x) = R m n x j which is called the Jacobian of g. i=1..m,j=1..n Paris, January

27 NLP Notation (cont d) The Lagrangian associated with the NLP is: L(x,y) = f(x)+y T g(x), where y R m,y 0 are Lagrange multipliers (dual variables). The first derivatives of the Lagrangian: x L(x,y) = f(x)+ g(x) T y y L(x,y) = g(x). The Hessian of the Lagrangian, Q(x,y) R n n : m Q(x,y) = 2 xxl(x,y) = 2 f(x)+ y i 2 g i (x). i=1 Paris, January

28 Convexity in NLP Lemma 2: If f : R n R and g : R n R m are convex, twice differentiable, then the Hessian of the Lagrangian m Q(x,y) = 2 f(x)+ y i 2 g i (x) is positive semidefinite for any x and any y 0. If f is strictly convex, then Q(x,y) is positive definite for any x and any y 0. Proof: UsingProperty8,theconvexityoff impliesthat 2 f(x)is positive semidefinite for any x. Similarly, the convexity of g implies that for all i = 1,2,...,m, 2 g i (x) is positive semidefinite for any x. Since y i 0 for all i = 1,2,...,m and Q(x,y) is the sum of positive semidefinite matrices, we conclude that Q(x, y) is positive semidefinite. If f is strictly convex, then 2 f(x) is positive definite and so is Q(x,y). Paris, January i=1

29 IPM for NLP Add slack variables to nonlinear inequalities: min f(x) s.t. g(x)+z = 0 z 0, where z R m. Replace inequality z 0 with the logarithmic barrier: min f(x) µ m lnz i i=1 s.t. g(x)+z = 0. Write out the Lagrangian L(x,y,z,µ) = f(x)+y T (g(x)+z) µ m i=1 lnz i, Paris, January

30 IPM for NLP For the Lagrangian L(x,y,z,µ) = f(x)+y T (g(x)+z) µ m lnz i, i=1 write the conditions for a stationary point x L(x,y,z,µ) = f(x)+ g(x) T y = 0 y L(x,y,z,µ) = g(x)+z = 0 z L(x,y,z,µ) = y µz 1 e = 0, where Z 1 = diag{z 1 1,z 1 2,,z 1 m }. The First Order Optimality Conditions are: f(x)+ g(x) T y = 0, g(x)+z = 0, YZe = µe. Paris, January

31 Newton Method for the FOC The first order optimality conditions for the barrier problem form a large system of nonlinear equations F(x,y,z) = 0, where F : R n+2m R n+2m is an application defined as follows: F(x,y,z) = f(x) + g(x)t y g(x) + z. YZe µe Note that all three terms of it are nonlinear. (In LP and QP the first two terms were linear.) Paris, January

32 Newton Method for the FOC Observe that F(x,y,z) = Q(x,y) A(x)T 0 A(x) 0 I 0 Z Y, where A(x) is the Jacobian of g and Q(x,y) is the Hessian of L. They are defined as follows: A(x) = g(x) Q(x,y) = 2 f(x)+ m i=1 R m n y i 2 g i (x) R n n Paris, January

33 Newton Method (cont d) Foragivenpoint(x,y,z)wefindtheNewtondirection( x, y, z) by solving the system of linear equations: Q(x,y) [ ] A(x)T 0 x A(x) 0 I y = f(x) A(x)T y g(x) z. 0 Z Y z µe YZe Using the third equation we eliminate z = µy 1 e Ze ZY 1 y, from the second equation and get [ ][ ] Q(x,y) A(x) T x A(x) ZY 1 y = [ f(x) A(x) T y g(x) µy 1 e ]. Paris, January

34 Interior-Point NLP Algorithm Initialize k = 0 (x 0,y 0,z 0 ) such that y 0 > 0 and z 0 > 0, µ 0 = 1 m (y0 ) T z 0 Repeat until optimality k = k +1 µ k = σµ k 1, where σ (0,1) Compute A(x) and Q(x,y) = Newton direction towards µ-center Ratio test: α 1 := max {α > 0 : y +α y 0}, α 2 := max {α > 0 : z +α z 0}. Choosethestep: (usetrustregionorlinesearch) α min{α 1,α 2 }. Make step: x k+1 = x k +α x, y k+1 = y k +α y, z k+1 = z k +α z. Paris, January

35 From QP to NLP Newton direction for QP Q AT I A 0 0 S 0 X [ x y s ] = [ ξd ξp ξ µ ]. Augmented system for QP [ Q SX 1 A T A 0 ][ x y ] = [ ξd X 1 ] ξ µ. ξ p Paris, January

36 From QP to NLP Newton direction for NLP Q(x,y) [ A(x)T 0 x A(x) 0 I y 0 Z Y z ] = f(x) A(x)T y g(x) z µe YZe. Augmented system for NLP [ ][ Q(x,y) A(x) T x A(x) ZY 1 y ] = [ f(x) A(x) T y g(x) µy 1 e ]. Conclusion: NLP is a natural extension of QP. Paris, January

37 Linear Algebra in IPM for NLP Newton direction for NLP Q(x,y) [ ] A(x)T 0 x A(x) 0 I y 0 Z Y z = f(x) A(x)T y g(x) z µe YZe The corresponding augmented system [ ][ ] [ Q(x,y) A(x) T x f(x) A(x) A(x) ZY 1 y = T ] y g(x) µy 1. e where A(x) R m n is the Jacobian of g and Q(x,y) R n n is the Hessian of L A(x) = g(x) Q(x,y) = 2 f(x)+ m y i 2 g i (x) Paris, January i=1.

38 Linear Algebra in IPM for NLP (cont d) Automatic differentiation is very useful... get Q(x, y) and A(x) from Algebraic Modeling Language. Model Solution AML Num. Anal. Package SOLVER Output Paris, January

39 Automatic Differentiation AD in the Internet: ADIFOR (FORTRAN code for AD): ADOL-C (C/C++ code for AD): AD Tools/adolc.anl/adolc.html AD page at Cornell: Paris, January

40 IPMs: Remarks Interior Point Methods provide the unified framework for convex optimization. IPMs provide polynomial algorithms for LP, QP and NLP. The linear algebra in LP, QP and NLP is very similar. Use IPMs to solve very large problems. Further Extensions: Nonconvex optimization. IPMs in the Internet: LP FAQ (Frequently Asked Questions): Interior Point Methods On-Line: NEOS (Network Enabled Optimization Services): Paris, January

41 Newton Method and Self-concordant Barriers Paris, January

42 Another View of Newton M. for Optimization Newton Method for Optimization Let f : R n R be a twice continuously differentiable function. Suppose we build a quadratic model f of f around a given point x k, i.e., we define x = x x k and write: f(x) = f(x k )+ f(x k ) T x+ 1 2 xt 2 f(x k ) x Now we optimize the model f instead of optimizing f. A minimum(or, more generally, a stationary point) of the quadratic model satisfies: f(x) = f(x k )+ 2 f(x k ) x = 0, i.e. x = x x k = ( 2 f(x k )) 1 f(x k ), which reduces to the usual equation: x k+1 = x k ( 2 f(x k )) 1 f(x k ). Paris, January

43 logx Barrier Function Consider the primal barrier linear program n min c T x µ lnx j s.t. Ax = b, j=1 where µ 0 is a barrier parameter. Write out the Lagrangian L(x,y,µ) = c T x y T (Ax b) µ and the conditions for a stationary point n lnx j, j=1 x L(x,y,µ) = c A T y µx 1 e = 0 y L(x,y,µ) = Ax b = 0, where X 1 = diag{x 1 1,x 1 2,,x 1 n }. Paris, January

44 log x Barrier Function (cont d) Let us denote s = µx 1 e, i.e. XSe = µe. The First Order Optimality Conditions are: Ax = b, A T y +s = c, XSe = µe. Paris, January

45 - logx bf: Newton Method The first order optimality conditions for the barrier problem form a large system of nonlinear equations F(x,y,s) = 0, where F : R 2n+m R 2n+m is an application defined as follows: Ax b F(x,y,s) = A T y +s c. XSe µe Actually, the first two terms of it are linear; only the last one, corresponding to the complementarity condition, is nonlinear. Note that F(x,y,s) = A A T I. S 0 X Paris, January

46 - log x bf: Newton Method (cont d) Thus, for a given point (x,y,s) we find the Newton direction ( x, y, s) by solving the system of linear equations: A 0 0 [ ] x 0 A T I y = b Ax c A T y s. S 0 X s µe XSe Paris, January

47 1/x α, α >0 Barrier Function Consider the primal barrier linear program n min c T 1 x µ s.t. Ax = b, j=1 where µ 0 is a barrier parameter and α > 0. Write out the Lagrangian x α j L(x,y,µ) = c T x y T (Ax b)+µ and the conditions for a stationary point n j=1 x L(x,y,µ) = c A T y µαx α 1 e = 0 y L(x,y,µ) = Ax b = 0, where X α 1 = diag{x α 1 1,x α 1 2,,x α 1 n }. Paris, January x α j,

48 1/x α, α >0 Barrier Function (cont d) Let us denote s = µαx α 1 e, i.e. X α+1 Se = µαe. The First Order Optimality Conditions are: Ax = b, A T y +s = c, X α+1 Se = µαe. Paris, January

49 1/x α, α>0 bf: Newton Method The first order optimality conditions for the barrier problem are F(x,y,s) = 0, where F : R 2n+m R 2n+m is an application defined as follows: Ax b F(x,y,s) = A T y +s c. X α+1 Se µαe As before, only the last term, corresponding to the complementarity condition, is nonlinear. Note that F(x,y,s) = A A T I (α+1)x α S 0 X α+1 Paris, January

50 1/x α, α>0 bf: Newton Method (cont d) Thus, for a given point (x,y,s) we find the Newton direction ( x, y, s) by solving the system of linear equations: A 0 0 [ ] x b Ax 0 A T I y = c A T y s (α+1)x α S 0 X α+1 s µαe X α+1 Se. Paris, January

51 e 1/x Barrier Function Consider the primal barrier linear program n min c T x µ e 1/x j s.t. Ax = b, j=1 where µ 0 is a barrier parameter. Write out the Lagrangian L(x,y,µ) = c T x y T (Ax b)+µ n e 1/x j, j=1 and the conditions for a stationary point x L(x,y,µ) = c A T y µx 2 exp(x 1 )e = 0 y L(x,y,µ) = Ax b = 0, where exp(x 1 ) = diag{e 1/x 1,e 1/x 2,,e 1/x n}. Paris, January

52 e 1/x Barrier Function Let us denote s = µx 2 exp(x 1 )e, i.e. X 2 exp( X 1 )Se = µe. The First Order Optimality Conditions are: Ax = b, A T y +s = c, X 2 exp( X 1 )Se = µe. Paris, January

53 e 1/x bf: Newton Method The first order optimality conditions are F(x,y,s) = 0, where F : R 2n+m R 2n+m is defined as follows: Ax b F(x,y,s) = A T y +s c. X 2 exp( X 1 )Se µe As before, only the last term, corresponding to the complementarity condition, is nonlinear. Note that A 0 0 F(x,y,s) = 0 A T I. (2X +I)exp( X 1 ) 0 X 2 exp( X 1 ) Paris, January

54 e 1/x bf: Newton Method (cont d) Newton direction( x, y, s) solves the following system of linear equations: A A T I (2X +I)exp( X 1 )S 0 X 2 exp( X 1 ) = [ x y s b Ax c A T y s µe X 2 exp( X 1 )Se ]. Paris, January

55 Why Log Barrier is the Best? The First Order Optimality Conditions: logx : XSe = µe, 1/x α : X α+1 Se = µαe, e 1/x : X 2 exp( X 1 )Se = µe. Log Barrier ensures the symmetry between the primal and the dual. 3rd row in the Newton Equation System: logx : F 3 = [S,0,X], 1/x α : F 3 = [(α+1)x α S,0,X α+1 ] e 1/x : F 3 = [(2X+I)exp( X 1 )S,0,X 2 exp( X 1 )] Log Barrier produces the weakest nonlinearity. Paris, January

56 Self-concordant Functions There is a nice property of the function that is responsible for a good behaviour of the Newton method. Def Let C R n be an open nonempty convex set. Let f : C R be a three times continuously differentiable convex function. A function f is called self-concordant if there exists a constant p > 0 such that 3 f(x)[h,h,h] 2p 1/2 ( 2 f(x)[h,h]) 3/2, x C, h : x+h C. (We then say that f is p-self-concordant). Note that a self-concordant function is always well approximated by the quadratic model because the error of such an approximation can be bounded by the 3/2 power of 2 f(x)[h,h]. Paris, January

57 Self-concordant Barriers Lemma The barrier function logx is self-concordant on R +. Proof Consider f(x) = logx. Compute f (x) = x 1, f (x) = x 2 and f (x) = 2x 3 and check that the self-concordance condition is satisfied for p = 1. Lemma The barrier function 1/x α, with α (0, ) is not self-concordant on R +. Lemma The barrier function e 1/x is not self-concordant on R +. Use self-concordant barriers in optimization Paris, January

Introduction to Nonlinear Stochastic Programming

School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS