Hot-Starting NLP Solvers

Transcription

1 Hot-Starting NLP Solvers Andreas Wächter Department of Industrial Engineering and Management Sciences Northwestern University 204 Mixed Integer Programming Workshop Ohio State University, Columbus, OH July 2, 204 Andreas Wächter (NU) Hot-starting NLP MIP 204 / 33

2 Motivation: Branch and Bound MILP Very efficient enumeration of LP nodes with Simplex Strong-branching and diving Change one bound at a time Dual Simplex can rapidly solve new instance Reason: an existing factorization of basis matrix can be used! Can speed up MILP branch-and-bound significantly Andreas Wächter (NU) Hot-starting NLP MIP / 33

3 Motivation: Branch and Bound MILP Very efficient enumeration of LP nodes with Simplex Strong-branching and diving MINLP Change one bound at a time Dual Simplex can rapidly solve new instance Reason: an existing factorization of basis matrix can be used! Can speed up MILP branch-and-bound significantly Generally, NLP solvers cannot use hot starts Derivative matrices change with each iterate Factorization of old matrices useless? Andreas Wächter (NU) Hot-starting NLP MIP / 33

4 Motivation: Branch and Bound MILP Very efficient enumeration of LP nodes with Simplex Strong-branching and diving MINLP Change one bound at a time Dual Simplex can rapidly solve new instance Reason: an existing factorization of basis matrix can be used! Can speed up MILP branch-and-bound significantly Generally, NLP solvers cannot use hot starts Derivative matrices change with each iterate Factorization of old matrices useless? Question: Can we hot-start NLP solvers, re-using existing factorizations? Andreas Wächter (NU) Hot-starting NLP MIP / 33

5 Sequential Quadratic Programming (SQP) min x R n f (x) s.t. c(x) = 0 x 0 At (x k, λ k ), step d k is computed from quadratic/linear model min d R n f (x k) T d + 2 dt W k d s.t. c k + c(x k ) T d = 0 λ k+ x k + d 0 QP(x k, λ k ) where W k = 2 f (x k ) + j λ(j) k 2 c(x k ). (Assume W k 0) x k+ = x k + d k, λ k+ from QP (usually with line search) Andreas Wächter (NU) Hot-starting NLP MIP / 33

6 Solving QPs in SQP min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 QP(x k, λ k ) d l k A k = c(x k ) T and W k = 2 L(x k, λ k ) depend on iterate (x k, λ k ) However, for closely related NLPs and subsequent SQP iterations, QP data, specifically Ak and W k, and active set do not change much. Andreas Wächter (NU) Hot-starting NLP MIP / 33

7 Solving QPs in SQP min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 QP(x k, λ k ) d l k A k = c(x k ) T and W k = 2 L(x k, λ k ) depend on iterate (x k, λ k ) However, for closely related NLPs and subsequent SQP iterations, QP data, specifically Ak and W k, and active set do not change much. Goal: Develop method for solving QP(x k, λ k ) (approximately) that efficiently uses information from previously solved QP(x, λ ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

8 Active Set QP Solver min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 Compute solution from linear system: (assume W k 0) [ Wk A T k A k 0 ]( d λ ) = ( gk c k ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

9 Active Set QP Solver min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k A {,..., n}: Guess of the active set, i.e., d A = l A k Compute solution from linear system: (assume W k 0) W k A T k (I A ) T A k 0 0 I A 0 0 d λ µ A = g k c k l A k Andreas Wächter (NU) Hot-starting NLP MIP / 33

10 Active Set QP Solver min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k A {,..., n}: Guess of the active set, i.e., d A = l A k Compute solution from linear system: (assume W k 0) W k A T k (I A ) T A k 0 0 I A 0 0 A is optimal, if d l k and µ A 0 d λ µ A = g k c k l A k Andreas Wächter (NU) Hot-starting NLP MIP / 33

11 Active Set QP Solver min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k A {,..., n}: Guess of the active set, i.e., d A = l A k Compute solution from linear system: (assume W k 0) W k A T k (I A ) T A k 0 0 I A 0 0 A is optimal, if d l k and µ A 0 d λ µ A = If A not optimal, add or remove variable from A Update factorization of KKT matrix Andreas Wächter (NU) Hot-starting NLP MIP / 33 g k c k l A k

12 Warm Start vs. Hot Start min d R n 2 dt W d + g T d s.t. A d + c = 0 d l min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k Andreas Wächter (NU) Hot-starting NLP MIP / 33

13 Warm Start vs. Hot Start Warm Start: min d R n 2 dt W d + g T d s.t. A d + c = 0 d l min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k Use A as starting guess, factorize with new matrix data Andreas Wächter (NU) Hot-starting NLP MIP / 33

14 Warm Start vs. Hot Start Warm Start: min d R n 2 dt W d + g T d s.t. A d + c = 0 d l min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k Use A as starting guess, factorize with new matrix data min d R n 2 dt W d + g T d s.t. A d + c = 0 d l min d R n 2 dt W d + gk T d s.t. A d + c k = 0 d l k Andreas Wächter (NU) Hot-starting NLP MIP / 33

15 Warm Start vs. Hot Start Warm Start: min d R n 2 dt W d + g T d s.t. A d + c = 0 d l min d R n 2 dt W k d + gk T d s.t. A k d + c k = 0 d l k Use A as starting guess, factorize with new matrix data Hot Start: min d R n 2 dt W d + g T d s.t. A d + c = 0 d l min d R n 2 dt W d + gk T d s.t. A d + c k = 0 d l k Use A as starting guess, reuse factorization with same matrix data Andreas Wächter (NU) Hot-starting NLP MIP / 33

16 Hot-Started NLP Solver in Strong Branching Root Node NLP x i x i x i x i NLP i NLP + i Andreas Wächter (NU) Hot-starting NLP MIP / 33

17 Hot-Started NLP Solver in Strong Branching. Solve root node NLP 2. SQP for each NLP ± i Root Node NLP x i x i x i x i NLP i NLP + i Andreas Wächter (NU) Hot-starting NLP MIP / 33

18 Hot-Started NLP Solver in Strong Branching. Solve root node NLP 2. SQP for each NLP ± i Root Node NLP x i x i x i x i - solve each QP ± i (x k, λ k ) NLP i NLP + i QP ± i (x 0, λ 0 ) QP ± i (x 0, λ 0 ) QP ± i (x, λ ) QP ± i (x, λ ) QP ± i (x k, λ k ) QP ± i (x k, λ k ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

19 Hot-Started NLP Solver in Strong Branching. Solve root node NLP, and store state of QP(x, λ ) 2. SQP for each NLP ± i - restore QP(x, λ ) state - solve each QP ± i (x k, λ k ) using several hot-starts for QP(x, λ ) NLP i QP ± i (x 0, λ 0 ) Root Node NLP Store QP(x, λ ) x i x i x i x i NLP + i QP ± i (x 0, λ 0 ) QP ± i (x, λ ) QP ± i (x, λ ) QP ± i (x k, λ k ) QP ± i (x k, λ k ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

20 Hot-Started NLP Solver in Strong Branching. Solve root node NLP, and store state of QP(x, λ ) 2. SQP for each NLP ± i - restore QP(x, λ ) state - solve each QP ± i (x k, λ k ) using several hot-starts for QP(x, λ ) Assumption: Several hot-starts for QP(x, λ ) are less work than a warm-start for QP ± i (x k, λ k ) NLP i QP ± i (x 0, λ 0 ) QP ± i (x, λ ) QP ± i (x k, λ k ) Root Node NLP Store QP(x, λ ) x i x i x i x i NLP + i QP ± i (x 0, λ 0 ) QP ± i (x, λ ) QP ± i (x k, λ k ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

21 Solve Sequence of Closely Related QPs Want algorithm for solving a sequence of closely related QPs min d R n 2 dt W d + g T d s.t. A d + c = 0 d l using hot-starts min d R n 2 dt W d + g T d s.t. Ad + c = 0 d l Andreas Wächter (NU) Hot-starting NLP MIP / 33

22 Solve Sequence of Closely Related QPs Want algorithm for solving a sequence of closely related QPs min d R n 2 dt W d + g T d s.t. A d + c = 0 d l using hot-starts min d R n 2 dt W d + g T d s.t. Ad + c = 0 d l Iterative, computes increasingly accurate solutions NLP algorithm needs to be able to handle inexact solutions E.g., Sl QP solver with inexact QP solves [Curtis, Johnson, Robinson, W 204] Andreas Wächter (NU) Hot-starting NLP MIP / 33

23 Solve Sequence of Closely Related QPs Want algorithm for solving a sequence of closely related QPs min d R n 2 dt W d + g T d s.t. A d + c = 0 d l using hot-starts min d R n 2 dt W d + g T d s.t. Ad + c = 0 d l Iterative, computes increasingly accurate solutions NLP algorithm needs to be able to handle inexact solutions E.g., Sl QP solver with inexact QP solves [Curtis, Johnson, Robinson, W 204] Other application: Real-time optimal control Solve a QP or NLP at each time step Data for process changes only slowly Andreas Wächter (NU) Hot-starting NLP MIP / 33

24 Equality Constrained QPs Recall: Solution of min d R n 2 dt W d + g T d s.t. Ad + c = 0 is computed from linear system: [ ]( ) ( W A T d g = A 0 λ c ) Assumptions: W positive definite on the null space of A A has full rank. Then KKT matrix is non-singular. Andreas Wächter (NU) Hot-starting NLP MIP / 33

25 Iterative Refinement For Linear Systems Want: Md = b Have factorization of W Cheap: M d = b Andreas Wächter (NU) Hot-starting NLP MIP / 33

26 Iterative Refinement For Linear Systems Want: Md = b Have factorization of W Cheap: M d = b Compute correction p Want: M(d + p) = b Equivalently Mp = b Md Cheap: M p = b Md Update: d d + p. Andreas Wächter (NU) Hot-starting NLP MIP / 33

27 Iterative Refinement For Linear Systems Want: Md = b Have factorization of W Cheap: M d = b Compute correction p Want: M(d + p) = b Equivalently Mp = b Md Cheap: M p = b Md Update: d d + p. Iterative Refinement Algorithm: Choose d 0 For i = 0,,... Solve: M p i = b Md i Update: di+ = d i + p i Fixed point iteration converges if σ(i M M) < Andreas Wächter (NU) Hot-starting NLP MIP / 33

28 Iterative Refinement Applied To Optimality Conditions Apply to [ W A T A 0 ]( d λ ) ( ) g = c In each iteration solve [ W A T A 0 ]( pi p λ i ) ( g = c ) ( W di + A T λ i Ad i ) Update (d i+, λ i+ ) = (d i, λ i ) + (p i, p λ i ) Andreas Wächter (NU) Hot-starting NLP MIP 204 / 33

29 Iterative Refinement Applied To Optimality Conditions Apply to [ W A T A 0 ]( d λ ) ( ) g = c In each iteration solve [ W A T A 0 ]( pi p λ i ) ( g = c ) ( W di + A T λ i Ad i ) Update (d i+, λ i+ ) = (d i, λ i ) + (p i, p λ i ) Equivalently, solve QP to get (p i, p λ i ) ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p λ i Andreas Wächter (NU) Hot-starting NLP MIP 204 / 33

30 Iterative Refinement for QPs Solve min d 2 dt W d + g T d s.t. Ad + c = 0 by computing a sequence of iterative refinement steps from ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p λ i Update (d i+, λ i+ ) = (d i, λ i ) + (p i, p λ i ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

31 Iterative Refinement for QPs Solve min d 2 dt W d + g T d s.t. Ad + c = 0 d l by computing a sequence of iterative refinement steps from ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p λ i Update (d i+, λ i+ ) = (d i, λ i ) + (p i, p λ i ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

32 Iterative Refinement for QPs Solve min d 2 dt W d + g T d s.t. Ad + c = 0 d l by computing a sequence of iterative refinement steps from ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p + d i l Update (d i+, λ i+ ) = (d i, λ i ) + (p i, p λ i ) p λ i Andreas Wächter (NU) Hot-starting NLP MIP / 33

33 Comments ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p + d i l p λ i Use hot-start for QP(W, A ) and products with W and A Andreas Wächter (NU) Hot-starting NLP MIP / 33

34 Comments ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p + d i l p λ i Use hot-start for QP(W, A ) and products with W and A If strict complementarity holds and (d i, λ i ) (d, λ ): QP eventually identifies correct active set Becomes iterative refinement for equality-only case Andreas Wächter (NU) Hot-starting NLP MIP / 33

35 Comments ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p + d i l p λ i Use hot-start for QP(W, A ) and products with W and A If strict complementarity holds and (d i, λ i ) (d, λ ): QP eventually identifies correct active set Becomes iterative refinement for equality-only case Do not need to track bound multipliers Key consequence: QP solver is responsible for handling active set Can use black-box QP solver Andreas Wächter (NU) Hot-starting NLP MIP / 33

36 Comments ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p + d i l p λ i Use hot-start for QP(W, A ) and products with W and A If strict complementarity holds and (d i, λ i ) (d, λ ): QP eventually identifies correct active set Becomes iterative refinement for equality-only case Do not need to track bound multipliers Key consequence: QP solver is responsible for handling active set Can use black-box QP solver Locally convergent (contraction, strict complementarity,... ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

37 Comments ( ) T min p 2 pt W p + g + W d i + A T λ i p s.t. A p + (c + Ad i ) = 0 p + d i l p λ i Use hot-start for QP(W, A ) and products with W and A If strict complementarity holds and (d i, λ i ) (d, λ ): QP eventually identifies correct active set Becomes iterative refinement for equality-only case Do not need to track bound multipliers Key consequence: QP solver is responsible for handling active set Can use black-box QP solver Locally convergent (contraction, strict complementarity,... ) But: Slow linear convergence rate Andreas Wächter (NU) Hot-starting NLP MIP / 33

38 Accelerated Iterative Linear Solver Can apply iterative linear solver (e.g., SQMR) to solve [ W A T A 0 ]( d λ ) ( ) g = c Andreas Wächter (NU) Hot-starting NLP MIP / 33

39 Accelerated Iterative Linear Solver Can apply iterative linear solver (e.g., SQMR) to solve [ W A T A 0 ]( d λ ) ( ) g = c Application of preconditioner requires solution of [ ]( ) ( W A T zi r g = A 0 z λ i i r c i ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

40 Accelerated Iterative Linear Solver Can apply iterative linear solver (e.g., SQMR) to solve [ W A T A 0 ]( d λ ) ( ) g = c Application of preconditioner requires solution of [ ]( ) ( W A T zi r g = A 0 z λ i i r c i ) Preconditioner QP (hot-start!) min z 2 zt W z + (r g i )T z s.t. A z + ri c = 0 zi λ Andreas Wächter (NU) Hot-starting NLP MIP / 33

41 Inequality Constraints and SQMR Iterative linear solver cannot handle inequality constraints Monitor active set A i = {j : d (j) i = l (j) } If A i = A i =: A during iterative refinement, set F = A C Apply SQMR in space of free variables (di A = l A fixed) [ ]( ) ( ) W FF (A F ) T d F g F A F = 0 λ c Preconditioner requires solution of [ W FF (A F ) T A F 0 ]( z z λ ) = ( r F g r c ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

42 Preconditioner QP [ W FF (A F ) T ]( d F ) ( g F ) A F 0 λ = c Preconditioner: [ W FF (A F ) T A F 0 ]( z z λ ) = ( r F g r c ) Could solve: min z 2 zt W z + (r F g ) T z F s.t. A z + r c = 0 z (j) = 0 for j A z λ i Andreas Wächter (NU) Hot-starting NLP MIP / 33

43 Preconditioner QP [ W FF (A F ) T ]( d F ) ( g F ) A F 0 λ = c Preconditioner: [ W FF (A F ) T A F 0 ]( z z λ ) = ( r F g r c ) Could solve: min z 2 zt W z + (r F g ) T z F s.t. A z + r c = 0 z (j) = 0 for j A z λ i Issues: As SQMR converges, we might have d F Stop SQMR, if di l l F Andreas Wächter (NU) Hot-starting NLP MIP / 33

44 Preconditioner QP [ W FF (A F ) T ]( d F ) ( g F ) A F 0 λ = c Preconditioner: [ W FF (A F ) T A F 0 ]( z z λ ) = ( r F g r c ) Could solve: min z 2 zt W z + (r F g ) T z F s.t. A z + r c = 0 z (j) = 0 for j A z λ i Issues: As SQMR converges, we might have d F l F Stop SQMR, if di l Negative bound multipliers if guess of A too large Andreas Wächter (NU) Hot-starting NLP MIP / 33

45 Preconditioner QP min z 2 zt W z + (rg F ) T z F s.t. A z + r c = 0 z (j) = 0 for j A Andreas Wächter (NU) Hot-starting NLP MIP / 33

46 Preconditioner QP ( min z 2 zt W z + (rg F ) T z F + g A + W A, d ) Tz i + (A A ) T A λ i s.t. A z + r c = 0 z (j) 0 for j A Andreas Wächter (NU) Hot-starting NLP MIP / 33

47 Preconditioner QP ( min z 2 zt W z + (rg F ) T z F + g A + W A, d ) Tz i + (A A ) T A λ i s.t. A z + r c = 0 z (j) 0 for j A If A too large Then eventually z (j) > 0 for some j A In that case, stop SQMR Andreas Wächter (NU) Hot-starting NLP MIP / 33

48 Preconditioner QP ( min z 2 zt W z + (rg F ) T z F + g A + W A, d ) Tz i + (A A ) T A λ i s.t. A z + r c = 0 z (j) 0 for j A If A too large Then eventually z (j) > 0 for some j A In that case, stop SQMR For fixed A Convergence does not depend on some contraction condition Convergence in finite number of iterations Andreas Wächter (NU) Hot-starting NLP MIP / 33

49 Algorithm Summary Two phases: Iterative refinement Solve iterative refinement QP Switch if active set same in two consecutive iterations Accelerated linear solver (SQMR) Solve preconditioning QP Switch if z A 0 or d i+ l Andreas Wächter (NU) Hot-starting NLP MIP / 33

50 Algorithm Summary Two phases: Iterative refinement Solve iterative refinement QP Switch if active set same in two consecutive iterations Accelerated linear solver (SQMR) Solve preconditioning QP Switch if z A 0 or d i+ l Requires one hot-started QP solve per iteration Requires only matrix-vector products with new matrices Andreas Wächter (NU) Hot-starting NLP MIP / 33

51 Algorithm Summary Two phases: Iterative refinement Solve iterative refinement QP Switch if active set same in two consecutive iterations Accelerated linear solver (SQMR) Solve preconditioning QP Switch if z A 0 or d i+ l Requires one hot-started QP solve per iteration Requires only matrix-vector products with new matrices Can use any black-box parametric QP solver Andreas Wächter (NU) Hot-starting NLP MIP / 33

52 Algorithm Summary Two phases: Iterative refinement Solve iterative refinement QP Switch if active set same in two consecutive iterations Accelerated linear solver (SQMR) Solve preconditioning QP Switch if z A 0 or d i+ l Requires one hot-started QP solve per iteration Requires only matrix-vector products with new matrices Can use any black-box parametric QP solver Locally convergent, assuming Strict complementarity Starting point close to (d, λ ) All QPs are feasible All KKT matrices are non-singular (No contraction condition with SQMR) Andreas Wächter (NU) Hot-starting NLP MIP / 33

53 Numerical Experiments Matlab implementation iqp QP solver: qpoases [Ferreau et al., 2008, Potschka et al., 200] Open source parametric QP solver in C++ Originally dense linear algebra Added sparse Schur complement approach (e.g. [Gill et al., 990]) (with Dennis Janka) Comparing iqp with qpoases CPU time measurements only include qpoases time Caveat: Sparse version not yet optimized Andreas Wächter (NU) Hot-starting NLP MIP / 33

54 Experiment : Randomly Generated Convex NLPs Convex NLP with random perturbation (dense matrices) min x R n 2 xt H 0 x + (q 0 ) T x + r 0 s.t. 2 xt H j x + (q j ) T x + r j 0 j =,..., m x 0 n {50, 200, 500, 000} m {0.2n, 0.5n, 0.8n,.5n} σ {0.0, 0.05, 0., 0.2} standard deviation of perturbation Solve sequences of 0 perturbed NLPs (3 for n = 000) Solve first NLP, store final QP as reference QP Sl QP implemented in p-sqp [Curtis] QPs solved to tight tolerance (0 8 to 0 2 ) Andreas Wächter (NU) Hot-starting NLP MIP / 33

55 Randomly Generated Convex NLPs Results Size of perturbation σ Successfully solved 99.3% 99.3% 97.7% 42.9% Avrg # iqp iters per SQP iter Avrg change in # active ineq 5.3% 9.0% 25.7% 32.4% Avrg change in # active bounds 3.4% 9.5% 3.8% 9.2% Andreas Wächter (NU) Hot-starting NLP MIP / 33

56 Randomly Generated Convex NLPs Results 00 m = 0.2n 00 m = 0.8n CPU qpoases CPU iqp 0 σ = 0.0 σ = 0.05 σ = 0. σ = 0.2. n=50 n=200 n=500 n= σ = 0.0 σ = 0.05 σ = 0. σ = 0.2. n=50 n=200 n=500 n= MatVec qpoases MatVec iqp 0 σ = 0.0 σ = 0.05 σ = 0. σ = 0.2. n=50 n=200 n=500 n=000 0 σ = 0.0 σ = 0.05 σ = 0. σ = 0.2. n=50 n=200 n=500 n=000 Andreas Wächter (NU) Hot-starting NLP MIP / 33

57 Experiment 2: Model Predictive Control Controlling hanging chain From starting position (left) Controll red ball to get chain to steady state (right) as quickly as possible Andreas Wächter (NU) Hot-starting NLP MIP / 33

58 Experiment 2: Model Predictive Control Controlling hanging chain (DAE model) T NPM min w v v i(t) w x x NPM (t) x e w u u(t) w sl u sl (t) 2 2 dt x( ),u( ) 0 i= s.t. ẋ i(t) = v i(t) t [0, T], i < N PM v i(t) = (F i+(t) F i(t)) N PM /m g t [0, T], i < N PM ẋ NPM (t) = u(t) t [0, T] x(0) = ˆx 0 u(t) [, ] 3 t [0, T] x low x (t) x high, x3 low u sl (t) x 3(t) t [0, T], u sl (t) 0, t [0, T]. Solve one QP per time step Reference QP obtained from steady state Multiple shooting approach Integrate differential equations over subintervals Computation of full derivative matrices expensive Here: Performance metric is number of matrix-vector products Andreas Wächter (NU) Hot-starting NLP MIP / 33

59 Model Predictive Control Results 5 multiple shooting intervals QP Size Solve Matrix-Vector Products N PM n m Success Min Mean Max Full A k , ,328, ,522, multiple shooting intervals QP Size Solve Matrix-Vector Products N PM n m Success Min Mean Max Full A k ,55, ,449, ,743, ,977, Andreas Wächter (NU) Hot-starting NLP MIP / 33

60 Experiment 3: MINLP Strong Branching Strong branching on root node Basic SQP method (no line search) Get approximate solution from iqp Fails if QP or NLP infeasible NLP i Root Node Store QP(x, λ ) xi xi xi xi NLP + i Instances from IBMCMU collection Convex problems, but Hessian singular Few nonlinearities Very sparse QP± i (x0, λ0) QP± i (x, λ) QP± i (xk, λk) QP i ± (x0, λ0) QP i ± (x, λ) QP i ± (xk, λk) Limitations of current qpoases code Only heuristic for handling Hessian that are not positive definite Failures due to numerical issues Refactorization triggered by ill-conditioned KKT matrix Too many failures for CLay and FLay instances Results very preliminary Andreas Wächter (NU) Hot-starting NLP MIP / 33

61 Strong Branching Results: Batch (scaled) 3 Instances total 0 Instances excluded b/c no NLP solved successfully 0 Instances excluded b/c more than 30% of NLP subproblem required refactorization for iqp 3 Instances in averages below iqp qpoases % Solved % Both solved % Factorized SQP it inner it 6.36 QP pivots iqp qpoases CPU.07 QP pivots 2.9 Fact 0.0 total CPU.36 total QP pivots.92 total Fact 0.03 : Average over all NLP subproblems of an instance Andreas Wächter (NU) Hot-starting NLP MIP / 33

62 Strong Branching Results: RSyn 48 Instances total 2 Instances excluded b/c no NLP solved successfully 27 Instances excluded b/c more than 30% of NLP subproblem required refactorization for iqp 9 Instances in averages below iqp qpoases % Solved % Both solved % Factorized SQP it inner it 4.82 QP pivots iqp qpoases CPU.03 QP pivots.58 Fact 0.7 total CPU 0.84 total QP pivots.34 total Fact 0.6 : Average over all NLP subproblems of an instance Andreas Wächter (NU) Hot-starting NLP MIP / 33

63 Strong Branching Results: SLay 4 Instances total 0 Instances excluded b/c no NLP solved successfully 0 Instances excluded b/c more than 30% of NLP subproblem required refactorization for iqp 4 Instances in averages below iqp qpoases % Solved % Both solved % Factorized SQP it inner it.00 QP pivots iqp qpoases CPU 0.73 QP pivots 0.88 Fact 0.00 total CPU 0.62 total QP pivots 0.8 total Fact 0.00 : Average over all NLP subproblems of an instance Andreas Wächter (NU) Hot-starting NLP MIP / 33

64 Strong Branching Results: Syn 48 Instances total 9 Instances excluded b/c no NLP solved successfully 6 Instances excluded b/c more than 30% of NLP subproblem required refactorization for iqp 33 Instances in averages below iqp qpoases % Solved % Both solved % Factorized SQP it inner it 2.93 QP pivots iqp qpoases CPU.96 QP pivots 3.20 Fact 0.04 total CPU 2.2 total QP pivots 3.45 total Fact 0.06 : Average over all NLP subproblems of an instance Andreas Wächter (NU) Hot-starting NLP MIP / 33

65 Experiments Summary Dense random NLPs Time factorization Time solve Significant speedup Optimal control Reducing expensive derivative calculations Strong branching Results very preliminary Sparse version of qpoases not yet mature and robust advanced tailored implementation would be faster Encouraging observations Solved many of the NLP subproblems When numerical problems are limited, similar performance Other MINLP problems More speedup for denser problems? Still rebust for problems with more nonlinearities? Andreas Wächter (NU) Hot-starting NLP MIP / 33

66 Conclusion New QP solver, using repeated hot starts for Iterative refinement Preconditioning in accelerated linear solver Somewhat encouraging preliminary results for MINLP Potential improvements Employ robust parametric QP solver Use matrix data of new QP when Updating factorization Refactorization necessary In MINLP context: Makes NLP-based branch-and-bound more attractive? Useful in other situations? Andreas Wächter (NU) Hot-starting NLP MIP / 33

67 THANK YOU! Andreas Wächter (NU) Hot-starting NLP MIP / 33