Nonlinear Optimization Solvers
|
|
- Godwin Marsh
- 5 years ago
- Views:
Transcription
1 ICE05 Argonne, July 19, 2005 Nonlinear Optimization Solvers Sven Leyffer Mathematics & Computer Science Division, Argonne National Laboratory Nonlinear Optimization Methods Optimization Software Troubleshooting Conclusions
2 NONLINEAR OPTIMIZATION METHODS Sven Leyffer Nonlinear Optimization Solvers 2 of 51
3 Overview: Nonlinear Optimization Methods Smooth nonlinear programming (NLP) problem Solution methods: minimize f (x) objective x subject to c(x) = 0 constraints x 0 variables 1. active set methods: SQP & projected gradient 2. interior point methods converge near solution x... Sven Leyffer Nonlinear Optimization Solvers 3 of 51
4 Sequential Quadratic Programming (SQP) General NLP minimize x f (x) subject to c(x) = 0 & x 0 REPEAT 1. Solve QP for (s, y k+1, z k+1 ) minimize s f T k s st H k s subject to c k + ck T s = 0 ( y k+1) x k + s 0 ( z k+1 0) 2. Set x k+1 = x k + s, & k = k + 1 UNTIL convergence... estimate active set A k := {i : x i = 0}... quadratic convergence near x... once A identified Sven Leyffer Nonlinear Optimization Solvers 4 of 51
5 General NLP Interior Point Methods (IPM) minimize x f (x) subject to c(x) = 0 & x 0 Perturbed µ > 0 optimality conditions (x, z 0) f (x) c(x) T y z F µ (x, y, z) = c(x) = 0 Xz µe Primal-dual formulation Central path {x(µ), y(µ), z(µ) : µ > 0} Apply Newton s method for sequence µ 0 Sven Leyffer Nonlinear Optimization Solvers 5 of 51
6 Interior Point Methods (IPM) REPEAT 1. Choose µ k 0 2. Solve F µk (x, y, z) = 0 to accuracy O(µ) 2 L k c(x k ) T I x c(x k ) 0 0 y = F µk (x k, y k, z k ) Z k 0 X k z where X k diagonal matrix of x k 3. Set x k+1 = x k + x,... & k = k + 1 UNTIL convergence... SQP & IPM are variants of Newton s method Sven Leyffer Nonlinear Optimization Solvers 6 of 51
7 GLOBAL CONVERGENCE Sven Leyffer Nonlinear Optimization Solvers 7 of 51
8 Overview: Global Convergence SQP & IPM converge quadratically near solution... but diverge far from solution convergence from remote starting points: 1. merit / penalty function to measure progress 2. enforce descent in merit function by line-search... backtrack on SQP step: x k + α x for α = 1, 1 2, restrict step with trust-region: x ρ k... interfere as little as possible with SQP/IPM Sven Leyffer Nonlinear Optimization Solvers 8 of 51
9 Penalty & Merit Functions Augmented Lagrangian (LANCELOT & MINOS) minimize x L(x, y k, ρ k ) = f (x) y T k c(x) ρ k c(x) 2 As y k y : x k x for ρ k > ρ No ill-conditioning, improves convergence rate Exact penalty function: π > 0 penalty parameter minimize x Φ(x, π) = f (x) + π c(x) equivalence of optimality exact for π > y D nonsmooth Sl 1 QP method Sven Leyffer Nonlinear Optimization Solvers 9 of 51
10 Trust Region Methods Globalize SQP (IPM) by adding trust region, k > 0 { minimize fk T s + 1 s 2 st H k s subject to c k + A T k s = 0, x k + s 0, s k Sven Leyffer Nonlinear Optimization Solvers 10 of 51
11 Trust Region Methods Globalize SQP (IPM) by adding trust region, k > 0 { minimize fk T s + 1 s 2 st H k s subject to c k + A T k s = 0, x k + s 0, s k Sven Leyffer Nonlinear Optimization Solvers 10 of 51
12 OPTIMIZATION SOFTWARE Sven Leyffer Nonlinear Optimization Solvers 11 of 51
13 Overview: Some Optimization Software Augmented Lagrangian LANCELOT: bound constrained; trust-region MINOS: linearly constrained; line-search PENNON: line-search or trust-region Sequential quadratic programming FILTER: trust-region; no penalty function KNITRO: trust-region; SLP-EQP... options "alg=3"; SNOPT: line-search; l 1 exact penalty function Interior point methods KNITRO: trust-region; SQP on barrier problem LOQO: line-search; diagonal perturbation IPOPT: line-search; no penalty function Sven Leyffer Nonlinear Optimization Solvers 12 of 51
14 Rule of Thumb for Solver Choice Subjective rules of thumb... trust-region more robust than line-search IPM faster than SQP... but... SQP more robust than IPM projected gradient is best for bound constrained optimization IPM may fail, if feasible set empty Sven Leyffer Nonlinear Optimization Solvers 13 of 51
15 Solver Options in AMPL 1. Setting solver options in AMPL: option snopt_options "outlev=2 \ timing=1"; 2. Useful solver options: outlev output level timing report solution times maxiter max. number of iterations mu initial barrier parameter compl_frm form of complements equation 3. Check with each solver for names, e.g. knitro -=... to see list of options (from unix prompt) Sven Leyffer Nonlinear Optimization Solvers 15 of 51
16 Life Cycle Savings param T integer; # number of periods param beta, default 0.9; # utility param param r, default 0.2; # interest rate param w{1..t} >= 0; # wages in period t var S{0..T} >= 0; # savings in period t var c{1..t} >= 0; # consumption in period t maximize utility: sum{t in 1..T} beta^t * (-exp(-c[t])); subject to budget{t in 0..T-1}: S[t+1] = (1+r)*S[t] + w[t+1] - c[t+1]; Cinit: S[0] = 0; Cend: S[T] = 0; Sven Leyffer Nonlinear Optimization Solvers 17 of 51
17 Solver Output for SNOPT Major Minors Step nobj Feasible Optimal Objective ns E E r E E E n rl E E E s E+00 9 (2.3E-07) E EXIT -- optimal solution found Problem name problem No. of iterations 28 Objective value E+00 No. of major iterations 8 Linear objective E+00 Penalty parameter 0.000E+00 Nonlinear objective E+00 No. of calls to funobj 10 No. of calls to funcon 0 No. of superbasics 13 No. of basic nonlinears 20 No. of degenerate steps 5 Percentage Norm of x (scaled) 6.4E+00 Norm of pi (scaled) 1.0E+00 Norm of x 3.6E+00 Norm of pi 1.0E+00 Max Prim inf(scaled) 0 0.0E+00 Max Dual inf(scaled) E-07 Max Primal infeas 0 0.0E+00 Max Dual infeas E-07 Sven Leyffer Nonlinear Optimization Solvers 19 of 51
18 Solver Output for LOQO LOQO 6.06: outlev=2 variables: non-neg 49, free 0, bdd 0, total 49 constraints: eq 25, ineq 0, ranged 0, total 25 nonzeros: A 73, Q Primal Dual Sig Iter Obj Value Infeas Obj Value Infeas Fig Status P M e e e e nonzeros: L 151, arith_ops e e e e e e e e e e e e-05 4 PF e e e e-07 6 PF DF e e e e-08 7 PF DF OPTIMAL SOLUTION FOUND; Finished call LOQO 6.06: optimal solution (15 iterations, 15 evaluations) primal objective dual objective Sven Leyffer Nonlinear Optimization Solvers 21 of 51
19 TROUBLESHOOTING Sven Leyffer Nonlinear Optimization Solvers 22 of 51
20 Something Under the Bed is Drooling 1. floating point (IEEE) exceptions 2. unbounded problems 2.1 unbounded objective 2.2 unbounded multipliers 3. (locally) inconsistent problems 4. suboptimal solutions... identify problem & suggest remedies Sven Leyffer Nonlinear Optimization Solvers 23 of 51
21 Floating Point (IEEE) Exceptions Bad example: minimize barrier function param mu default 1; var x{1..2} >= -10, <= 10; var s; minimize barrier: x[1]^2 + x[2]^2 - mu*log(s); subject to cons: s = x[1] + x[2]^2-1;... results in error message like Cannot evaluate objective at start... change initialization of s: var s := 1;... difficult, if IEEE during solve... Sven Leyffer Nonlinear Optimization Solvers 25 of 51
22 Unbounded Objective Penalized MPEC π = 1: minimize x1 2 + x2 2 4x 1 x 2 + πx 1 x 2 x subject to x 1, x unbounded below for all π < 2 Sven Leyffer Nonlinear Optimization Solvers 26 of 51
23 Unbounded Multipliers for MPECs A nasty MPEC example... multipliers do not exist... var z{1..2} >= 0; var z3; minimize objf: z[1] + z[2] - z3; subject to lin1: -4 * z[1] + z3 <= 0; lin2: -4 * z[2] + z3 <= 0; compl: z[1]*z[2] <= 0;... solution: (z[1],z[2],z3) = (0,0,0) with objf = 0 Sven Leyffer Nonlinear Optimization Solvers 28 of 51
24 Unbounded Multipliers LOQO LOQO 6.06: outlev=2 Primal Dual Iter Obj Value Infeas Obj Value Infeas e e e e e e e e e e e e e e e e e e e e nan nan nan nan 500 nan nan nan nan nan = not a number... desaster... switch solver Sven Leyffer Nonlinear Optimization Solvers 30 of 51
25 Unbounded Multipliers FILTER iter rho d f / hj c /hjt : : : E E E-12 25: E E E-13 Norm of KKT residual max( lam_i * a_i ) Largest modulus multiplier large multiplier of z[1]*z[2]<=0 multipliers NLP solvers converge linearly luckily this behaviour is very rare Sven Leyffer Nonlinear Optimization Solvers 32 of 51
26 Unbounded Complementarity Multiplier Multiplier ξ of X 1 x 2 0 becomes large MPEC multipliers (B-stationary) Remedy: 1. check residuals of constraints (if large then failed) 2. find degenerate indices D := {i : x 1i + x 2i < ɛ} 3. fix x 1i = 0 or x 2i = 0 for all i D In previous example: solve; fix z[1] := 0; solve; Prove optimality: examine 2 D combinations x 1i or x 2i = 0 Sven Leyffer Nonlinear Optimization Solvers 33 of 51
27 Locally Inconsistent Problems NLP may have no feasible point feasible set: intersection of circles Sven Leyffer Nonlinear Optimization Solvers 35 of 51
28 Locally Inconsistent Problems NLP may have no feasible point var x{1..2} >= -1; minimize objf: -1000*x[2]; subject to con1: (x[1]+2)^2 + x[2]^2 <= 1; con2: (x[1]-2)^2 + x[2]^2 <= 1; not all solvers recognize this... finding feasible point global optimization Sven Leyffer Nonlinear Optimization Solvers 35 of 51
29 Locally Inconsistent Problems LOQO Primal Dual Iter Obj Value Infeas Obj Value Infeas e e e e-00 [...] e e e e-01 LOQO 6.06: iteration limit... fails to converge... not useful for user dual unbounded primal infeasible Sven Leyffer Nonlinear Optimization Solvers 37 of 51
30 Locally Inconsistent Problems FILTER iter rho d f / hj c /hjt : : [...] 8: E E-04 9: E E-10 filtersqp: Nonlinear constraints locally infeasible... fast convergence to minimum infeasibility... identify blocking constraints... modify model/data Sven Leyffer Nonlinear Optimization Solvers 39 of 51
31 Locally Inconsistent Problems Remedies for locally infeasible problems: 1. check your model: print constraints & residuals, e.g. solve; display conname, con.lb, con.body, con.ub; display varname, var.lb, var, var.ub;... look at violated and active constraints 2. try different nonlinear solvers (easy with AMPL) 3. build-up model from few constraints at a time 4. try different starting points... global optimization Sven Leyffer Nonlinear Optimization Solvers 40 of 51
32 Inconsistent Problems: Sequential Approach Example heart6.mod SNOPT: nonlinear infeasibilities minimized... after 549 iters model heart6.mod; # load model problem heart614: obj, # dummy objective a, c, t, u, v, w, # variables cons1, cons2, cons3, # partial cons4; #... constraints solve heart614; # partial model partial solution after 6 iterations... Sven Leyffer Nonlinear Optimization Solvers 42 of 51
33 Inconsistent Problems: Sequential Approach Example heart6.mod SNOPT: nonlinear infeasibilities minimized... after 549 iters #... continue from prev. slide problem heart616: obj, # dummy objective a, c, t, u, v, w, # variables cons1, cons2, cons3, # all cons4, cons5, cons6; #... constraints solve heart616; # full model optimal solution after 6+20 iterations...excellent Sven Leyffer Nonlinear Optimization Solvers 44 of 51
34 Suboptimal Solution & Multi-start Problems can have many local mimimizers default start point converges to local minimizer Sven Leyffer Nonlinear Optimization Solvers 45 of 51
35 Suboptimal Solution & Multi-start Problems can have many local mimimizers param pi := ; param n integer, >= 0, default 2; set N := 1..n; var x{n} >= 0, <= 32*pi, := 1; minimize objf: - sum{i in N} x[i]*sin(sqrt(x[i])); default start point converges to local minimizer Sven Leyffer Nonlinear Optimization Solvers 45 of 51
36 Suboptimal Solution & Multi-start param nd := 5; # discretization set D := 1..nD; param hd := 32*pi/(nD-1); param optval{d,d}; model schwefel.mod; # load model for {i in D}{ let x[1] := (i-1)*hd; for {j in D}{ let x[2] := (j-1)*hd; solve; let optval[i,j] := objf; }; # end for }; # end for Sven Leyffer Nonlinear Optimization Solvers 47 of 51
37 Suboptimal Solution & Multi-start display optval; optval [*,*] : := ;... there exist better multi-start procedures Sven Leyffer Nonlinear Optimization Solvers 49 of 51
38 CONCLUSIONS Sven Leyffer Nonlinear Optimization Solvers 50 of 51
39 Conclusions Nonlinear optimization much harder than LP: there is no best solver best solver is problem dependent learn a little about solvers/methods help with troubleshooting multistart & model build-up in AMPL please write good models... Sven Leyffer Nonlinear Optimization Solvers 51 of 51
Infeasibility Detection in Nonlinear Optimization
Infeasibility Detection in Nonlinear Optimization Frank E. Curtis, Lehigh University Hao Wang, Lehigh University SIAM Conference on Optimization 2011 May 16, 2011 (See also Infeasibility Detection in Nonlinear
More informationSolving MPECs Implicit Programming and NLP Methods
Solving MPECs Implicit Programming and NLP Methods Michal Kočvara Academy of Sciences of the Czech Republic September 2005 1 Mathematical Programs with Equilibrium Constraints Mechanical motivation Mechanical
More informationWhat s New in Active-Set Methods for Nonlinear Optimization?
What s New in Active-Set Methods for Nonlinear Optimization? Philip E. Gill Advances in Numerical Computation, Manchester University, July 5, 2011 A Workshop in Honor of Sven Hammarling UCSD Center for
More informationInfeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization
Infeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with James V. Burke, University of Washington Daniel
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationSurvey of NLP Algorithms. L. T. Biegler Chemical Engineering Department Carnegie Mellon University Pittsburgh, PA
Survey of NLP Algorithms L. T. Biegler Chemical Engineering Department Carnegie Mellon University Pittsburgh, PA NLP Algorithms - Outline Problem and Goals KKT Conditions and Variable Classification Handling
More informationNonlinear Programming, Elastic Mode, SQP, MPEC, MPCC, complementarity
Preprint ANL/MCS-P864-1200 ON USING THE ELASTIC MODE IN NONLINEAR PROGRAMMING APPROACHES TO MATHEMATICAL PROGRAMS WITH COMPLEMENTARITY CONSTRAINTS MIHAI ANITESCU Abstract. We investigate the possibility
More informationLecture 13: Constrained optimization
2010-12-03 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems
More informationPenalty and Barrier Methods General classical constrained minimization problem minimize f(x) subject to g(x) 0 h(x) =0 Penalty methods are motivated by the desire to use unconstrained optimization techniques
More informationInterior Methods for Mathematical Programs with Complementarity Constraints
Interior Methods for Mathematical Programs with Complementarity Constraints Sven Leyffer, Gabriel López-Calva and Jorge Nocedal July 14, 25 Abstract This paper studies theoretical and practical properties
More informationConstrained Nonlinear Optimization Algorithms
Department of Industrial Engineering and Management Sciences Northwestern University waechter@iems.northwestern.edu Institute for Mathematics and its Applications University of Minnesota August 4, 2016
More information1 Computing with constraints
Notes for 2017-04-26 1 Computing with constraints Recall that our basic problem is minimize φ(x) s.t. x Ω where the feasible set Ω is defined by equality and inequality conditions Ω = {x R n : c i (x)
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationLecture 15: SQP methods for equality constrained optimization
Lecture 15: SQP methods for equality constrained optimization Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 15: SQP methods for equality constrained
More informationPOWER SYSTEMS in general are currently operating
TO APPEAR IN IEEE TRANSACTIONS ON POWER SYSTEMS 1 Robust Optimal Power Flow Solution Using Trust Region and Interior-Point Methods Andréa A. Sousa, Geraldo L. Torres, Member IEEE, Claudio A. Cañizares,
More informationINTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: COMPLEMENTARITY CONSTRAINTS
INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: COMPLEMENTARITY CONSTRAINTS HANDE Y. BENSON, DAVID F. SHANNO, AND ROBERT J. VANDERBEI Operations Research and Financial Engineering Princeton
More informationLecture 3 Interior Point Methods and Nonlinear Optimization
Lecture 3 Interior Point Methods and Nonlinear Optimization Robert J. Vanderbei April 16, 2012 Machine Learning Summer School La Palma http://www.princeton.edu/ rvdb Example: Basis Pursuit Denoising L
More informationINTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: CONVERGENCE ANALYSIS AND COMPUTATIONAL PERFORMANCE
INTERIOR-POINT METHODS FOR NONCONVEX NONLINEAR PROGRAMMING: CONVERGENCE ANALYSIS AND COMPUTATIONAL PERFORMANCE HANDE Y. BENSON, ARUN SEN, AND DAVID F. SHANNO Abstract. In this paper, we present global
More informationUniversity of Erlangen-Nürnberg and Academy of Sciences of the Czech Republic. Solving MPECs by Implicit Programming and NLP Methods p.
Solving MPECs by Implicit Programming and NLP Methods Michal Kočvara University of Erlangen-Nürnberg and Academy of Sciences of the Czech Republic Solving MPECs by Implicit Programming and NLP Methods
More informationSF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren
SF2822 Applied Nonlinear Optimization Lecture 9: Sequential quadratic programming Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH / 24 Lecture 9, 207/208 Preparatory question. Try to solve theory
More informationMultidisciplinary System Design Optimization (MSDO)
Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential
More informationAn Inexact Sequential Quadratic Optimization Method for Nonlinear Optimization
An Inexact Sequential Quadratic Optimization Method for Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with Travis Johnson, Northwestern University Daniel P. Robinson, Johns
More informationA New Penalty-SQP Method
Background and Motivation Illustration of Numerical Results Final Remarks Frank E. Curtis Informs Annual Meeting, October 2008 Background and Motivation Illustration of Numerical Results Final Remarks
More informationLP. Kap. 17: Interior-point methods
LP. Kap. 17: Interior-point methods the simplex algorithm moves along the boundary of the polyhedron P of feasible solutions an alternative is interior-point methods they find a path in the interior of
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationarxiv: v1 [math.na] 8 Jun 2018
arxiv:1806.03347v1 [math.na] 8 Jun 2018 Interior Point Method with Modified Augmented Lagrangian for Penalty-Barrier Nonlinear Programming Martin Neuenhofen June 12, 2018 Abstract We present a numerical
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationIP-PCG An interior point algorithm for nonlinear constrained optimization
IP-PCG An interior point algorithm for nonlinear constrained optimization Silvia Bonettini (bntslv@unife.it), Valeria Ruggiero (rgv@unife.it) Dipartimento di Matematica, Università di Ferrara December
More informationNonlinear optimization
Nonlinear optimization Anders Forsgren Optimization and Systems Theory Department of Mathematics Royal Institute of Technology (KTH) Stockholm, Sweden evita Winter School 2009 Geilo, Norway January 11
More information2.3 Linear Programming
2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are
More informationThe Squared Slacks Transformation in Nonlinear Programming
Technical Report No. n + P. Armand D. Orban The Squared Slacks Transformation in Nonlinear Programming August 29, 2007 Abstract. We recall the use of squared slacks used to transform inequality constraints
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationPenalty and Barrier Methods. So we again build on our unconstrained algorithms, but in a different way.
AMSC 607 / CMSC 878o Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 3: Penalty and Barrier Methods Dianne P. O Leary c 2008 Reference: N&S Chapter 16 Penalty and Barrier
More informationA Primal-Dual Augmented Lagrangian Penalty-Interior-Point Filter Line Search Algorithm
Journal name manuscript No. (will be inserted by the editor) A Primal-Dual Augmented Lagrangian Penalty-Interior-Point Filter Line Search Algorithm Rene Kuhlmann Christof Büsens Received: date / Accepted:
More informationCONVERGENCE ANALYSIS OF AN INTERIOR-POINT METHOD FOR NONCONVEX NONLINEAR PROGRAMMING
CONVERGENCE ANALYSIS OF AN INTERIOR-POINT METHOD FOR NONCONVEX NONLINEAR PROGRAMMING HANDE Y. BENSON, ARUN SEN, AND DAVID F. SHANNO Abstract. In this paper, we present global and local convergence results
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationPart 5: Penalty and augmented Lagrangian methods for equality constrained optimization. Nick Gould (RAL)
Part 5: Penalty and augmented Lagrangian methods for equality constrained optimization Nick Gould (RAL) x IR n f(x) subject to c(x) = Part C course on continuoue optimization CONSTRAINED MINIMIZATION x
More informationSolving Multi-Leader-Common-Follower Games
ARGONNE NATIONAL LABORATORY 9700 South Cass Avenue Argonne, Illinois 60439 Solving Multi-Leader-Common-Follower Games Sven Leyffer and Todd Munson Mathematics and Computer Science Division Preprint ANL/MCS-P1243-0405
More informationPDE-Constrained and Nonsmooth Optimization
Frank E. Curtis October 1, 2009 Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP)
More informationNumerical Optimization for Economists
Numerical Optimization for Economists Todd Munson Mathematics and Computer Science Division Argonne National Laboratory July 18 21, 2011 1 / 104 Part I Numerical Optimization I: Static Models 2 / 104 Model
More informationCS6375: Machine Learning Gautam Kunapuli. Support Vector Machines
Gautam Kunapuli Example: Text Categorization Example: Develop a model to classify news stories into various categories based on their content. sports politics Use the bag-of-words representation for this
More informationOptimization Problems with Constraints - introduction to theory, numerical Methods and applications
Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationAn Inexact Newton Method for Nonlinear Constrained Optimization
An Inexact Newton Method for Nonlinear Constrained Optimization Frank E. Curtis Numerical Analysis Seminar, January 23, 2009 Outline Motivation and background Algorithm development and theoretical results
More informationAN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING
AN AUGMENTED LAGRANGIAN AFFINE SCALING METHOD FOR NONLINEAR PROGRAMMING XIAO WANG AND HONGCHAO ZHANG Abstract. In this paper, we propose an Augmented Lagrangian Affine Scaling (ALAS) algorithm for general
More informationREGULARIZED SEQUENTIAL QUADRATIC PROGRAMMING METHODS
REGULARIZED SEQUENTIAL QUADRATIC PROGRAMMING METHODS Philip E. Gill Daniel P. Robinson UCSD Department of Mathematics Technical Report NA-11-02 October 2011 Abstract We present the formulation and analysis
More informationINTERIOR-POINT ALGORITHMS, PENALTY METHODS AND EQUILIBRIUM PROBLEMS
INTERIOR-POINT ALGORITHMS, PENALTY METHODS AND EQUILIBRIUM PROBLEMS HANDE Y. BENSON, ARUN SEN, DAVID F. SHANNO, AND ROBERT J. VANDERBEI Abstract. In this paper we consider the question of solving equilibrium
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationThe use of second-order information in structural topology optimization. Susana Rojas Labanda, PhD student Mathias Stolpe, Senior researcher
The use of second-order information in structural topology optimization Susana Rojas Labanda, PhD student Mathias Stolpe, Senior researcher What is Topology Optimization? Optimize the design of a structure
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationSolution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark
Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT
More informationSolving a Signalized Traffic Intersection Problem with NLP Solvers
Solving a Signalized Traffic Intersection Problem with NLP Solvers Teófilo Miguel M. Melo, João Luís H. Matias, M. Teresa T. Monteiro CIICESI, School of Technology and Management of Felgueiras, Polytechnic
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationInexact Newton Methods and Nonlinear Constrained Optimization
Inexact Newton Methods and Nonlinear Constrained Optimization Frank E. Curtis EPSRC Symposium Capstone Conference Warwick Mathematics Institute July 2, 2009 Outline PDE-Constrained Optimization Newton
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization Instructor: Michael Saunders Spring 2015 Notes 11: NPSOL and SNOPT SQP Methods 1 Overview
More informationDerivative-Based Numerical Method for Penalty-Barrier Nonlinear Programming
Derivative-Based Numerical Method for Penalty-Barrier Nonlinear Programming Martin Peter Neuenhofen October 26, 2018 Abstract We present an NLP solver for nonlinear optimization with quadratic penalty
More informationLecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming
Lecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 11 and
More informationNumerical Optimization
Constrained Optimization - Algorithms Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Consider the problem: Barrier and Penalty Methods x X where X
More informationAn Inexact Newton Method for Optimization
New York University Brown Applied Mathematics Seminar, February 10, 2009 Brief biography New York State College of William and Mary (B.S.) Northwestern University (M.S. & Ph.D.) Courant Institute (Postdoc)
More informationApolynomialtimeinteriorpointmethodforproblemswith nonconvex constraints
Apolynomialtimeinteriorpointmethodforproblemswith nonconvex constraints Oliver Hinder, Yinyu Ye Department of Management Science and Engineering Stanford University June 28, 2018 The problem I Consider
More informationNumerical Nonlinear Optimization with WORHP
Numerical Nonlinear Optimization with WORHP Christof Büskens Optimierung & Optimale Steuerung London, 8.9.2011 Optimization & Optimal Control Nonlinear Optimization WORHP Concept Ideas Features Results
More informationGAMS/MINOS MINOS 1 GAMS/MINOS. Table of Contents:
GAMS/MINOS MINOS 1 GAMS/MINOS Table of Contents:... 1. INTRODUCTION... 2 2. HOW TO RUN A MODEL WITH GAMS/MINOS... 2 3. OVERVIEW OF GAMS/MINOS... 2 3.1. Linear programming... 3 3.2. Problems with a Nonlinear
More informationALADIN An Algorithm for Distributed Non-Convex Optimization and Control
ALADIN An Algorithm for Distributed Non-Convex Optimization and Control Boris Houska, Yuning Jiang, Janick Frasch, Rien Quirynen, Dimitris Kouzoupis, Moritz Diehl ShanghaiTech University, University of
More informationSoftware for Integer and Nonlinear Optimization
Software for Integer and Nonlinear Optimization Sven Leyffer, leyffer@mcs.anl.gov Mathematics & Computer Science Division Argonne National Laboratory Roger Fletcher & Jeff Linderoth Advanced Methods and
More informationarxiv: v2 [math.oc] 11 Jan 2018
A one-phase interior point method for nonconvex optimization Oliver Hinder, Yinyu Ye January 12, 2018 arxiv:1801.03072v2 [math.oc] 11 Jan 2018 Abstract The work of Wächter and Biegler [40] suggests that
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationImplementation of an Interior Point Multidimensional Filter Line Search Method for Constrained Optimization
Proceedings of the 5th WSEAS Int. Conf. on System Science and Simulation in Engineering, Tenerife, Canary Islands, Spain, December 16-18, 2006 391 Implementation of an Interior Point Multidimensional Filter
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for
More informationPrimal-Dual Interior-Point Methods
Primal-Dual Interior-Point Methods Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 Outline Today: Primal-dual interior-point method Special case: linear programming
More informationLarge-Scale Nonlinear Optimization with Inexact Step Computations
Large-Scale Nonlinear Optimization with Inexact Step Computations Andreas Wächter IBM T.J. Watson Research Center Yorktown Heights, New York andreasw@us.ibm.com IPAM Workshop on Numerical Methods for Continuous
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationNUMERICAL OPTIMIZATION. J. Ch. Gilbert
NUMERICAL OPTIMIZATION J. Ch. Gilbert Numerical optimization (past) The discipline deals with the classical smooth (nonconvex) problem min {f(x) : c E (x) = 0, c I (x) 0}. Applications: variable added
More informationA PRIMAL-DUAL TRUST REGION ALGORITHM FOR NONLINEAR OPTIMIZATION
Optimization Technical Report 02-09, October 2002, UW-Madison Computer Sciences Department. E. Michael Gertz 1 Philip E. Gill 2 A PRIMAL-DUAL TRUST REGION ALGORITHM FOR NONLINEAR OPTIMIZATION 7 October
More informationINTERIOR-POINT METHODS FOR NONLINEAR, SECOND-ORDER CONE, AND SEMIDEFINITE PROGRAMMING. Hande Yurttan Benson
INTERIOR-POINT METHODS FOR NONLINEAR, SECOND-ORDER CONE, AND SEMIDEFINITE PROGRAMMING Hande Yurttan Benson A DISSERTATION PRESENTED TO THE FACULTY OF PRINCETON UNIVERSITY IN CANDIDACY FOR THE DEGREE OF
More informationNumerical Methods for PDE-Constrained Optimization
Numerical Methods for PDE-Constrained Optimization Richard H. Byrd 1 Frank E. Curtis 2 Jorge Nocedal 2 1 University of Colorado at Boulder 2 Northwestern University Courant Institute of Mathematical Sciences,
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationSupport Vector Machine (continued)
Support Vector Machine continued) Overlapping class distribution: In practice the class-conditional distributions may overlap, so that the training data points are no longer linearly separable. We need
More informationHot-Starting NLP Solvers
Hot-Starting NLP Solvers Andreas Wächter Department of Industrial Engineering and Management Sciences Northwestern University waechter@iems.northwestern.edu 204 Mixed Integer Programming Workshop Ohio
More informationINTERIOR-POINT ALGORITHMS, PENALTY METHODS AND EQUILIBRIUM PROBLEMS
INTERIOR-POINT ALGORITHMS, PENALTY METHODS AND EQUILIBRIUM PROBLEMS HANDE Y. BENSON, ARUN SEN, DAVID F. SHANNO, AND ROBERT J. VANDERBEI Abstract. In this paper we consider the question of solving equilibrium
More informationRecent Adaptive Methods for Nonlinear Optimization
Recent Adaptive Methods for Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with James V. Burke (U. of Washington), Richard H. Byrd (U. of Colorado), Nicholas I. M. Gould
More informationTrust-Region SQP Methods with Inexact Linear System Solves for Large-Scale Optimization
Trust-Region SQP Methods with Inexact Linear System Solves for Large-Scale Optimization Denis Ridzal Department of Computational and Applied Mathematics Rice University, Houston, Texas dridzal@caam.rice.edu
More informationOn sequential optimality conditions for constrained optimization. José Mario Martínez martinez
On sequential optimality conditions for constrained optimization José Mario Martínez www.ime.unicamp.br/ martinez UNICAMP, Brazil 2011 Collaborators This talk is based in joint papers with Roberto Andreani
More informationPreprint ANL/MCS-P , Dec 2002 (Revised Nov 2003, Mar 2004) Mathematics and Computer Science Division Argonne National Laboratory
Preprint ANL/MCS-P1015-1202, Dec 2002 (Revised Nov 2003, Mar 2004) Mathematics and Computer Science Division Argonne National Laboratory A GLOBALLY CONVERGENT LINEARLY CONSTRAINED LAGRANGIAN METHOD FOR
More informationIPSOL: AN INTERIOR POINT SOLVER FOR NONCONVEX OPTIMIZATION PROBLEMS
IPSOL: AN INTERIOR POINT SOLVER FOR NONCONVEX OPTIMIZATION PROBLEMS A DISSERTATION SUBMITTED TO THE PROGRAM IN SCIENTIFIC COMPUTING AND COMPUTATIONAL MATHEMATICS AND THE COMMITTEE ON GRADUATE STUDIES OF
More informationEffective reformulations of the truss topology design problem
Effective reformulations of the truss topology design problem Michal Kočvara and Jiří V. Outrata Abstract We present a new formulation of the truss topology problem that results in unique design and unique
More informationSome Recent Advances in Mixed-Integer Nonlinear Programming
Some Recent Advances in Mixed-Integer Nonlinear Programming Andreas Wächter IBM T.J. Watson Research Center Yorktown Heights, New York andreasw@us.ibm.com SIAM Conference on Optimization 2008 Boston, MA
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationAN INTERIOR-POINT METHOD FOR NONLINEAR OPTIMIZATION PROBLEMS WITH LOCATABLE AND SEPARABLE NONSMOOTHNESS
AN INTERIOR-POINT METHOD FOR NONLINEAR OPTIMIZATION PROBLEMS WITH LOCATABLE AND SEPARABLE NONSMOOTHNESS MARTIN SCHMIDT Abstract. Many real-world optimization models comse nonconvex and nonlinear as well
More informationSteering Exact Penalty Methods for Nonlinear Programming
Steering Exact Penalty Methods for Nonlinear Programming Richard H. Byrd Jorge Nocedal Richard A. Waltz April 10, 2007 Technical Report Optimization Technology Center Northwestern University Evanston,
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationSparse Optimization Lecture: Basic Sparse Optimization Models
Sparse Optimization Lecture: Basic Sparse Optimization Models Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know basic l 1, l 2,1, and nuclear-norm
More informationLinear programming II
Linear programming II Review: LP problem 1/33 The standard form of LP problem is (primal problem): max z = cx s.t. Ax b, x 0 The corresponding dual problem is: min b T y s.t. A T y c T, y 0 Strong Duality
More informationPattern Classification, and Quadratic Problems
Pattern Classification, and Quadratic Problems (Robert M. Freund) March 3, 24 c 24 Massachusetts Institute of Technology. 1 1 Overview Pattern Classification, Linear Classifiers, and Quadratic Optimization
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More information