Numerical Optimization: Basic Concepts and Algorithms
|
|
- Laura French
- 5 years ago
- Views:
Transcription
1 May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1
2 Outline Some basic concepts in optimization Some classical descent algorithms Some (less classical) semi-deterministic approaches Illustrations on various analytical problems Constrained optimality Some algorithm to account for constraints R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 2
3 Some basic concepts R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 3
4 Problem description Definition of a single-criterion parametric problem with real unknown Minimize f (x) x R n cost fonction Submitted to g i (x) = 0 i = 1,, l equality constraints h j (x) 0 j = 1,, m inequality constraints What does your cost function look like? Convex problem Multi-modal problem Noisy problem R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 4
5 Some commonly used algorithms Descent methods : adapted to convex cost functions steepest descent, conjugate gradient, quasi-newton, Newton, etc. Evolutionary methods : adapted to multi-modal cost functions genetic algorithms, evolution strategies, particle swarm, ant colony, simulated annealing, etc. Pattern search methods : adapted to noisy cost functions Nelder-Mead simplex, Torczon s multidirectional search, etc. R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 5
6 Optimality conditions Definition of a minimum x is a minimum of f : R n R if and only if there exists ρ > 0 such as: f defined on B(x, ρ) f (x ) < f (y) y B(x, ρ) y x not very useful to build algorithms... Characterization A sufficient condition for x to be a minimum is (if f twice differentiable): f (x ) = 0 (stationarity of gradient vector) 2 f (x ) > 0 (Hessian matrix positive definite) R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 6
7 Some classical descent algorithms R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 7
8 Descent methods Model algorithm For each iteration k (starting from x k ): Evaluate gradient f (x k ) Define a search direction d k ( f (x k )) Line search : choice of step length ρ k Update: x k+1 = x k + ρ k d k R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 8
9 Choice of the search direction Steepest-descent method: d k = f (x k ) Descent condition ensured : f (x k ) d k = f (x k ) f (x k ) < 0 But this yields an oscillatory path: d k+1 d k = ( f (x k+1 )) d k = 0 (if exact line search) Linear convergence rate: lim k x k+1 x x k x = a > 0 Illustration of steepest-descent path R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 9
10 Choice of the search direction quasi-newton method d k = H 1 k f (x k ) où H k approximate of the Hessian matrix 2 f (x k ) H should fulfill the following conditions: Symmetry Positive definite: f (x k ) d k = f (x k ) H 1 f (x k ) < 0 1D approximation of the curvature: H k+1 (x k+1 x k ) = f (x k+1 ) f (x k ) Ex : BFGS method H k+1 = H k 1 H s k T H k s k s k s T k k HT k + 1 y y k T s k yk T k où s k = x k+1 x k et y k = f (x k+1 ) f (x k ) Illustration of quasi-newton method Super-linear convergence rate : lim k x k+1 x x k x = 0 R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 10
11 Choice of the step length A classical criterion to ensure convergence : Armijo-Goldstein f (x k + ρ k d k ) < f (x k ) + α f (x k ) ρ k d k (Armijo) f (x k + ρ k d k ) > f (x k ) + β f (x k ) ρ k d k (Goldstein) Illustration of Armijo-Goldstein criterion R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 11
12 Choice of the step length An other criterion to ensure convergence (gradient required) : Armijo-Wolfe f (x k + ρ k d k ) < f (x k ) + α f (x k ) ρ k d k (Armijo) f (x k + ρ k d k ) d k > β f (x k ) d k (Wolfe) Illustration of Armijo-Wolfe criterion R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 12
13 Choice of the step length The step length is determined using an iterative 1D search: Start from an initial guess ρ (p) k (p = 0) Update to ρ (p+1) k : Bisection method Polynomial interpolation... until stopping criteria are fulfilled A balance is necessary between the computational cost and the accuracy R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 13
14 Some (less classical) semi-deterministic approaches R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 14
15 Evolutionary algorithms Principles Inspired by Darwinian theory of evolution : A population is composed of individuals who have different characteristics Most fitted individuals can survive and reproduce An offspring population is generated from survivors Mechanisms to improve progressively the population performance! R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 15
16 Evolution strategies Model algorithm (λ, µ)-es At each iteration k, a population is characterized by its mean x k and its variance σ 2 k. Generation of population k + 1 : Generation of λ perturbation amplitudes σ i = σ k e τ N(0,1) Generation of λ new individuals x i = x k + σ i N(0, Id) (mutation) with N(0, Id) multi-variate normal distribution Evaluation of the fitness of the λ individuals Choice of µ survivors among the λ new individuals (selection) Update of the population characteristics (crossover et self-adaptation) : µ µ x k+1 = 1 x i σ k+1 = 1 σ i µ µ i=1 i=1 R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 16
17 Evolution strategy Some results Proof of convergence towards the global optimum in a statistical sense : ɛ > 0 lim k P( f ( x k ) f (x ) ɛ) = 1 Linear convergence rate Capability to avoid local optima Limited to a rather small number of parameters (O(10)) Illustration of evolution strategy step R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 17
18 Evolution strategies Method CMA-ES (Covariance Matrix Adaption) Imprvement of ES algorithm by using an anisotropic distribution offspring population is generated using a covariance matrix C k : x i = x k + σ k N(0, C k ) = x k + σ k B k D k N(0, Id) avec B k matrix of eigenvectors of C 1/2 k et D k eigenvalues matrix Iterative construction of the covariance matrix: C 0 = Id C k+1 = (1 c)c k + c }{{} m p kpk T }{{} previous estimation 1D update p k evolution path (last moves) et y i = (x i x k )/σ k + c(1 1 µ m ) ω i (y i )(y i ) T with : } i=1 {{ } covariance of parents R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 18
19 Some illustrations using analytical functions R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 19
20 Rosenbrock function Non-convex unimodal function "Banana valley" Dimension n = 16 R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 20
21 Rosenbrock function Steepest descent Quasi-Newton R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 21
22 Rosenbrock function ES CMA-ES R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 22
23 Camelback function Dimension n = 2 Six local minima Two global minima R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 23
24 Camelback function Quasi-Newton Optimization path R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 24
25 Camelback function ES Optimization path R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 25
26 Constrained optimality R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 26
27 Introduction Necessity of constraints Often required to define a well-posed problem from mathematical point of view (existence, unicity) Often required to define a problem that make sense from industrial point of view (manufacturing) Different types of constraints Equality / inequality constraints Linear / non-linear constraints R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 27
28 Linear contraints Optimality conditions A sufficient condition for x to be a minimum of f subject to A x = b : A x = b (admissibility) f (x ) = λ A with λ Lagrange multipliers (stationnarity) A 2 f (x ) A > 0 (projected Hessian positive definite) Illustration of optimality conditions for linear constraints R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 28
29 Linear constraints Projection algorithm for descent methods At each iteration k, from an admissible point x k : Evaluation of gradient f (x k ) Choice of an admissible search direction Z d k with Z a projection matrix (in the admissible space: A Z = 0) Line search: choice of step length ρ k Update : x k+1 = x k + ρ k Z d k R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 29
30 Non-linear constraints Optimality conditions A sufficient condition for x to be a minimum of f subject to c(x) = 0 : c(x ) = 0 (admissibility) f (x ) = λ A(x ) with A(x) = c(x) (stationnarity) A(x ) 2 L(x, λ ) A(x ) > 0 with L(x, λ) = f (x) λ c(x) (projected Lagrangian positive definite) Illustration of optimality conditions for non-linear constraints R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 30
31 Non-linear constraints Quadratic penalization algorithm Cost function with penalization: f q(x, κ) = f (x) + κ c(x) c(x) 2 It can be shown that: lim κ x (κ) = x Algorithm with quadratic penalization: Initialisation of κ Minimisation of f q(x, κ) Increase κ to reduce constraint violation Illustration of quadratic penalization R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 31
32 Non-linear constraints Absolute penalization algorithm Cost function with penalization: f a(x, κ) = f (x) + κ c(x) It can be shown that: κ such that x (κ) = x κ > κ Algorithm with absolute penalization : Initialisation of κ Minimisation of f a(x, κ) Increase κ until constraint satisfied Illustration of absolute penalization R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 32
33 Non linear constraints Optimality condition in terms of Lagrangian L(x, λ) = f (x) λ c(x) λ L(x, λ ) = 0 (admissibility) x L(x, λ ) = 0 (stationnarity) A(x) 2 L(x, λ ) A(x) > 0 (positive-definite) SQP algorithm (Sequential Quadratic Programing) At each iteration k, Newton method applied to (x, λ): ( 2 f (x k ) λ k 2 ) c(x k ) A(x k ) A(x k ) 0 ( ) δx = δλ ( ) f (xk ) + λ k A(x k ) c(x k ) R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 33
34 Some references Classical methods G. N. Venderplaats. Numerical optimization techniques for engineering design. McGraw-Hill, R. Fletcher. Practical Methods of Optimization. John Wiley & Sons, P. E. Gill, W. Murray, and M. H. Wright. Practical Optimization. Academic Press, Evolutionary methods Z. Michalewics. Genetic algorithms + data structures = evolutionary programs. AI series. Springer-Verlag, New York, D. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison Wesley Company Inc., R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 34
8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationNumerical Optimization of Partial Differential Equations
Numerical Optimization of Partial Differential Equations Part I: basic optimization concepts in R n Bartosz Protas Department of Mathematics & Statistics McMaster University, Hamilton, Ontario, Canada
More informationOptimization and Root Finding. Kurt Hornik
Optimization and Root Finding Kurt Hornik Basics Root finding and unconstrained smooth optimization are closely related: Solving ƒ () = 0 can be accomplished via minimizing ƒ () 2 Slide 2 Basics Root finding
More informationOptimization Methods
Optimization Methods Decision making Examples: determining which ingredients and in what quantities to add to a mixture being made so that it will meet specifications on its composition allocating available
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationOutline. Scientific Computing: An Introductory Survey. Optimization. Optimization Problems. Examples: Optimization Problems
Outline Scientific Computing: An Introductory Survey Chapter 6 Optimization 1 Prof. Michael. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATH-GA.2043 or CSCI-GA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationNumerical Optimization. Review: Unconstrained Optimization
Numerical Optimization Finding the best feasible solution Edward P. Gatzke Department of Chemical Engineering University of South Carolina Ed Gatzke (USC CHE ) Numerical Optimization ECHE 589, Spring 2011
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationStatic unconstrained optimization
Static unconstrained optimization 2 In unconstrained optimization an objective function is minimized without any additional restriction on the decision variables, i.e. min f(x) x X ad (2.) with X ad R
More informationIntroduction to Optimization
Introduction to Optimization Blackbox Optimization Marc Toussaint U Stuttgart Blackbox Optimization The term is not really well defined I use it to express that only f(x) can be evaluated f(x) or 2 f(x)
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationContents. Preface. 1 Introduction Optimization view on mathematical models NLP models, black-box versus explicit expression 3
Contents Preface ix 1 Introduction 1 1.1 Optimization view on mathematical models 1 1.2 NLP models, black-box versus explicit expression 3 2 Mathematical modeling, cases 7 2.1 Introduction 7 2.2 Enclosing
More informationStatistics 580 Optimization Methods
Statistics 580 Optimization Methods Introduction Let fx be a given real-valued function on R p. The general optimization problem is to find an x ɛ R p at which fx attain a maximum or a minimum. It is of
More informationMultidisciplinary System Design Optimization (MSDO)
Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More informationECS550NFB Introduction to Numerical Methods using Matlab Day 2
ECS550NFB Introduction to Numerical Methods using Matlab Day 2 Lukas Laffers lukas.laffers@umb.sk Department of Mathematics, University of Matej Bel June 9, 2015 Today Root-finding: find x that solves
More informationMS&E 318 (CME 338) Large-Scale Numerical Optimization
Stanford University, Management Science & Engineering (and ICME) MS&E 318 (CME 338) Large-Scale Numerical Optimization 1 Origins Instructor: Michael Saunders Spring 2015 Notes 9: Augmented Lagrangian Methods
More informationA COMPARATIVE STUDY ON OPTIMIZATION METHODS FOR THE CONSTRAINED NONLINEAR PROGRAMMING PROBLEMS
A COMPARATIVE STUDY ON OPTIMIZATION METHODS FOR THE CONSTRAINED NONLINEAR PROGRAMMING PROBLEMS OZGUR YENIAY Received 2 August 2004 and in revised form 12 November 2004 Constrained nonlinear programming
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More information1 Numerical optimization
Contents 1 Numerical optimization 5 1.1 Optimization of single-variable functions............ 5 1.1.1 Golden Section Search................... 6 1.1. Fibonacci Search...................... 8 1. Algorithms
More informationBio-inspired Continuous Optimization: The Coming of Age
Bio-inspired Continuous Optimization: The Coming of Age Anne Auger Nikolaus Hansen Nikolas Mauny Raymond Ros Marc Schoenauer TAO Team, INRIA Futurs, FRANCE http://tao.lri.fr First.Last@inria.fr CEC 27,
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More information1 Numerical optimization
Contents Numerical optimization 5. Optimization of single-variable functions.............................. 5.. Golden Section Search..................................... 6.. Fibonacci Search........................................
More informationOptimization Methods for Circuit Design
Technische Universität München Department of Electrical Engineering and Information Technology Institute for Electronic Design Automation Optimization Methods for Circuit Design Compendium H. Graeb Version
More informationCondensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C.
Condensed Table of Contents for Introduction to Stochastic Search and Optimization: Estimation, Simulation, and Control by J. C. Spall John Wiley and Sons, Inc., 2003 Preface... xiii 1. Stochastic Search
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationAn Inexact Newton Method for Nonlinear Constrained Optimization
An Inexact Newton Method for Nonlinear Constrained Optimization Frank E. Curtis Numerical Analysis Seminar, January 23, 2009 Outline Motivation and background Algorithm development and theoretical results
More informationAM 205: lecture 19. Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality Today: Newton s method for optimization, survey of optimization methods Optimality Conditions: Equality Constrained Case As another example of equality
More informationFALL 2018 MATH 4211/6211 Optimization Homework 4
FALL 2018 MATH 4211/6211 Optimization Homework 4 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationOptimization: an Overview
Optimization: an Overview Moritz Diehl University of Freiburg and University of Leuven (some slide material was provided by W. Bangerth and K. Mombaur) Overview of presentation Optimization: basic definitions
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationScientific Computing: An Introductory Survey
Scientific Computing: An Introductory Survey Chapter 6 Optimization Prof. Michael T. Heath Department of Computer Science University of Illinois at Urbana-Champaign Copyright c 2002. Reproduction permitted
More information5 Quasi-Newton Methods
Unconstrained Convex Optimization 26 5 Quasi-Newton Methods If the Hessian is unavailable... Notation: H = Hessian matrix. B is the approximation of H. C is the approximation of H 1. Problem: Solve min
More informationChapter 4. Unconstrained optimization
Chapter 4. Unconstrained optimization Version: 28-10-2012 Material: (for details see) Chapter 11 in [FKS] (pp.251-276) A reference e.g. L.11.2 refers to the corresponding Lemma in the book [FKS] PDF-file
More informationImage restoration: numerical optimisation
Image restoration: numerical optimisation Short and partial presentation Jean-François Giovannelli Groupe Signal Image Laboratoire de l Intégration du Matériau au Système Univ. Bordeaux CNRS BINP / 6 Context
More informationUnconstrained Multivariate Optimization
Unconstrained Multivariate Optimization Multivariate optimization means optimization of a scalar function of a several variables: and has the general form: y = () min ( ) where () is a nonlinear scalar-valued
More informationOptimization Methods
Optimization Methods Categorization of Optimization Problems Continuous Optimization Discrete Optimization Combinatorial Optimization Variational Optimization Common Optimization Concepts in Computer Vision
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationThe Squared Slacks Transformation in Nonlinear Programming
Technical Report No. n + P. Armand D. Orban The Squared Slacks Transformation in Nonlinear Programming August 29, 2007 Abstract. We recall the use of squared slacks used to transform inequality constraints
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationOptimization Problems with Constraints - introduction to theory, numerical Methods and applications
Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationAM 205: lecture 19. Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods
AM 205: lecture 19 Last time: Conditions for optimality, Newton s method for optimization Today: survey of optimization methods Quasi-Newton Methods General form of quasi-newton methods: x k+1 = x k α
More informationYou should be able to...
Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set
More informationExamination paper for TMA4180 Optimization I
Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationIntroduction to Black-Box Optimization in Continuous Search Spaces. Definitions, Examples, Difficulties
1 Introduction to Black-Box Optimization in Continuous Search Spaces Definitions, Examples, Difficulties Tutorial: Evolution Strategies and CMA-ES (Covariance Matrix Adaptation) Anne Auger & Nikolaus Hansen
More informationMachine Learning CS 4900/5900. Lecture 03. Razvan C. Bunescu School of Electrical Engineering and Computer Science
Machine Learning CS 4900/5900 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Machine Learning is Optimization Parametric ML involves minimizing an objective function
More informationPDE-Constrained and Nonsmooth Optimization
Frank E. Curtis October 1, 2009 Outline PDE-Constrained Optimization Introduction Newton s method Inexactness Results Summary and future work Nonsmooth Optimization Sequential quadratic programming (SQP)
More informationCE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review
CE 191: Civil & Environmental Engineering Systems Analysis LEC 17 : Final Review Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 2014 Prof. Moura UC Berkeley
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More information2.3 Linear Programming
2.3 Linear Programming Linear Programming (LP) is the term used to define a wide range of optimization problems in which the objective function is linear in the unknown variables and the constraints are
More informationOptimization: Nonlinear Optimization without Constraints. Nonlinear Optimization without Constraints 1 / 23
Optimization: Nonlinear Optimization without Constraints Nonlinear Optimization without Constraints 1 / 23 Nonlinear optimization without constraints Unconstrained minimization min x f(x) where f(x) is
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationKrzysztof Tesch. Continuous optimisation algorithms
Krzysztof Tesch Continuous optimisation algorithms Gdańsk 16 GDAŃSK UNIVERSITY OF TECHNOLOGY PUBLISHERS CHAIRMAN OF EDITORIAL BOARD Janusz T. Cieśliński REVIEWER Krzysztof Kosowski COVER DESIGN Katarzyna
More informationConstrained Nonlinear Optimization Algorithms
Department of Industrial Engineering and Management Sciences Northwestern University waechter@iems.northwestern.edu Institute for Mathematics and its Applications University of Minnesota August 4, 2016
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationLINEAR AND NONLINEAR PROGRAMMING
LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGraw-Hill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico
More informationOptimization Concepts and Applications in Engineering
Optimization Concepts and Applications in Engineering Ashok D. Belegundu, Ph.D. Department of Mechanical Engineering The Pennsylvania State University University Park, Pennsylvania Tirupathi R. Chandrupatia,
More informationIntroduction to Scientific Computing
Introduction to Scientific Computing Benson Muite benson.muite@ut.ee http://kodu.ut.ee/ benson https://courses.cs.ut.ee/2018/isc/spring 26 March 2018 [Public Domain,https://commons.wikimedia.org/wiki/File1
More informationHomework 5. Convex Optimization /36-725
Homework 5 Convex Optimization 10-725/36-725 Due Tuesday November 22 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationThree Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms
Three Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms Yong Wang and Zhi-Zhong Liu School of Information Science and Engineering Central South University ywang@csu.edu.cn
More informationNONLINEAR. (Hillier & Lieberman Introduction to Operations Research, 8 th edition)
NONLINEAR PROGRAMMING (Hillier & Lieberman Introduction to Operations Research, 8 th edition) Nonlinear Programming g Linear programming has a fundamental role in OR. In linear programming all its functions
More informationDetermination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms: A Comparative Study
International Journal of Mathematics And Its Applications Vol.2 No.4 (2014), pp.47-56. ISSN: 2347-1557(online) Determination of Feasible Directions by Successive Quadratic Programming and Zoutendijk Algorithms:
More informationCMA-ES a Stochastic Second-Order Method for Function-Value Free Numerical Optimization
CMA-ES a Stochastic Second-Order Method for Function-Value Free Numerical Optimization Nikolaus Hansen INRIA, Research Centre Saclay Machine Learning and Optimization Team, TAO Univ. Paris-Sud, LRI MSRC
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 15: Nonlinear optimization Prof. John Gunnar Carlsson November 1, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 1, 2010 1 / 24
More information, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues
More informationMath 409/509 (Spring 2011)
Math 409/509 (Spring 2011) Instructor: Emre Mengi Study Guide for Homework 2 This homework concerns the root-finding problem and line-search algorithms for unconstrained optimization. Please don t hesitate
More informationScientific Data Computing: Lecture 3
Scientific Data Computing: Lecture 3 Benson Muite benson.muite@ut.ee 23 April 2018 Outline Monday 10-12, Liivi 2-207 Monday 12-14, Liivi 2-205 Topics Introduction, statistical methods and their applications
More informationLecture V. Numerical Optimization
Lecture V Numerical Optimization Gianluca Violante New York University Quantitative Macroeconomics G. Violante, Numerical Optimization p. 1 /19 Isomorphism I We describe minimization problems: to maximize
More informationNumerical Analysis of Electromagnetic Fields
Pei-bai Zhou Numerical Analysis of Electromagnetic Fields With 157 Figures Springer-Verlag Berlin Heidelberg New York London Paris Tokyo Hong Kong Barcelona Budapest Contents Part 1 Universal Concepts
More informationConstrained optimization. Unconstrained optimization. One-dimensional. Multi-dimensional. Newton with equality constraints. Active-set method.
Optimization Unconstrained optimization One-dimensional Multi-dimensional Newton s method Basic Newton Gauss- Newton Quasi- Newton Descent methods Gradient descent Conjugate gradient Constrained optimization
More informationThree Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms
Three Steps toward Tuning the Coordinate Systems in Nature-Inspired Optimization Algorithms Yong Wang and Zhi-Zhong Liu School of Information Science and Engineering Central South University ywang@csu.edu.cn
More informationIPAM Summer School Optimization methods for machine learning. Jorge Nocedal
IPAM Summer School 2012 Tutorial on Optimization methods for machine learning Jorge Nocedal Northwestern University Overview 1. We discuss some characteristics of optimization problems arising in deep
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationStochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions
International Journal of Control Vol. 00, No. 00, January 2007, 1 10 Stochastic Optimization with Inequality Constraints Using Simultaneous Perturbations and Penalty Functions I-JENG WANG and JAMES C.
More informationOptimization II: Unconstrained Multivariable
Optimization II: Unconstrained Multivariable CS 205A: Mathematical Methods for Robotics, Vision, and Graphics Doug James (and Justin Solomon) CS 205A: Mathematical Methods Optimization II: Unconstrained
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationMinimization of Static! Cost Functions!
Minimization of Static Cost Functions Robert Stengel Optimal Control and Estimation, MAE 546, Princeton University, 2017 J = Static cost function with constant control parameter vector, u Conditions for
More informationQuasi-Newton Methods
Quasi-Newton Methods Werner C. Rheinboldt These are excerpts of material relating to the boos [OR00 and [Rhe98 and of write-ups prepared for courses held at the University of Pittsburgh. Some further references
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationOptimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization
5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The
More informationA Primer on Multidimensional Optimization
A Primer on Multidimensional Optimization Prof. Dr. Florian Rupp German University of Technology in Oman (GUtech) Introduction to Numerical Methods for ENG & CS (Mathematics IV) Spring Term 2016 Eercise
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationAn Inexact Newton Method for Optimization
New York University Brown Applied Mathematics Seminar, February 10, 2009 Brief biography New York State College of William and Mary (B.S.) Northwestern University (M.S. & Ph.D.) Courant Institute (Postdoc)
More informationIntroduction to unconstrained optimization - direct search methods
Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the
More informationLecture 4: Optimization. Maximizing a function of a single variable
Lecture 4: Optimization Maximizing or Minimizing a Function of a Single Variable Maximizing or Minimizing a Function of Many Variables Constrained Optimization Maximizing a function of a single variable
More information