Optimization Methods. Lecture 18: Optimality Conditions and. Gradient Methods. for Unconstrained Optimization
|
|
- Jeremy Tate
- 5 years ago
- Views:
Transcription
1 5.93 Optimization Methods Lecture 8: Optimality Conditions and Gradient Methods for Unconstrained Optimization
2 Outline. Necessary and sucient optimality conditions Slide. Gradient m e t h o d s 3. The steepest descent algorithm 4. Rate of convergence 5. Line search algorithms Last Lecture Nonlinear Optimization Applications Portfolio Selection Slide Facility Location (Geometry Problems) Trac Assignment, Routing 3 The general problem f : < n 7! < is a continuous (usually dierentiable) function of n variables Slide 3 g i : < n 7! < i = : : : m h j : < n 7! < j = : : : l N L P : min f (x) s.t. g (x). g m (x) h (x) =. h l (x) = 3. Local vs Global Minima Slide 4 x F is a local minimum of N LP if there exists > su ch that f (x) f (y) for all y B(x ) \ F x F is a global minimum of N LP if f (x) f (y) for all y F.
3 4 Convex Sets and Functions n A subset S < is a convex set if Slide 5 x y S ) x + ( ; )y S 8 [ ] A function f (x) is a convex function if f (x + ( ; )y) f (x) + ( ; )f (y) 8x y 8 [ ] 5 Convex Optimization 5. Convexity and Minima COP is called a convex optimization problem if f (x) g (x) : : : g m(x) are convex functions This implies that the objective function is convex and the feasible region F is a convex set. Implication: If COP i s a c o n vex optimization problem, then any local minimum will be a global minimum. 6 Optimality Conditions Necessary Conds for Local Optima \If x is local optimum then x must satisfy..." Slide 6 Slide 7 Identies all candidates for local optima. Sucient Conds for Local Optima \If x satises...,then x must be a local optimum " 7 Optimality Conditions 7. Necessary conditions Consider min f (x) x< n Slide 8
4 Zero rst order variation along all directions Theorem Let f (x) b e c o n tinuously dierentiable. If x < n is a local minimum of f (x), then rf (x ) = and r f (x ) P SD 7. Proof Zero slope at local min x f (x ) f (x + d) for all d < n < Slide 9 Pic > f (x + d) ; f (x ) Tae limits as! r f (x ) n d 8d < Since d arbitrary, replace with ;d ) r f (x ) = : Nonnegative curvature at a local min x ) f (x + d) ; f (x = rf (x ) (d) + (d) r f (x )(d) + jjdjj R(x d) where R(x y)! as y!. Since rf (x ), = Slide = d r f (x )d + jjdjj R(x d) ) f (x + d) ; f (x ) d = r f (x )d + jjdjj R(x d) If r f (x ) is not PSD, 9d : d r f (x )d < ) f (x + d ) < f ( 8 x), su. QED. small 7.3 Example f (x) = x + x :x + x ; 4x ; 4x ; x 3 rf (x) = ( x + x ; 4 x + 4 x ; 4 ; 3x ) Candidates x = (4 ) and x = (3 ) r f (x) = 4 ; 6x r f (x ) = 4 PSD x = (3 ) r f ( x) = ; Indenite matrix x is the only candidate for local min Slide Slide 3
5 7.4 Sucient conditions Theorem f twice continuously dierentiable. If rf (x ) = and r f (x) P S D in B(x ), then x is a local minimum. Proof: Taylor series expansion: For all x B(x ) Slide 3 for some [ ] f (x) = f (x ) + rf (x ) (x ; x ) + (x ; x ) r f (x + (x ; x ))(x ; x ) ) f (x) f (x ) 7.5 Example Continued... Slide 4 At x = ( 4 ), rf (x ) = a n d r f (x) = 4 ; 6x is PSD for x B(x ) Slide 5 3 f (x) = x + x and rf (x) = ( 3 x x ) x = ( ) r 6x f (x) = is not PSD in B( ) f (; ) = ; 3 < = f (x ) 7.6 Characterization of convex functions Theorem Let f (x) b e c o n tinuously dierentiable. Then f (x) i s c o n vex if and only if Slide 6 rf (x) (x ; x) f (x) ; f (x) 7.7 Proof By convexity As!, f (x + ( ; )x) f (x) + ( ; )f (x) f (x + (x ; x)) ; f (x) f (x) ; f (x) rf (x) (x ; x) f (x) ; f (x) Slide 7 4
6 If ( rf x) d <, then for small, > 7.8 Convex functions Theorem Let f (x) b e a c o n tinuously dierentiable convex function. Then x is a m in im um of f if and only if Slide 8 Proof: If f convex and rf (x ) = rf (x ) = ) f (x) ; f (x ) r f (x (x ; x ) = 7.9 Descent Directions Interesting Observation Slide 9 f di/ble at x 9d: rf ( x) d < ) 8 >, su. small, f ( x + d) < f ( x) (d: descent direction) 7. Proof where R( x d) ;!! f ( x + d) = f ( x) + rf ( x) t d + jjdjjr( x d ) Slide f ( x + d) ; f ( x) = rf ( x) t d + jjdjjr( x d ) x) rf ( t d <, R( x d) ;!! ) 8 > su. sm all f ( x + d) < f ( x). QED 8 Algorithms for unconstrained optimization 8. Gradient Methods-Motivation Slide Decrease f (x) until rf (x ) = f ( x + d) f ( x) + rf x) ( d x + d) < ( f ( f x) 5
7 Principal example: 9 Gradient Methods 9. A generic algorithm + x = x + d If rf (x ) 6=, direction d satises: rf (x ) d < Slide Step-length > D + = x x ; D rf (x ) positive denite symmetric matrix 9. Principal directions Steepest descent: Newton's method: 9.3 Other directions Diagonally scaled steepest descent + x = x ; rf (x ) x = x ; (r f (x )) ; rf (x ) + D = Diagonal approximation to (r f (x )) ; Slide 3 Slide 4 Modied Newton's method D = Diagonal approximation to (r f (x )) ; = Gauss-Newton method for least squares problems f (x) = jjg(x)jj D (rg(x )rg(x ) ) Steepest descent. The algorithm Step Given x, set :=. Step d := ;rf (x ). If jjd jj, then stop. Slide 5 Step Solve min h() := f (x + d ) for the step-length, perhaps chosen by an exact or inexact line-search. + Step 3 Set x x + d,. +. Go to Step 6
8 . An example minimize f (x x ) = 5 x + x + 4 x x ; 4x ; 6x + x = ( x x ) = ( ) Slide 6 Given x f (x ) = Slide 7 d d ;x = ;rf(x x ) = ; 4x + 4 = ;x d ; 4x + 6 ) h() = f (x + d = 5(x ) + ( x + d )(x + d + d ) + 4( x + d ); ;4(x + d ) ; 6(x + d ) + (d ) + ( d ) = Slide 8 (5(d ) + ( d ) + 4 d d ) Start at x = ( ) ;6 " = x x d d jjd jj f (x ) : : ;6: ;4: 9:59646 :866 6: ;:578 8: : ;:56798 :9734 :8 :576 3 : :64 ;6: ;3:43 7: :866 : :485 :93535 : ;:6648 :7583 :8 : :9443 :53789 ;:55367 ;:83659 : :866 : :8565 :4653 :846 ;:5344 :73934 :8 : :98539 :3468 ;: ;:456 : :866 :669 8 :95485 :3749 :58 ;:37436 :45845 :8 :68 9 :99649 :338 ;:984 ;:4999 : :866 :638 : :786 :498 ;:95 :3937 :8 :56 x x d d jjd jj f (x ) :9997 :7856 ;:695 ;: : :866 :38 :9976 :6797 :5 ;:37 :5476 :8 :9 3 : :9 ;:5548 ;:987 :637 :866 : 4 :99936 :66 :94 ;:547 :69 :8 : 5 : :469 ;:356 ;:73 :543 :866 : 6 :99983 :46 :7 ;:34 :583 :8 : 7 : :5 ;:33 ;:79 :37653 :866 : 8 : :99 :8 ;:33 :37 :8 : 9 : :8 ;:8 ;:44 :94 :866 : :99999 :4 :4 ;:8 :97 :83 : : :7 ;: ;: :5 :866 : : :6 : ;: : :87 : 3 : : ;:5 ;:3 :55 :866 : 4 : : : ;: :54 : : Slide 9 Slide 3 7
9 5 x +4 y +3 x y+7 x y 5 5 x 5 Slide Important Properties f(x + ) <f(x ) < <f(x ) (because d are descent directions) Slide 3 Under reasonable assumptions of f(x), the sequence x x ::: will have at least one cluster point x Every cluster point x will satisfy rf(x) = Implication: If f(x) is a convex function, x will be an optimal solution 8
10 Global Convergence Result Theorem: f : R n! R is continuously di/ble on F = fx R n : f (x) f (x )g closed, bo unded set Every cluster point x of fx g satises rf ( x) =.. Wor Per Iteration Two computation tass at each iteration of steepest descent: Slide 33 Slide 34 Compute rf (x ) (for quadratic objective functions, it taes O(n ) steps) to determine d = ;rf (x ) Perform line-search o f h() = f (x + d ) to determine = arg min h() = arg min f (x + d ) Rate of convergence of algorithms Let z : : : z n : : :! z be a c o n vergent sequence. We s a y that the order of convergence of this sequence is p if jz + ; zj p = sup p : lim sup <! jz ; zj p Let jz + ; zj = lim sup jz ; zj p! The larger p, the faster the convergence. Types of convergence. p = < <, then linear (or geometric) rate of convergence Slide 35 Slide 36. p = =, super-linear convergence 3. p = =, sub-linear convergence 4. p = quadratic convergence 9
11 . Examples z = a < a < c o n verges linearly to zero, = a Slide 37 z = a < a < c o n verges quadratically to zero z = converges sub-linearly to zero z = converges super-linearly to zero.3 Steepest descent z = f (x ), z = f (x ), where x = arg min f (x) Slide 38 Then an algorithm exhibits linear convergence if there is a constant < such that + f (x ) ; f (x ) f (x ) ; f (x, ) for all suciently large, where x is an optimal solution..3. Discussion + f (x ) ; f (x ) < f (x ) ; f (x ) Slide 39 optimal alue. If = :, every iteration adds another digit of accuracy to the objective v If = :9, every iterations add another digit of accuracy to the optimal objective v alue, because (:9) :. 3 Rate of convergence of steepest descent 3. Quadratic Case 3.. Theorem Suppose f (x) = Q is psd x Qx ; c x Slide 4
12 max = largest eigenvalue of Q min = smallest eigenvalues Q of Linear Convergence T h e orem : If f(x) is a quadratic function and Q is psd, then + f(x ) ; f(x max ) ; A f(x ) ; f(x ) max min Discussion f(x + ) ; f(x max ) min A f(x ) ; f(x ) max min + Slide 4 (Q) := (Q) max is the condition number of Q min (Q) plays an extremely important role in analyzing computation involving Q (Q) = max min + f(x ) ; f(x ) (Q) ; f(x ) ; f(x ) (Q) + Upper Bound on Number of Iterations to Reduce Convergence Constant the Optimality Gap by For (Q) O() converges fast. For large (Q) Slide 4 Slide 43 Therefore In (Q) ; ) ( ; ; (Q) + (Q) (Q) (f(x ) ; f(x )) ( ; ) (f(x ) ; f(x )) (Q) (Q)(;ln ) iterations, nds x : (f(x ) ; f(x )) (f(x ) ; f(x ))
13 6 3. Example (Q) = 3 :34 (Q); = (Q)+ = :876 f (x) = x Qx ; c x Q = c = 5 Slide 44 Slide x 4: 5: : :7635 7:47 :986 : :844 7: :684 x ;: ;99:6967 ;64:6683 ;64: ;4:4698 ;4:8563 ;6:5894 ;6:9455 ;6:46575 ;5:98969 jjd jj 86: : : : : : : :437 53: : :56 :56 :56 :56 :56 f (x ) 65: 398:6958 6: : : : : : : :9596 f (x ) ; f (x ) f (x ; ) ; f (x) Slide 46 x : ;: ;:48 5 ;:36 6 ;: ;: ;: ;:33333 x : :3899 3:975 3:3885 3:3378 3: : :33333 jjd jj : :9698 : :338 :473 :55 :636 :78 : f (x ) 3:666 :96583 :93388 :93334 : : : : f (x ) ; f (x ) f (x ; ) ; f (x) :65878 :6585 : : Slide Example 3 f (x) = x Qx ; c x Q = c = Slide 48
14 3 (Q) = :854 (Q) ; = = :896 (Q) + Slide x 4: 9:8678 :534 : :74549 :736 :66755 : :6578 :6577 : :65763 :65768 :65767 :65767 x ;: ;:56 ;4:5558 :35 ;:53347 :66834 :5898 :69366 :68996 :69486 :69468 :6949 :6949 :6949 :6949 jjd jj 434: : : :445 3:98573 : : :44973 :764 :99 :3349 :99 :58 :45 :75 :74 :459 :74 :459 :74 :459 :74 :459 :74 :459 :74 :459 :74 :459 : f (x ) 765: 359: :5893 :8678 5: : : : : : : : : : :88836 f (x ) ; f (x ) f (x ; ) ; f (x) :4766 :4766 :4766 :4766 :4766 :4766 :4766 :4766 :4766 :4766 :476 :4768 :45 : Slide Empirical behavior The convergence constant bound is not just theoretical. It is typically experienced in practice. Slide 5 Analysis is due to Leonid Kantorovich, who won the Nobel Memorial Prize in Economic Science in 975 for his contributions to optimization and economic planning. Slide 5 What about non-quadratic functions? { Suppose x = a r g m i n x f (x)
15 { r f (x ) is the Hessian of f (x) at x = x { Rate of convergence will depend on (r f (x )) 4 Summary. Optimality Conditions Slide 53. The steepest descent algorithm - Convergence 3. Rate of convergence of Steepest Descent 5 Choices of step sizes M in f (x + d ) Slide 54 Limited Minimization: M in [ s]f (x + d ) Constant stepsize constant = s P Diminishing stepsize:! = Armijo Rule 4
Optimization Methods. Lecture 19: Line Searches and Newton s Method
15.93 Optimization Methods Lecture 19: Line Searches and Newton s Method 1 Last Lecture Necessary Conditions for Optimality (identifies candidates) x local min f(x ) =, f(x ) PSD Slide 1 Sufficient Conditions
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationYou should be able to...
Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set
More information1 Newton s Method. Suppose we want to solve: x R. At x = x, f (x) can be approximated by:
Newton s Method Suppose we want to solve: (P:) min f (x) At x = x, f (x) can be approximated by: n x R. f (x) h(x) := f ( x)+ f ( x) T (x x)+ (x x) t H ( x)(x x), 2 which is the quadratic Taylor expansion
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 14: Unconstrained optimization Prof. John Gunnar Carlsson October 27, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I October 27, 2010 1
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationNumerical optimization
Numerical optimization Lecture 4 Alexander & Michael Bronstein tosca.cs.technion.ac.il/book Numerical geometry of non-rigid shapes Stanford University, Winter 2009 2 Longest Slowest Shortest Minimal Maximal
More informationIntroduction to Nonlinear Optimization Paul J. Atzberger
Introduction to Nonlinear Optimization Paul J. Atzberger Comments should be sent to: atzberg@math.ucsb.edu Introduction We shall discuss in these notes a brief introduction to nonlinear optimization concepts,
More information6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection
6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods
More information5 Overview of algorithms for unconstrained optimization
IOE 59: NLP, Winter 22 c Marina A. Epelman 9 5 Overview of algorithms for unconstrained optimization 5. General optimization algorithm Recall: we are attempting to solve the problem (P) min f(x) s.t. x
More informationUnconstrained optimization
Chapter 4 Unconstrained optimization An unconstrained optimization problem takes the form min x Rnf(x) (4.1) for a target functional (also called objective function) f : R n R. In this chapter and throughout
More informationNumerical optimization. Numerical optimization. Longest Shortest where Maximal Minimal. Fastest. Largest. Optimization problems
1 Numerical optimization Alexander & Michael Bronstein, 2006-2009 Michael Bronstein, 2010 tosca.cs.technion.ac.il/book Numerical optimization 048921 Advanced topics in vision Processing and Analysis of
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More informationmin f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;
Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many
More informationARE202A, Fall Contents
ARE202A, Fall 2005 LECTURE #2: WED, NOV 6, 2005 PRINT DATE: NOVEMBER 2, 2005 (NPP2) Contents 5. Nonlinear Programming Problems and the Kuhn Tucker conditions (cont) 5.2. Necessary and sucient conditions
More information, b = 0. (2) 1 2 The eigenvectors of A corresponding to the eigenvalues λ 1 = 1, λ 2 = 3 are
Quadratic forms We consider the quadratic function f : R 2 R defined by f(x) = 2 xt Ax b T x with x = (x, x 2 ) T, () where A R 2 2 is symmetric and b R 2. We will see that, depending on the eigenvalues
More informationECE 680 Modern Automatic Control. Gradient and Newton s Methods A Review
ECE 680Modern Automatic Control p. 1/1 ECE 680 Modern Automatic Control Gradient and Newton s Methods A Review Stan Żak October 25, 2011 ECE 680Modern Automatic Control p. 2/1 Review of the Gradient Properties
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationECE580 Fall 2015 Solution to Midterm Exam 1 October 23, Please leave fractions as fractions, but simplify them, etc.
ECE580 Fall 2015 Solution to Midterm Exam 1 October 23, 2015 1 Name: Solution Score: /100 This exam is closed-book. You must show ALL of your work for full credit. Please read the questions carefully.
More informationN. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:
0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-3: Unconstrained optimization II Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationA New Trust Region Algorithm Using Radial Basis Function Models
A New Trust Region Algorithm Using Radial Basis Function Models Seppo Pulkkinen University of Turku Department of Mathematics July 14, 2010 Outline 1 Introduction 2 Background Taylor series approximations
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 15: Nonlinear optimization Prof. John Gunnar Carlsson November 1, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 1, 2010 1 / 24
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More informationLecture 3: Basics of set-constrained and unconstrained optimization
Lecture 3: Basics of set-constrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018 Optimization basics Outline Optimization
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More information154 ADVANCES IN NONLINEAR PROGRAMMING Abstract: We propose an algorithm for nonlinear optimization that employs both trust region techniques and line
7 COMBINING TRUST REGION AND LINE SEARCH TECHNIQUES Jorge Nocedal Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL 60208-3118, USA. Ya-xiang Yuan State Key Laboratory
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationGRADIENT = STEEPEST DESCENT
GRADIENT METHODS GRADIENT = STEEPEST DESCENT Convex Function Iso-contours gradient 0.5 0.4 4 2 0 8 0.3 0.2 0. 0 0. negative gradient 6 0.2 4 0.3 2.5 0.5 0 0.5 0.5 0 0.5 0.4 0.5.5 0.5 0 0.5 GRADIENT DESCENT
More informationNumerical Optimization Professor Horst Cerjak, Horst Bischof, Thomas Pock Mat Vis-Gra SS09
Numerical Optimization 1 Working Horse in Computer Vision Variational Methods Shape Analysis Machine Learning Markov Random Fields Geometry Common denominator: optimization problems 2 Overview of Methods
More informationGradient Descent. Dr. Xiaowei Huang
Gradient Descent Dr. Xiaowei Huang https://cgi.csc.liv.ac.uk/~xiaowei/ Up to now, Three machine learning algorithms: decision tree learning k-nn linear regression only optimization objectives are discussed,
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationUnconstrained Optimization
1 / 36 Unconstrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University February 2, 2015 2 / 36 3 / 36 4 / 36 5 / 36 1. preliminaries 1.1 local approximation
More informationLECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION
15-382 COLLECTIVE INTELLIGENCE - S19 LECTURE 22: SWARM INTELLIGENCE 3 / CLASSICAL OPTIMIZATION TEACHER: GIANNI A. DI CARO WHAT IF WE HAVE ONE SINGLE AGENT PSO leverages the presence of a swarm: the outcome
More informationLecture 7 Unconstrained nonlinear programming
Lecture 7 Unconstrained nonlinear programming Weinan E 1,2 and Tiejun Li 2 1 Department of Mathematics, Princeton University, weinan@princeton.edu 2 School of Mathematical Sciences, Peking University,
More informationCE 191: Civil and Environmental Engineering Systems Analysis. LEC 05 : Optimality Conditions
CE 191: Civil and Environmental Engineering Systems Analysis LEC : Optimality Conditions Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 214 Prof. Moura
More informationIE 5531: Engineering Optimization I
IE 5531: Engineering Optimization I Lecture 19: Midterm 2 Review Prof. John Gunnar Carlsson November 22, 2010 Prof. John Gunnar Carlsson IE 5531: Engineering Optimization I November 22, 2010 1 / 34 Administrivia
More informationNONLINEAR. (Hillier & Lieberman Introduction to Operations Research, 8 th edition)
NONLINEAR PROGRAMMING (Hillier & Lieberman Introduction to Operations Research, 8 th edition) Nonlinear Programming g Linear programming has a fundamental role in OR. In linear programming all its functions
More information4 Newton Method. Unconstrained Convex Optimization 21. H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion:
Unconstrained Convex Optimization 21 4 Newton Method H(x)p = f(x). Newton direction. Why? Recall second-order staylor series expansion: f(x + p) f(x)+p T f(x)+ 1 2 pt H(x)p ˆf(p) In general, ˆf(p) won
More informationMVE165/MMG631 Linear and integer optimization with applications Lecture 13 Overview of nonlinear programming. Ann-Brith Strömberg
MVE165/MMG631 Overview of nonlinear programming Ann-Brith Strömberg 2015 05 21 Areas of applications, examples (Ch. 9.1) Structural optimization Design of aircraft, ships, bridges, etc Decide on the material
More informationIPAM Summer School Optimization methods for machine learning. Jorge Nocedal
IPAM Summer School 2012 Tutorial on Optimization methods for machine learning Jorge Nocedal Northwestern University Overview 1. We discuss some characteristics of optimization problems arising in deep
More informationOptimization. Yuh-Jye Lee. March 21, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 29
Optimization Yuh-Jye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 21, 2017 1 / 29 You Have Learned (Unconstrained) Optimization in Your High School Let f (x) = ax
More informationMethods for Unconstrained Optimization Numerical Optimization Lectures 1-2
Methods for Unconstrained Optimization Numerical Optimization Lectures 1-2 Coralia Cartis, University of Oxford INFOMM CDT: Modelling, Analysis and Computation of Continuous Real-World Problems Methods
More information17 Solution of Nonlinear Systems
17 Solution of Nonlinear Systems We now discuss the solution of systems of nonlinear equations. An important ingredient will be the multivariate Taylor theorem. Theorem 17.1 Let D = {x 1, x 2,..., x m
More informationComplexity analysis of second-order algorithms based on line search for smooth nonconvex optimization
Complexity analysis of second-order algorithms based on line search for smooth nonconvex optimization Clément Royer - University of Wisconsin-Madison Joint work with Stephen J. Wright MOPTA, Bethlehem,
More information1. Introduction In this paper we study a new type of trust region method for solving the unconstrained optimization problem, min f(x): (1.1) x2< n Our
Combining Trust Region and Line Search Techniques Jorge Nocedal y and Ya-xiang Yuan z March 14, 1998 Report OTC 98/04 Optimization Technology Center Abstract We propose an algorithm for nonlinear optimization
More informationNumerical Optimization: Basic Concepts and Algorithms
May 27th 2015 Numerical Optimization: Basic Concepts and Algorithms R. Duvigneau R. Duvigneau - Numerical Optimization: Basic Concepts and Algorithms 1 Outline Some basic concepts in optimization Some
More informationOptimality conditions for unconstrained optimization. Outline
Optimality conditions for unconstrained optimization Daniel P. Robinson Department of Applied Mathematics and Statistics Johns Hopkins University September 13, 2018 Outline 1 The problem and definitions
More informationAM 205: lecture 18. Last time: optimization methods Today: conditions for optimality
AM 205: lecture 18 Last time: optimization methods Today: conditions for optimality Existence of Global Minimum For example: f (x, y) = x 2 + y 2 is coercive on R 2 (global min. at (0, 0)) f (x) = x 3
More informationOPER 627: Nonlinear Optimization Lecture 14: Mid-term Review
OPER 627: Nonlinear Optimization Lecture 14: Mid-term Review Department of Statistical Sciences and Operations Research Virginia Commonwealth University Oct 16, 2013 (Lecture 14) Nonlinear Optimization
More informationLecture 15 Newton Method and Self-Concordance. October 23, 2008
Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications
More informationUnconstrained optimization I Gradient-type methods
Unconstrained optimization I Gradient-type methods Antonio Frangioni Department of Computer Science University of Pisa www.di.unipi.it/~frangio frangio@di.unipi.it Computational Mathematics for Learning
More informationConvex Optimization. 9. Unconstrained minimization. Prof. Ying Cui. Department of Electrical Engineering Shanghai Jiao Tong University
Convex Optimization 9. Unconstrained minimization Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2017 Autumn Semester SJTU Ying Cui 1 / 40 Outline Unconstrained minimization
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationLecture Notes: Geometric Considerations in Unconstrained Optimization
Lecture Notes: Geometric Considerations in Unconstrained Optimization James T. Allison February 15, 2006 The primary objectives of this lecture on unconstrained optimization are to: Establish connections
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationCopositive Programming and Combinatorial Optimization
Copositive Programming and Combinatorial Optimization Franz Rendl http://www.math.uni-klu.ac.at Alpen-Adria-Universität Klagenfurt Austria joint work with I.M. Bomze (Wien) and F. Jarre (Düsseldorf) IMA
More informationPaul Schrimpf. October 18, UBC Economics 526. Unconstrained optimization. Paul Schrimpf. Notation and definitions. First order conditions
Unconstrained UBC Economics 526 October 18, 2013 .1.2.3.4.5 Section 1 Unconstrained problem x U R n F : U R. max F (x) x U Definition F = max x U F (x) is the maximum of F on U if F (x) F for all x U and
More informationNumerical Optimization
Numerical Optimization Unit 2: Multivariable optimization problems Che-Rung Lee Scribe: February 28, 2011 (UNIT 2) Numerical Optimization February 28, 2011 1 / 17 Partial derivative of a two variable function
More informationMethods that avoid calculating the Hessian. Nonlinear Optimization; Steepest Descent, Quasi-Newton. Steepest Descent
Nonlinear Optimization Steepest Descent and Niclas Börlin Department of Computing Science Umeå University niclas.borlin@cs.umu.se A disadvantage with the Newton method is that the Hessian has to be derived
More informationThe Conjugate Gradient Method
The Conjugate Gradient Method Lecture 5, Continuous Optimisation Oxford University Computing Laboratory, HT 2006 Notes by Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The notion of complexity (per iteration)
More informationA new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality constraints
Journal of Computational and Applied Mathematics 161 (003) 1 5 www.elsevier.com/locate/cam A new ane scaling interior point algorithm for nonlinear optimization subject to linear equality and inequality
More informationUnconstrained minimization: assumptions
Unconstrained minimization I terminology and assumptions I gradient descent method I steepest descent method I Newton s method I self-concordant functions I implementation IOE 611: Nonlinear Programming,
More information2 JOSE HERSKOVITS The rst stage to get an optimal design is to dene the Optimization Model. That is, to select appropriate design variables, an object
A VIEW ON NONLINEAR OPTIMIZATION JOSE HERSKOVITS Mechanical Engineering Program COPPE / Federal University of Rio de Janeiro 1 Caixa Postal 68503, 21945-970 Rio de Janeiro, BRAZIL 1. Introduction Once
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationChapter 2. Optimization. Gradients, convexity, and ALS
Chapter 2 Optimization Gradients, convexity, and ALS Contents Background Gradient descent Stochastic gradient descent Newton s method Alternating least squares KKT conditions 2 Motivation We can solve
More informationCalculus and optimization
Calculus an optimization These notes essentially correspon to mathematical appenix 2 in the text. 1 Functions of a single variable Now that we have e ne functions we turn our attention to calculus. A function
More informationMotivation Subgradient Method Stochastic Subgradient Method. Convex Optimization. Lecture 15 - Gradient Descent in Machine Learning
Convex Optimization Lecture 15 - Gradient Descent in Machine Learning Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 21 Today s Lecture 1 Motivation 2 Subgradient Method 3 Stochastic
More informationMath (P)refresher Lecture 8: Unconstrained Optimization
Math (P)refresher Lecture 8: Unconstrained Optimization September 2006 Today s Topics : Quadratic Forms Definiteness of Quadratic Forms Maxima and Minima in R n First Order Conditions Second Order Conditions
More informationIntroduction to unconstrained optimization - direct search methods
Introduction to unconstrained optimization - direct search methods Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Structure of optimization methods Typically Constraint handling converts the
More informationHigher-Order Methods
Higher-Order Methods Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. PCMI, July 2016 Stephen Wright (UW-Madison) Higher-Order Methods PCMI, July 2016 1 / 25 Smooth
More informationECE580 Exam 1 October 4, Please do not write on the back of the exam pages. Extra paper is available from the instructor.
ECE580 Exam 1 October 4, 2012 1 Name: Solution Score: /100 You must show ALL of your work for full credit. This exam is closed-book. Calculators may NOT be used. Please leave fractions as fractions, etc.
More informationOptimization. Escuela de Ingeniería Informática de Oviedo. (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30
Optimization Escuela de Ingeniería Informática de Oviedo (Dpto. de Matemáticas-UniOvi) Numerical Computation Optimization 1 / 30 Unconstrained optimization Outline 1 Unconstrained optimization 2 Constrained
More informationCHAPTER 2: QUADRATIC PROGRAMMING
CHAPTER 2: QUADRATIC PROGRAMMING Overview Quadratic programming (QP) problems are characterized by objective functions that are quadratic in the design variables, and linear constraints. In this sense,
More informationNumerical Optimization Prof. Shirish K. Shevade Department of Computer Science and Automation Indian Institute of Science, Bangalore
Numerical Optimization Prof. Shirish K. Shevade Department of Computer Science and Automation Indian Institute of Science, Bangalore Lecture - 13 Steepest Descent Method Hello, welcome back to this series
More informationCE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review
CE 191: Civil & Environmental Engineering Systems Analysis LEC 17 : Final Review Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 2014 Prof. Moura UC Berkeley
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More information3E4: Modelling Choice. Introduction to nonlinear programming. Announcements
3E4: Modelling Choice Lecture 7 Introduction to nonlinear programming 1 Announcements Solutions to Lecture 4-6 Homework will be available from http://www.eng.cam.ac.uk/~dr241/3e4 Looking ahead to Lecture
More information14. Nonlinear equations
L. Vandenberghe ECE133A (Winter 2018) 14. Nonlinear equations Newton method for nonlinear equations damped Newton method for unconstrained minimization Newton method for nonlinear least squares 14-1 Set
More informationChapter 3 Numerical Methods
Chapter 3 Numerical Methods Part 2 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization 1 Outline 3.2 Systems of Equations 3.3 Nonlinear and Constrained Optimization Summary 2 Outline 3.2
More informationExamination paper for TMA4180 Optimization I
Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted
More informationMATH 5720: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 2018
MATH 57: Unconstrained Optimization Hung Phan, UMass Lowell September 13, 18 1 Global and Local Optima Let a function f : S R be defined on a set S R n Definition 1 (minimizers and maximizers) (i) x S
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More informationMore First-Order Optimization Algorithms
More First-Order Optimization Algorithms Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 3, 8, 3 The SDM
More informationMATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2. Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year
MATHEMATICS FOR COMPUTER VISION WEEK 8 OPTIMISATION PART 2 1 Dr Fabio Cuzzolin MSc in Computer Vision Oxford Brookes University Year 2013-14 OUTLINE OF WEEK 8 topics: quadratic optimisation, least squares,
More informationDeep Learning. Authors: I. Goodfellow, Y. Bengio, A. Courville. Chapter 4: Numerical Computation. Lecture slides edited by C. Yim. C.
Chapter 4: Numerical Computation Deep Learning Authors: I. Goodfellow, Y. Bengio, A. Courville Lecture slides edited by 1 Chapter 4: Numerical Computation 4.1 Overflow and Underflow 4.2 Poor Conditioning
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More information