Lecture 15 Newton Method and SelfConcordance. October 23, 2008


 Iris Tyler
 1 years ago
 Views:
Transcription
1 Newton Method and SelfConcordance October 23, 2008
2 Outline Lecture 15 Selfconcordance Notion Selfconcordant Functions Operations Preserving Selfconcordance Properties of Selfconcordant Functions Implications for Newton s Method Equality Constrained Minimization Convex Optimization 1
3 Classical Convergence Analysis of Newton s Method Lecture 15 Given a level set L 0 = {x f(x) f(x 0 )}, it requires for all x, y L 0 : 2 f(x) 2 f(y) L x y 2 mi 2 f(x) MI for some constants L > 0, m > 0, and M > 0 Given a desired error level ɛ, we are interested in an ɛsolution of the problem i.e., a vector x such that f( x) f + ɛ An upper bound on the number of iterations to generate an ɛsolution is given by f(x 0 ) f ( ) ɛ0 + log 2 log 2 γ ɛ where γ = σβη 2 m, M 2 η (0, m 2 /L), and ɛ 0 = 2m 3 /L 2 Convex Optimization 2
4 This follows from L 2m 2 f(x k+k) ( ) L 2 K 2m 2 f(x k) ( 1 2 ) 2 K where k is such that f(x k ) η and η is small enough so that η L 2m 2 1. By the above relation and strong convexity of f, we have 2 f(x k ) f 1 2m f(x k) 2 2m3 L 2 ( 1 2 ) 2 K The bound is conceptually informative, but not practical Furthermore, the constants L, m, and M change with affine transformations of the space, while Newton s method is affine invariant Can a bound be obtained in terms of problem data that is affine invariant and, moreover, practically verifiable? Convex Optimization 3
5 Selfconcordance Nesterov and Nemirovski introduced a notion of selfconcordance and a class of selfconcordant functions Importance of the selfconcordance: Possesses affine invariant property Provides a new tool for analyzing Newton s method that exploits the affine invariance of the method Results in a practical upper bound on the Newton s iterations Plays a crucial role in performance analysis of interior point method Def. A function f : R R is selfconcordant when f is convex and f (x) 2f (x) 3/2 for all x domf The rate of change in curvature of f is bounded by the curvature Note: One can use a constant κ other than 2 in the definition Convex Optimization 4
6 Examples Linear and quadratic functions are selfconcordant [f (x) = 0 for all x] Negative logarithm f(x) = ln x, x > 0 is selfconcordant: f (x) = 1 x 2, f (x) = 2 f (x) x 3, = 2 for all x > 0 f (x) 3/2 Exponential function e x is not selfconcordant: f (x) = f (x) = e x, f (x) f (x) = ex 3/2 e = f (x) 3x/2 e x/2, as x f (x) 3/2 Evenpower monomial x 2p, p > 2 is not selfconcordant: f (x) = 2p(2p 1)x 2p 2, f (x) = 2p(2p 1)(2p 2)x 2p 3, f (x) f (x) = p 3 x 2p 3 3/2 p 2 x 3(p 1), f (x) as x 0 f (x) 3/2 Convex Optimization 5
7 Scaling and Affine Invariance In the definition of the selfconcordant function, one can use any other κ Let f be selfconcordant with κ, then f(x) = κ2 f(x) is selfconcordant 4 with κ = 2: f (x) 8κ2 f (x) = f (x) 3/2 4κ 3 f (x) 2 3/2 Selfconcordant function is affine invariant: for f selfconcordant and x = ay + b, we have f(y) = f(x) selfconcordant with the same κ: Convexity is preserved: f is convex f (y) = a 3 f (x), f (y) = a 2 f (x) f (x) a 3 f (x) = f (x) 3/2 a 3 f (x) κ 3/2 Convex Optimization 6
8 Selfconcordant Function in R n Def. A function f : R n R is selfconcordant when it is selfconcordant along every line, i.e., f is convex g(t) = f(x + tv) is selfconcordant for all x domf and all v, Note: The constant κ of selfconcordance is independent of the choice of x and v, i.e., g (t) g (t) 3/2 2 for all x domf, v R n, and t R such that x + tv domf Convex Optimization 7
9 Operations Preserving Selfconcordance Lecture 15 Note: To start with, these operations have to preserve convexity Scaling with a positive factor of at least 1: when f is selfconcordant and a > 1, then af is also selfconcordant The sum f 1 + f 2 of two selfconcordant functions is selfconcordant (extends to any finite sum) Composition with affine mapping: when f(y) is selfconcordant, then f(ax + b) is selfconcordant Composition with lnfunction: Let g : R R be a convex function with domg = R ++, and g (x) 3 g (x) x for all x > 0 Then: f(x) = ln ( g(x)) ln(x) over {x > 0 g(x) < 0} is selfconcordant HW7 Convex Optimization 8
10 Implications When f (x) > 0 for all x (then f is strictly convex), the selfconcordant condition can be written as d dx ( ) f (x) for all x domf. Assuming that for some t > 0, the interval [0, t] lies in domf, we can integrate from 0 to t, and obtain t t 0 d dx ( ) f (x) 1 2 dx t. Convex Optimization 9
11 This implies the following lower and upper bounds on f (x), f (0) ( ) 2 f (t) 1 + t f (0) f (0) ( ) 2, (1) 1 t f (0) where the lower bound is valid for all t 0 with t domf, and the upper bound is valid for all t domf with 0 t < f (0) 1/2. Convex Optimization 10
12 Bounds on f Consider f : R n R with 2 f(x) > 0. Let v be a descent direction, i.e., such that f(x) T v < 0. Consider g(t) = f(x + tv) f(x) for t > 0. Integrating the lower bound of Eq. (1) twice, we obtain ( ) g(t) g(0) + tg (0) + t g (0) ln 1 + t g (0) Consider now v = Newton s direction, i.e., v = [ 2 f(x)] 1 f(x). Let h(t) = f(x + tv). We have h (0) = f(x)v = λ 2 (x), h (0) = v T 2 f(0)v = λ 2 (x). Integrating the lower bound of Eq. (1) twice, we obtain for 0 t < 1 λ(x) h(t) h(0) tλ 2 (x) tλ(x) ln (1 tλ(x)) (2) Convex Optimization 11
13 Newton Direction for Selfconcordant Functions Lecture 15 Let f : R n R be selfconcordant and 2 f(x) 0 for all x L 0 Using selfconcordance it can be seen that For Newton s decrement λ(x) and any v R n : λ(x) = sup v 0 v T f(x) (v T 2 f(x)v) 1/2 HW7 with the supremum attainment at v = [ 2 f(x)] 1 f(x) Convex Optimization 12
14 SelfConcordant Functions: Newton Method Analysis Consider Newton s method, started at x 0, with backtracking line search. Assume that f : R n R is selfconcordant and strictly convex. (If domf R n, assume the level set {x f(x) f(x 0 )} is closed. Also, assume that f is bounded from below. Under the preceding assumptions an optimizer x of f over R n exists. HW7 Analysis is similar to the classic one except Selfconcordance replaces strict convexity and Lipschitz Hessian assumptions Newton decrement replaces the role of the gradient norm Convex Optimization 13
15 Convergence Result Theorem 1 Assume that f is selfconcordant strictly convex function that is bounded below over R n. Then, there exist η (0, 1/4) and γ such that Damped phase If λ k > η, we have f(x k+1 ) f(x k ) γ. Pure Newton If λ k η, the backtracking line search selects α = 1 and 2λ k+1 (2λ k ) 2 where λ k = λ(x k ). Convex Optimization 14
16 Proof Let h(α) = f(x k + αd k ). As seen in Eq. (2), we have for 0 α < 1 λ k h(α) h(0) αλ 2 k αλ k ln (1 αλ k ) (3) We use this relation to show that the stepsize ˆα = 1 1+λ k satisfies the exit condition of the line search. In particular, letting α = ˆα, we have h(ˆα) h(0) ˆαλ 2 k ˆαλ k ln(1 ˆαλ k ) = h(0) λ k + ln(1 + λ k ) Using the inequality t + ln(1 + t) we obtain t2 2(1+t) for t 0, and σ 1/2, Convex Optimization 15
17 h(ˆα) h(0) σ λ 2 k 1 + λ k = h(0) σˆαλ 2 k. Therefore, at the exit of the line search, we have α k β 1+λ k, implying λ2 k h(α k ) h(0) σβ. 1 + λ k When λ k > η, since h(α k ) = f(x k+1 ) and h(0) = f(x k ), we have f(x k+1 ) f(x k ) γ, γ = σβ η2 1 + η. Assume that λ k 1 2σ, from Eq. (3) and the inequality 2 x ln(1 x) x 2 /2 + x 3 for 0 x 0.81, we have Convex Optimization 16
18 h(1) h(0) 1 2 λ2 k + λ3 k h(0) σλ2 k, where the last inequality follows from λ k 1 2σ 2. Thus, the unit step satisfies the sufficient decrease condition (no damping). The proof that 2λ k+1 (2λ k ) 2 left for HW7. The preceding material is from Chapter 9.6 of the book by Boyd and Vandenberghe. Convex Optimization 17
19 Newton s Method for Selfconcordant Functions Lecture 15 Let f : R n R be selfconcordant and 2 f(x) 0 for all x L 0 The analysis of Newton s method with backtracking line search: For an ɛsolution, i.e., x with f( x) f + ɛ An upper bound on the number of iterations is given by f(x 0 ) f Γ 1 η2 + log 2 log 2, Γ = σβ ɛ 1 + η, η = 1 2σ 4 Explicitly: f(x 0 ) f Γ +log 2 log 2 1 ɛ = 20 8σ σβ(1 2σ) [f(x 0) f 1 ]+log 2 2 log 2 ɛ The bound is practical (when f can be underestimated) Convex Optimization 18
20 Equality Constrained Minimization minimize f(x) subject to Ax = b Lecture 15 Assumption: The function f is convex, twice continuously differentiable The matrix A R p n has Rank A = p Optimal value f is finite and attained If Ax = b for some x relint(domf), there is no duality gap and there exists an optimal dual solution KKT Optimality Conditions state x is optimal if and only if there exists λ such that Ax = b, f(x ) + A T λ = 0 Solving the problem is equivalent to solving the KKT conditions: n + p equations in n + p variables, x R n and λ R p Convex Optimization 19
21 Equality Constrained Quadratic Minimization 1 minimize 2 xt P x + q T x + r with P S+ n subject to Ax = b Important in extending the Newton s method to equality constraints KKT Optimality Conditions [ ] [ ] [ ] P A T x q A 0 λ = b Coefficient matrix is referred to as KKT matrix The KKT matrix is nonsingular if and only if Ax = 0, x 0 = x T P x > 0 [P is psd over null space of A] Equivalent conditions for nonsingularity of A: P + A T A 0 N (A) N (P ) = {0} Convex Optimization 20
22 Newton s Method with Equality Constraints Almost the same as Newton s method, except for two differences: The initial iterate x 0 has to be feasible, Ax 0 = b The Newton s directions d k have to be feasible, Ad k = 0 Given iterate x k, Newton s direction d k is determined as a solution to minimize f(x k ) + f(x k )d dt 2 f(x k )d subject to A(x k + d) = b The KKT conditions:(w k is optimal dual for the quadratic minimization) [ 2 f(x k ) A T ] [ dk ] [ f(xk ) ] A 0 w k = 0 Convex Optimization 21
23 Newton s Method with Equality Constraints Given starting point x domf with Ax = b and tolerance ɛ > 0. Repeat 1. Compute Newton s direction d N (x) by solving the corresponding KKT system. 2. Compute the decrement λ(x) 3. Stopping criterion. Quit if λ 2 /2 ɛ. 4. Line search. Choose stepsize α by backtracking line search. 5. Update. x := x + αd N (x). When f is strictly convex and selfconcordant, the bound on the number of iterations required to achieve an ɛaccuracy is the same as in the unconstrained case Convex Optimization 22
24 Newton s Method with Infeasible Points Useful when determining an initial feasible iterate x 0 is difficult Linearizing optimality conditions Ax = b and f(x ) + A T λ at some x + d x, with x is possibly infeasible, and using w λ We have f(x ) f(x + d) f(x) + 2 f(x)d Approximate KKT A(x + d) = b, f(x) + 2 f(x)d + A T w = 0 At infeasible x k, the set of linear inequalities determining d k and w k : [ 2 f(x k ) A T ] [ ] [ ] dk f(xk ) = A 0 Ax k b Note: When x k is feasible, the system of equations is the same as in the feasible point Newton s method w k Convex Optimization 23
25 Equality Constrained Analytic Centering minimize f(x) = n i=1 ln x i subject to Ax = b Feasible point Newton s method: g = f(x), H = 2 f(x) [ H A T A 0 ] [ d w ] = [ g 0 ], g = The Hessian is positive definite 1 x 1. 1 x n [ 1, H = diag,..., 1 x 2 1 x 2 n Lecture 15 KKT matrix first row: Hd+A T w = g d = H 1 (g +A T w) (1) KKT matrix second row, Ad = 0, and Eq. (1) AH 1 (g+a T w) = 0 The matrix A has full row rank, thus AH 1 A T is invertible, hence w = ( AH 1 A ) T 1 AH 1 g, H 1 = diag [ x 2 1,..., x2 n The matrix AH 1 A T is known as Schur complement of H (any H) ] ] Convex Optimization 24
26 Network Flow Optimization minimize nl=1 φ l (x l ) subject to Ax = b Directed graph with n arcs and p + 1 nodes Variable x l : flow through arc l Cost φ l : cost flow function for arc l, with φ l (t) > 0 Nodeincidence matrix Ã R (p+1) n defined as Ã il = 1 arc j originates at node i 1 arc j ends at node i 0 otherwise Reduced nodeincidence matrix A R p n is Ã with last row removed b R p is (reduced) source vector Rank A = p when the graph is connected Convex Optimization 25
27 KKT system for infeasible Newton s method [ H A T A 0 ] [ d w ] = where h = Ax b is a measure of infeasibility at the current point x g = [φ 1(x 1 ),..., φ n (x n )] T [ g h ] H = diag [φ 1(x 1 ),..., φ n(x n )] with positive diagonal entries Solve via elimination: w = (AH 1 A T ) 1 [h AH 1 g], d = H 1 (g + A T w) Convex Optimization 26
10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method selfconcordant functions implementation
More information10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method selfconcordant functions implementation
More informationUnconstrained minimization
CSCI5254: Convex Optimization & Its Applications Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method selfconcordant functions 1 Unconstrained
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationNewton s Method. Javier Peña Convex Optimization /36725
Newton s Method Javier Peña Convex Optimization 10725/36725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36725
Newton s Method Ryan Tibshirani Convex Optimization 10725/36725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More information11. Equality constrained minimization
Convex Optimization Boyd & Vandenberghe 11. Equality constrained minimization equality constrained minimization eliminating equality constraints Newton s method with equality constraints infeasible start
More informationLecture 14: October 17
1725/36725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationBarrier Method. Javier Peña Convex Optimization /36725
Barrier Method Javier Peña Convex Optimization 10725/36725 1 Last time: Newton s method For rootfinding F (x) = 0 x + = x F (x) 1 F (x) For optimization x f(x) x + = x 2 f(x) 1 f(x) Assume f strongly
More informationORIE 6326: Convex Optimization. QuasiNewton Methods
ORIE 6326: Convex Optimization QuasiNewton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationConvex Optimization. 9. Unconstrained minimization. Prof. Ying Cui. Department of Electrical Engineering Shanghai Jiao Tong University
Convex Optimization 9. Unconstrained minimization Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2017 Autumn Semester SJTU Ying Cui 1 / 40 Outline Unconstrained minimization
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationEquality constrained minimization
Chapter 10 Equality constrained minimization 10.1 Equality constrained minimization problems In this chapter we describe methods for solving a convex optimization problem with equality constraints, minimize
More informationLecture 14: Newton s Method
10725/36725: Conve Optimization Fall 2016 Lecturer: Javier Pena Lecture 14: Newton s ethod Scribes: Varun Joshi, Xuan Li Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationConvex Optimization. Problem set 2. Due Monday November 4th
Convex Optimization Problem set 2 Due Monday November 4th 1 Gradent Decent without Linesearch In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Linesearch In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationA Distributed Newton Method for Network Utility Maximization, II: Convergence
A Distributed Newton Method for Network Utility Maximization, II: Convergence Ermin Wei, Asuman Ozdaglar, and Ali Jadbabaie October 31, 2012 Abstract The existing distributed algorithms for Network Utility
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More informationUnconstrained minimization: assumptions
Unconstrained minimization I terminology and assumptions I gradient descent method I steepest descent method I Newton s method I selfconcordant functions I implementation IOE 611: Nonlinear Programming,
More informationHomework 4. Convex Optimization /36725
Homework 4 Convex Optimization 10725/36725 Due Friday November 4 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationConvex Optimization and l 1 minimization
Convex Optimization and l 1 minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationCSCI 1951G Optimization Methods in Finance Part 09: Interior Point Methods
CSCI 1951G Optimization Methods in Finance Part 09: Interior Point Methods March 23, 2018 1 / 35 This material is covered in S. Boyd, L. Vandenberge s book Convex Optimization https://web.stanford.edu/~boyd/cvxbook/.
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primaldual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primaldual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More informationConvex Optimization. Lecture 12  Equality Constrained Optimization. Instructor: Yuanzhang Xiao. Fall University of Hawaii at Manoa
Convex Optimization Lecture 12  Equality Constrained Optimization Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 19 Today s Lecture 1 Basic Concepts 2 for Equality Constrained
More informationConvex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470  Convex Optimization Fall 201718, HKUST, Hong Kong Outline of Lecture Definition convex function Examples
More informationConvex Optimization. Prof. Nati Srebro. Lecture 12: InfeasibleStart Newton s Method Interior Point Methods
Convex Optimization Prof. Nati Srebro Lecture 12: InfeasibleStart Newton s Method Interior Point Methods Equality Constrained Optimization f 0 (x) s. t. A R p n, b R p Using access to: 2 nd order oracle
More information1. Gradient method. gradient method, firstorder methods. quadratic bounds on convex functions. analysis of gradient method
L. Vandenberghe EE236C (Spring 2016) 1. Gradient method gradient method, firstorder methods quadratic bounds on convex functions analysis of gradient method 11 Approximate course outline Firstorder
More informationConditional Gradient (FrankWolfe) Method
Conditional Gradient (FrankWolfe) Method Lecturer: Aarti Singh Coinstructor: Pradeep Ravikumar Convex Optimization 10725/36725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationSelfConcordant Barrier Functions for Convex Optimization
Appendix F SelfConcordant Barrier Functions for Convex Optimization F.1 Introduction In this Appendix we present a framework for developing polynomialtime algorithms for the solution of convex optimization
More informationLecture 9 Sequential unconstrained minimization
S. Boyd EE364 Lecture 9 Sequential unconstrained minimization brief history of SUMT & IP methods logarithmic barrier function central path UMT & SUMT complexity analysis feasibility phase generalized inequalities
More information12. Interiorpoint methods
12. Interiorpoint methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 function with f () on (, L) and that you have explicit formulae for
More informationmin f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;
Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More information14. Nonlinear equations
L. Vandenberghe ECE133A (Winter 2018) 14. Nonlinear equations Newton method for nonlinear equations damped Newton method for unconstrained minimization Newton method for nonlinear least squares 141 Set
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationMore FirstOrder Optimization Algorithms
More FirstOrder Optimization Algorithms Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 3, 8, 3 The SDM
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equalityconstrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equalityconstrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationPrimalDual InteriorPoint Methods for Linear Programming based on Newton s Method
PrimalDual InteriorPoint Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions logconcave and logconvex functions
More informationLecture 4: Convex Functions, Part I February 1
IE 521: Convex Optimization Instructor: Niao He Lecture 4: Convex Functions, Part I February 1 Spring 2017, UIUC Scribe: Shuanglong Wang Courtesy warning: These notes do not necessarily cover everything
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationIE 521 Convex Optimization
Lecture 5: Convex II 6th February 2019 Convex Local Lipschitz Outline Local Lipschitz 1 / 23 Convex Local Lipschitz Convex Function: f : R n R is convex if dom(f ) is convex and for any λ [0, 1], x, y
More informationSolving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
More information12. Interiorpoint methods
12. Interiorpoint methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationUses of duality. Geoff Gordon & Ryan Tibshirani Optimization /
Uses of duality Geoff Gordon & Ryan Tibshirani Optimization 10725 / 36725 1 Remember conjugate functions Given f : R n R, the function is called its conjugate f (y) = max x R n yt x f(x) Conjugates appear
More informationGradient Descent. Ryan Tibshirani Convex Optimization /36725
Gradient Descent Ryan Tibshirani Convex Optimization 10725/36725 Last time: canonical convex programs Linear program (LP): takes the form min x subject to c T x Gx h Ax = b Quadratic program (QP): like
More informationLecture 5: September 12
10725/36725: Convex Optimization Fall 2015 Lecture 5: September 12 Lecturer: Lecturer: Ryan Tibshirani Scribes: Scribes: Barun Patra and Tyler Vuong Note: LaTeX template courtesy of UC Berkeley EECS
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationCSE4830 Kernel Methods in Machine Learning
CSE4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationConvex Optimization. Ofer Meshi. Lecture 6: Lower Bounds Constrained Optimization
Convex Optimization Ofer Meshi Lecture 6: Lower Bounds Constrained Optimization Lower Bounds Some upper bounds: #iter μ 2 M #iter 2 M #iter L L μ 2 Oracle/ops GD κ log 1/ε M x # ε L # x # L # ε # με f
More informationAn Optimal Affine Invariant Smooth Minimization Algorithm.
An Optimal Affine Invariant Smooth Minimization Algorithm. Alexandre d Aspremont, CNRS & École Polytechnique. Joint work with Martin Jaggi. Support from ERC SIPA. A. d Aspremont IWSL, Moscow, June 2013,
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationLecture 16: October 22
0725/36725: Conve Optimization Fall 208 Lecturer: Ryan Tibshirani Lecture 6: October 22 Scribes: Nic Dalmasso, Alan Mishler, Benja LeRoy Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions
More information6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection
6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection TwoMetric Projection Methods Manifold Suboptimization Methods
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very wellwritten and a pleasure to read. The
More informationChemical Equilibrium: A Convex Optimization Problem
Chemical Equilibrium: A Convex Optimization Problem Linyi Gao June 4, 2014 1 Introduction The equilibrium composition of a mixture of reacting molecules is essential to many physical and chemical systems,
More informationLecture 14 Barrier method
L. Vandenberghe EE236A (Fall 201314) Lecture 14 Barrier method centering problem Newton decrement local convergence of Newton method shortstep barrier method global convergence of Newton method predictorcorrector
More informationEE364a Homework 8 solutions
EE364a, Winter 200708 Prof. S. Boyd EE364a Homework 8 solutions 9.8 Steepest descent method in l norm. Explain how to find a steepest descent direction in the l norm, and give a simple interpretation.
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationDescent methods. min x. f(x)
Gradient Descent Descent methods min x f(x) 5 / 34 Descent methods min x f(x) x k x k+1... x f(x ) = 0 5 / 34 Gradient methods Unconstrained optimization min f(x) x R n. 6 / 34 Gradient methods Unconstrained
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms  A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on ChongZak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More information6. Proximal gradient method
L. Vandenberghe EE236C (Spring 2016) 6. Proximal gradient method motivation proximal mapping proximal gradient method with fixed step size proximal gradient method with line search 61 Proximal mapping
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 3: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationA function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3
Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is welldefined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all
More informationLecture 4  The Gradient Method Objective: find an optimal solution of the problem
Lecture 4  The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationPrimalDual InteriorPoint Methods
PrimalDual InteriorPoint Methods Lecturer: Aarti Singh Coinstructor: Pradeep Ravikumar Convex Optimization 10725/36725 Outline Today: Primaldual interiorpoint method Special case: linear programming
More informationSelected Topics in Optimization. Some slides borrowed from
Selected Topics in Optimization Some slides borrowed from http://www.stat.cmu.edu/~ryantibs/convexopt/ Overview Optimization problems are almost everywhere in statistics and machine learning. Input Model
More informationLecture 5: Gradient Descent. 5.1 Unconstrained minimization problems and Gradient descent
10725/36725: Convex Optimization Spring 2015 Lecturer: Ryan Tibshirani Lecture 5: Gradient Descent Scribes: Loc Do,2,3 Disclaimer: These notes have not been subjected to the usual scrutiny reserved for
More informationLecture 4  The Gradient Method Objective: find an optimal solution of the problem
Lecture 4  The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More information10725/ Optimization Midterm Exam
10725/36725 Optimization Midterm Exam November 6, 2012 NAME: ANDREW ID: Instructions: This exam is 1hr 20mins long Except for a single twosided sheet of notes, no other material or discussion is permitted
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Postdoctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More information5. Subgradient method
L. Vandenberghe EE236C (Spring 2016) 5. Subgradient method subgradient method convergence analysis optimal step size when f is known alternating projections optimality 51 Subgradient method to minimize
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions logconcave and logconvex functions
More informationPrimaldual Subgradient Method for Convex Problems with Functional Constraints
Primaldual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primaldual
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationChapter 2. Optimization. Gradients, convexity, and ALS
Chapter 2 Optimization Gradients, convexity, and ALS Contents Background Gradient descent Stochastic gradient descent Newton s method Alternating least squares KKT conditions 2 Motivation We can solve
More informationFrankWolfe Method. Ryan Tibshirani Convex Optimization
FrankWolfe Method Ryan Tibshirani Convex Optimization 10725 Last time: ADMM For the problem min x,z f(x) + g(z) subject to Ax + Bz = c we form augmented Lagrangian (scaled form): L ρ (x, z, w) = f(x)
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationSecond Order Optimization Algorithms I
Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The
More informationLecture 3: Basics of setconstrained and unconstrained optimization
Lecture 3: Basics of setconstrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018 Optimization basics Outline Optimization
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationSWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University
SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University General NLO problem (NLO : Non Linear Optimization) (N LO) min
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More information