Lecture 15 Newton Method and Self-Concordance. October 23, 2008
|
|
- Iris Tyler
- 6 years ago
- Views:
Transcription
1 Newton Method and Self-Concordance October 23, 2008
2 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications for Newton s Method Equality Constrained Minimization Convex Optimization 1
3 Classical Convergence Analysis of Newton s Method Lecture 15 Given a level set L 0 = {x f(x) f(x 0 )}, it requires for all x, y L 0 : 2 f(x) 2 f(y) L x y 2 mi 2 f(x) MI for some constants L > 0, m > 0, and M > 0 Given a desired error level ɛ, we are interested in an ɛ-solution of the problem i.e., a vector x such that f( x) f + ɛ An upper bound on the number of iterations to generate an ɛ-solution is given by f(x 0 ) f ( ) ɛ0 + log 2 log 2 γ ɛ where γ = σβη 2 m, M 2 η (0, m 2 /L), and ɛ 0 = 2m 3 /L 2 Convex Optimization 2
4 This follows from L 2m 2 f(x k+k) ( ) L 2 K 2m 2 f(x k) ( 1 2 ) 2 K where k is such that f(x k ) η and η is small enough so that η L 2m 2 1. By the above relation and strong convexity of f, we have 2 f(x k ) f 1 2m f(x k) 2 2m3 L 2 ( 1 2 ) 2 K The bound is conceptually informative, but not practical Furthermore, the constants L, m, and M change with affine transformations of the space, while Newton s method is affine invariant Can a bound be obtained in terms of problem data that is affine invariant and, moreover, practically verifiable? Convex Optimization 3
5 Self-concordance Nesterov and Nemirovski introduced a notion of self-concordance and a class of self-concordant functions Importance of the self-concordance: Possesses affine invariant property Provides a new tool for analyzing Newton s method that exploits the affine invariance of the method Results in a practical upper bound on the Newton s iterations Plays a crucial role in performance analysis of interior point method Def. A function f : R R is self-concordant when f is convex and f (x) 2f (x) 3/2 for all x domf The rate of change in curvature of f is bounded by the curvature Note: One can use a constant κ other than 2 in the definition Convex Optimization 4
6 Examples Linear and quadratic functions are self-concordant [f (x) = 0 for all x] Negative logarithm f(x) = ln x, x > 0 is self-concordant: f (x) = 1 x 2, f (x) = 2 f (x) x 3, = 2 for all x > 0 f (x) 3/2 Exponential function e x is not self-concordant: f (x) = f (x) = e x, f (x) f (x) = ex 3/2 e = f (x) 3x/2 e x/2, as x f (x) 3/2 Even-power monomial x 2p, p > 2 is not self-concordant: f (x) = 2p(2p 1)x 2p 2, f (x) = 2p(2p 1)(2p 2)x 2p 3, f (x) f (x) = p 3 x 2p 3 3/2 p 2 x 3(p 1), f (x) as x 0 f (x) 3/2 Convex Optimization 5
7 Scaling and Affine Invariance In the definition of the self-concordant function, one can use any other κ Let f be self-concordant with κ, then f(x) = κ2 f(x) is self-concordant 4 with κ = 2: f (x) 8κ2 f (x) = f (x) 3/2 4κ 3 f (x) 2 3/2 Self-concordant function is affine invariant: for f self-concordant and x = ay + b, we have f(y) = f(x) self-concordant with the same κ: Convexity is preserved: f is convex f (y) = a 3 f (x), f (y) = a 2 f (x) f (x) a 3 f (x) = f (x) 3/2 a 3 f (x) κ 3/2 Convex Optimization 6
8 Self-concordant Function in R n Def. A function f : R n R is self-concordant when it is self-concordant along every line, i.e., f is convex g(t) = f(x + tv) is self-concordant for all x domf and all v, Note: The constant κ of self-concordance is independent of the choice of x and v, i.e., g (t) g (t) 3/2 2 for all x domf, v R n, and t R such that x + tv domf Convex Optimization 7
9 Operations Preserving Self-concordance Lecture 15 Note: To start with, these operations have to preserve convexity Scaling with a positive factor of at least 1: when f is self-concordant and a > 1, then af is also self-concordant The sum f 1 + f 2 of two self-concordant functions is self-concordant (extends to any finite sum) Composition with affine mapping: when f(y) is self-concordant, then f(ax + b) is self-concordant Composition with ln-function: Let g : R R be a convex function with domg = R ++, and g (x) 3 g (x) x for all x > 0 Then: f(x) = ln ( g(x)) ln(x) over {x > 0 g(x) < 0} is self-concordant HW7 Convex Optimization 8
10 Implications When f (x) > 0 for all x (then f is strictly convex), the self-concordant condition can be written as d dx ( ) f (x) for all x domf. Assuming that for some t > 0, the interval [0, t] lies in domf, we can integrate from 0 to t, and obtain t t 0 d dx ( ) f (x) 1 2 dx t. Convex Optimization 9
11 This implies the following lower and upper bounds on f (x), f (0) ( ) 2 f (t) 1 + t f (0) f (0) ( ) 2, (1) 1 t f (0) where the lower bound is valid for all t 0 with t domf, and the upper bound is valid for all t domf with 0 t < f (0) 1/2. Convex Optimization 10
12 Bounds on f Consider f : R n R with 2 f(x) > 0. Let v be a descent direction, i.e., such that f(x) T v < 0. Consider g(t) = f(x + tv) f(x) for t > 0. Integrating the lower bound of Eq. (1) twice, we obtain ( ) g(t) g(0) + tg (0) + t g (0) ln 1 + t g (0) Consider now v = Newton s direction, i.e., v = [ 2 f(x)] 1 f(x). Let h(t) = f(x + tv). We have h (0) = f(x)v = λ 2 (x), h (0) = v T 2 f(0)v = λ 2 (x). Integrating the lower bound of Eq. (1) twice, we obtain for 0 t < 1 λ(x) h(t) h(0) tλ 2 (x) tλ(x) ln (1 tλ(x)) (2) Convex Optimization 11
13 Newton Direction for Self-concordant Functions Lecture 15 Let f : R n R be self-concordant and 2 f(x) 0 for all x L 0 Using self-concordance it can be seen that For Newton s decrement λ(x) and any v R n : λ(x) = sup v 0 v T f(x) (v T 2 f(x)v) 1/2 HW7 with the supremum attainment at v = [ 2 f(x)] 1 f(x) Convex Optimization 12
14 Self-Concordant Functions: Newton Method Analysis Consider Newton s method, started at x 0, with backtracking line search. Assume that f : R n R is self-concordant and strictly convex. (If domf R n, assume the level set {x f(x) f(x 0 )} is closed. Also, assume that f is bounded from below. Under the preceding assumptions an optimizer x of f over R n exists. HW7 Analysis is similar to the classic one except Self-concordance replaces strict convexity and Lipschitz Hessian assumptions Newton decrement replaces the role of the gradient norm Convex Optimization 13
15 Convergence Result Theorem 1 Assume that f is self-concordant strictly convex function that is bounded below over R n. Then, there exist η (0, 1/4) and γ such that Damped phase If λ k > η, we have f(x k+1 ) f(x k ) γ. Pure Newton If λ k η, the backtracking line search selects α = 1 and 2λ k+1 (2λ k ) 2 where λ k = λ(x k ). Convex Optimization 14
16 Proof Let h(α) = f(x k + αd k ). As seen in Eq. (2), we have for 0 α < 1 λ k h(α) h(0) αλ 2 k αλ k ln (1 αλ k ) (3) We use this relation to show that the stepsize ˆα = 1 1+λ k satisfies the exit condition of the line search. In particular, letting α = ˆα, we have h(ˆα) h(0) ˆαλ 2 k ˆαλ k ln(1 ˆαλ k ) = h(0) λ k + ln(1 + λ k ) Using the inequality t + ln(1 + t) we obtain t2 2(1+t) for t 0, and σ 1/2, Convex Optimization 15
17 h(ˆα) h(0) σ λ 2 k 1 + λ k = h(0) σˆαλ 2 k. Therefore, at the exit of the line search, we have α k β 1+λ k, implying λ2 k h(α k ) h(0) σβ. 1 + λ k When λ k > η, since h(α k ) = f(x k+1 ) and h(0) = f(x k ), we have f(x k+1 ) f(x k ) γ, γ = σβ η2 1 + η. Assume that λ k 1 2σ, from Eq. (3) and the inequality 2 x ln(1 x) x 2 /2 + x 3 for 0 x 0.81, we have Convex Optimization 16
18 h(1) h(0) 1 2 λ2 k + λ3 k h(0) σλ2 k, where the last inequality follows from λ k 1 2σ 2. Thus, the unit step satisfies the sufficient decrease condition (no damping). The proof that 2λ k+1 (2λ k ) 2 left for HW7. The preceding material is from Chapter 9.6 of the book by Boyd and Vandenberghe. Convex Optimization 17
19 Newton s Method for Self-concordant Functions Lecture 15 Let f : R n R be self-concordant and 2 f(x) 0 for all x L 0 The analysis of Newton s method with backtracking line search: For an ɛ-solution, i.e., x with f( x) f + ɛ An upper bound on the number of iterations is given by f(x 0 ) f Γ 1 η2 + log 2 log 2, Γ = σβ ɛ 1 + η, η = 1 2σ 4 Explicitly: f(x 0 ) f Γ +log 2 log 2 1 ɛ = 20 8σ σβ(1 2σ) [f(x 0) f 1 ]+log 2 2 log 2 ɛ The bound is practical (when f can be underestimated) Convex Optimization 18
20 Equality Constrained Minimization minimize f(x) subject to Ax = b Lecture 15 Assumption: The function f is convex, twice continuously differentiable The matrix A R p n has Rank A = p Optimal value f is finite and attained If Ax = b for some x relint(domf), there is no duality gap and there exists an optimal dual solution KKT Optimality Conditions state x is optimal if and only if there exists λ such that Ax = b, f(x ) + A T λ = 0 Solving the problem is equivalent to solving the KKT conditions: n + p equations in n + p variables, x R n and λ R p Convex Optimization 19
21 Equality Constrained Quadratic Minimization 1 minimize 2 xt P x + q T x + r with P S+ n subject to Ax = b Important in extending the Newton s method to equality constraints KKT Optimality Conditions [ ] [ ] [ ] P A T x q A 0 λ = b Coefficient matrix is referred to as KKT matrix The KKT matrix is nonsingular if and only if Ax = 0, x 0 = x T P x > 0 [P is psd over null space of A] Equivalent conditions for nonsingularity of A: P + A T A 0 N (A) N (P ) = {0} Convex Optimization 20
22 Newton s Method with Equality Constraints Almost the same as Newton s method, except for two differences: The initial iterate x 0 has to be feasible, Ax 0 = b The Newton s directions d k have to be feasible, Ad k = 0 Given iterate x k, Newton s direction d k is determined as a solution to minimize f(x k ) + f(x k )d dt 2 f(x k )d subject to A(x k + d) = b The KKT conditions:(w k is optimal dual for the quadratic minimization) [ 2 f(x k ) A T ] [ dk ] [ f(xk ) ] A 0 w k = 0 Convex Optimization 21
23 Newton s Method with Equality Constraints Given starting point x domf with Ax = b and tolerance ɛ > 0. Repeat 1. Compute Newton s direction d N (x) by solving the corresponding KKT system. 2. Compute the decrement λ(x) 3. Stopping criterion. Quit if λ 2 /2 ɛ. 4. Line search. Choose stepsize α by backtracking line search. 5. Update. x := x + αd N (x). When f is strictly convex and self-concordant, the bound on the number of iterations required to achieve an ɛ-accuracy is the same as in the unconstrained case Convex Optimization 22
24 Newton s Method with Infeasible Points Useful when determining an initial feasible iterate x 0 is difficult Linearizing optimality conditions Ax = b and f(x ) + A T λ at some x + d x, with x is possibly infeasible, and using w λ We have f(x ) f(x + d) f(x) + 2 f(x)d Approximate KKT A(x + d) = b, f(x) + 2 f(x)d + A T w = 0 At infeasible x k, the set of linear inequalities determining d k and w k : [ 2 f(x k ) A T ] [ ] [ ] dk f(xk ) = A 0 Ax k b Note: When x k is feasible, the system of equations is the same as in the feasible point Newton s method w k Convex Optimization 23
25 Equality Constrained Analytic Centering minimize f(x) = n i=1 ln x i subject to Ax = b Feasible point Newton s method: g = f(x), H = 2 f(x) [ H A T A 0 ] [ d w ] = [ g 0 ], g = The Hessian is positive definite 1 x 1. 1 x n [ 1, H = diag,..., 1 x 2 1 x 2 n Lecture 15 KKT matrix first row: Hd+A T w = g d = H 1 (g +A T w) (1) KKT matrix second row, Ad = 0, and Eq. (1) AH 1 (g+a T w) = 0 The matrix A has full row rank, thus AH 1 A T is invertible, hence w = ( AH 1 A ) T 1 AH 1 g, H 1 = diag [ x 2 1,..., x2 n The matrix AH 1 A T is known as Schur complement of H (any H) ] ] Convex Optimization 24
26 Network Flow Optimization minimize nl=1 φ l (x l ) subject to Ax = b Directed graph with n arcs and p + 1 nodes Variable x l : flow through arc l Cost φ l : cost flow function for arc l, with φ l (t) > 0 Node-incidence matrix à R (p+1) n defined as à il = 1 arc j originates at node i 1 arc j ends at node i 0 otherwise Reduced node-incidence matrix A R p n is à with last row removed b R p is (reduced) source vector Rank A = p when the graph is connected Convex Optimization 25
27 KKT system for infeasible Newton s method [ H A T A 0 ] [ d w ] = where h = Ax b is a measure of infeasibility at the current point x g = [φ 1(x 1 ),..., φ n (x n )] T [ g h ] H = diag [φ 1(x 1 ),..., φ n(x n )] with positive diagonal entries Solve via elimination: w = (AH 1 A T ) 1 [h AH 1 g], d = H 1 (g + A T w) Convex Optimization 26
10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation
More informationUnconstrained minimization
CSCI5254: Convex Optimization & Its Applications Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions 1 Unconstrained
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More information11. Equality constrained minimization
Convex Optimization Boyd & Vandenberghe 11. Equality constrained minimization equality constrained minimization eliminating equality constraints Newton s method with equality constraints infeasible start
More informationLecture 14: October 17
1-725/36-725: Convex Optimization Fall 218 Lecture 14: October 17 Lecturer: Lecturer: Ryan Tibshirani Scribes: Pengsheng Guo, Xian Zhou Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationBarrier Method. Javier Peña Convex Optimization /36-725
Barrier Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: Newton s method For root-finding F (x) = 0 x + = x F (x) 1 F (x) For optimization x f(x) x + = x 2 f(x) 1 f(x) Assume f strongly
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationConvex Optimization. 9. Unconstrained minimization. Prof. Ying Cui. Department of Electrical Engineering Shanghai Jiao Tong University
Convex Optimization 9. Unconstrained minimization Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2017 Autumn Semester SJTU Ying Cui 1 / 40 Outline Unconstrained minimization
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationEquality constrained minimization
Chapter 10 Equality constrained minimization 10.1 Equality constrained minimization problems In this chapter we describe methods for solving a convex optimization problem with equality constraints, minimize
More informationLecture 14: Newton s Method
10-725/36-725: Conve Optimization Fall 2016 Lecturer: Javier Pena Lecture 14: Newton s ethod Scribes: Varun Joshi, Xuan Li Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationConvex Optimization. Problem set 2. Due Monday April 26th
Convex Optimization Problem set 2 Due Monday April 26th 1 Gradient Decent without Line-search In this problem we will consider gradient descent with predetermined step sizes. That is, instead of determining
More informationA Distributed Newton Method for Network Utility Maximization, II: Convergence
A Distributed Newton Method for Network Utility Maximization, II: Convergence Ermin Wei, Asuman Ozdaglar, and Ali Jadbabaie October 31, 2012 Abstract The existing distributed algorithms for Network Utility
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More informationUnconstrained minimization: assumptions
Unconstrained minimization I terminology and assumptions I gradient descent method I steepest descent method I Newton s method I self-concordant functions I implementation IOE 611: Nonlinear Programming,
More informationConvex Optimization and l 1 -minimization
Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationCSCI 1951-G Optimization Methods in Finance Part 09: Interior Point Methods
CSCI 1951-G Optimization Methods in Finance Part 09: Interior Point Methods March 23, 2018 1 / 35 This material is covered in S. Boyd, L. Vandenberge s book Convex Optimization https://web.stanford.edu/~boyd/cvxbook/.
More informationHomework 4. Convex Optimization /36-725
Homework 4 Convex Optimization 10-725/36-725 Due Friday November 4 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primal-dual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More informationConvex Optimization. Lecture 12 - Equality Constrained Optimization. Instructor: Yuanzhang Xiao. Fall University of Hawaii at Manoa
Convex Optimization Lecture 12 - Equality Constrained Optimization Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 19 Today s Lecture 1 Basic Concepts 2 for Equality Constrained
More informationConvex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples
More informationConvex Optimization. Prof. Nati Srebro. Lecture 12: Infeasible-Start Newton s Method Interior Point Methods
Convex Optimization Prof. Nati Srebro Lecture 12: Infeasible-Start Newton s Method Interior Point Methods Equality Constrained Optimization f 0 (x) s. t. A R p n, b R p Using access to: 2 nd order oracle
More information1. Gradient method. gradient method, first-order methods. quadratic bounds on convex functions. analysis of gradient method
L. Vandenberghe EE236C (Spring 2016) 1. Gradient method gradient method, first-order methods quadratic bounds on convex functions analysis of gradient method 1-1 Approximate course outline First-order
More informationConditional Gradient (Frank-Wolfe) Method
Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationSelf-Concordant Barrier Functions for Convex Optimization
Appendix F Self-Concordant Barrier Functions for Convex Optimization F.1 Introduction In this Appendix we present a framework for developing polynomial-time algorithms for the solution of convex optimization
More informationLecture 9 Sequential unconstrained minimization
S. Boyd EE364 Lecture 9 Sequential unconstrained minimization brief history of SUMT & IP methods logarithmic barrier function central path UMT & SUMT complexity analysis feasibility phase generalized inequalities
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for
More informationmin f(x). (2.1) Objectives consisting of a smooth convex term plus a nonconvex regularization term;
Chapter 2 Gradient Methods The gradient method forms the foundation of all of the schemes studied in this book. We will provide several complementary perspectives on this algorithm that highlight the many
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More information14. Nonlinear equations
L. Vandenberghe ECE133A (Winter 2018) 14. Nonlinear equations Newton method for nonlinear equations damped Newton method for unconstrained minimization Newton method for nonlinear least squares 14-1 Set
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationMore First-Order Optimization Algorithms
More First-Order Optimization Algorithms Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 3, 8, 3 The SDM
More informationThe Steepest Descent Algorithm for Unconstrained Optimization
The Steepest Descent Algorithm for Unconstrained Optimization Robert M. Freund February, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 1 Steepest Descent Algorithm The problem
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 8. Smooth convex unconstrained and equality-constrained minimization
E5295/5B5749 Convex optimization with engineering applications Lecture 8 Smooth convex unconstrained and equality-constrained minimization A. Forsgren, KTH 1 Lecture 8 Convex optimization 2006/2007 Unconstrained
More informationPrimal-Dual Interior-Point Methods for Linear Programming based on Newton s Method
Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationUnconstrained minimization of smooth functions
Unconstrained minimization of smooth functions We want to solve min x R N f(x), where f is convex. In this section, we will assume that f is differentiable (so its gradient exists at every point), and
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationLecture 4: Convex Functions, Part I February 1
IE 521: Convex Optimization Instructor: Niao He Lecture 4: Convex Functions, Part I February 1 Spring 2017, UIUC Scribe: Shuanglong Wang Courtesy warning: These notes do not necessarily cover everything
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationIE 521 Convex Optimization
Lecture 5: Convex II 6th February 2019 Convex Local Lipschitz Outline Local Lipschitz 1 / 23 Convex Local Lipschitz Convex Function: f : R n R is convex if dom(f ) is convex and for any λ [0, 1], x, y
More informationSolving Dual Problems
Lecture 20 Solving Dual Problems We consider a constrained problem where, in addition to the constraint set X, there are also inequality and linear equality constraints. Specifically the minimization problem
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationUses of duality. Geoff Gordon & Ryan Tibshirani Optimization /
Uses of duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Remember conjugate functions Given f : R n R, the function is called its conjugate f (y) = max x R n yt x f(x) Conjugates appear
More informationGradient Descent. Ryan Tibshirani Convex Optimization /36-725
Gradient Descent Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: canonical convex programs Linear program (LP): takes the form min x subject to c T x Gx h Ax = b Quadratic program (QP): like
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationLecture 5: September 12
10-725/36-725: Convex Optimization Fall 2015 Lecture 5: September 12 Lecturer: Lecturer: Ryan Tibshirani Scribes: Scribes: Barun Patra and Tyler Vuong Note: LaTeX template courtesy of UC Berkeley EECS
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationConvex Optimization. Ofer Meshi. Lecture 6: Lower Bounds Constrained Optimization
Convex Optimization Ofer Meshi Lecture 6: Lower Bounds Constrained Optimization Lower Bounds Some upper bounds: #iter μ 2 M #iter 2 M #iter L L μ 2 Oracle/ops GD κ log 1/ε M x # ε L # x # L # ε # με f
More informationAn Optimal Affine Invariant Smooth Minimization Algorithm.
An Optimal Affine Invariant Smooth Minimization Algorithm. Alexandre d Aspremont, CNRS & École Polytechnique. Joint work with Martin Jaggi. Support from ERC SIPA. A. d Aspremont IWSL, Moscow, June 2013,
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationOptimization Tutorial 1. Basic Gradient Descent
E0 270 Machine Learning Jan 16, 2015 Optimization Tutorial 1 Basic Gradient Descent Lecture by Harikrishna Narasimhan Note: This tutorial shall assume background in elementary calculus and linear algebra.
More informationLecture 16: October 22
0-725/36-725: Conve Optimization Fall 208 Lecturer: Ryan Tibshirani Lecture 6: October 22 Scribes: Nic Dalmasso, Alan Mishler, Benja LeRoy Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationMATH 4211/6211 Optimization Basics of Optimization Problems
MATH 4211/6211 Optimization Basics of Optimization Problems Xiaojing Ye Department of Mathematics & Statistics Georgia State University Xiaojing Ye, Math & Stat, Georgia State University 0 A standard minimization
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions
More information6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection
6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods
More informationChemical Equilibrium: A Convex Optimization Problem
Chemical Equilibrium: A Convex Optimization Problem Linyi Gao June 4, 2014 1 Introduction The equilibrium composition of a mixture of reacting molecules is essential to many physical and chemical systems,
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationLecture 14 Barrier method
L. Vandenberghe EE236A (Fall 2013-14) Lecture 14 Barrier method centering problem Newton decrement local convergence of Newton method short-step barrier method global convergence of Newton method predictor-corrector
More informationEE364a Homework 8 solutions
EE364a, Winter 2007-08 Prof. S. Boyd EE364a Homework 8 solutions 9.8 Steepest descent method in l -norm. Explain how to find a steepest descent direction in the l -norm, and give a simple interpretation.
More information8 Numerical methods for unconstrained problems
8 Numerical methods for unconstrained problems Optimization is one of the important fields in numerical computation, beside solving differential equations and linear systems. We can see that these fields
More informationDescent methods. min x. f(x)
Gradient Descent Descent methods min x f(x) 5 / 34 Descent methods min x f(x) x k x k+1... x f(x ) = 0 5 / 34 Gradient methods Unconstrained optimization min f(x) x R n. 6 / 34 Gradient methods Unconstrained
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More information6. Proximal gradient method
L. Vandenberghe EE236C (Spring 2016) 6. Proximal gradient method motivation proximal mapping proximal gradient method with fixed step size proximal gradient method with line search 6-1 Proximal mapping
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationA function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3
Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is well-defined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all
More informationPrimal-Dual Interior-Point Methods
Primal-Dual Interior-Point Methods Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 Outline Today: Primal-dual interior-point method Special case: linear programming
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationLecture 4 - The Gradient Method Objective: find an optimal solution of the problem
Lecture 4 - The Gradient Method Objective: find an optimal solution of the problem min{f (x) : x R n }. The iterative algorithms that we will consider are of the form x k+1 = x k + t k d k, k = 0, 1,...
More informationSelected Topics in Optimization. Some slides borrowed from
Selected Topics in Optimization Some slides borrowed from http://www.stat.cmu.edu/~ryantibs/convexopt/ Overview Optimization problems are almost everywhere in statistics and machine learning. Input Model
More information10-725/ Optimization Midterm Exam
10-725/36-725 Optimization Midterm Exam November 6, 2012 NAME: ANDREW ID: Instructions: This exam is 1hr 20mins long Except for a single two-sided sheet of notes, no other material or discussion is permitted
More informationLecture 5: Gradient Descent. 5.1 Unconstrained minimization problems and Gradient descent
10-725/36-725: Convex Optimization Spring 2015 Lecturer: Ryan Tibshirani Lecture 5: Gradient Descent Scribes: Loc Do,2,3 Disclaimer: These notes have not been subjected to the usual scrutiny reserved for
More informationConstrained optimization: direct methods (cont.)
Constrained optimization: direct methods (cont.) Jussi Hakanen Post-doctoral researcher jussi.hakanen@jyu.fi Direct methods Also known as methods of feasible directions Idea in a point x h, generate a
More information5. Subgradient method
L. Vandenberghe EE236C (Spring 2016) 5. Subgradient method subgradient method convergence analysis optimal step size when f is known alternating projections optimality 5-1 Subgradient method to minimize
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationPrimal-dual Subgradient Method for Convex Problems with Functional Constraints
Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationChapter 2. Optimization. Gradients, convexity, and ALS
Chapter 2 Optimization Gradients, convexity, and ALS Contents Background Gradient descent Stochastic gradient descent Newton s method Alternating least squares KKT conditions 2 Motivation We can solve
More informationFrank-Wolfe Method. Ryan Tibshirani Convex Optimization
Frank-Wolfe Method Ryan Tibshirani Convex Optimization 10-725 Last time: ADMM For the problem min x,z f(x) + g(z) subject to Ax + Bz = c we form augmented Lagrangian (scaled form): L ρ (x, z, w) = f(x)
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationSecond Order Optimization Algorithms I
Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The
More informationLecture 3: Basics of set-constrained and unconstrained optimization
Lecture 3: Basics of set-constrained and unconstrained optimization (Chap 6 from textbook) Xiaoqun Zhang Shanghai Jiao Tong University Last updated: October 9, 2018 Optimization basics Outline Optimization
More informationSWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm. Convex Optimization. Computing and Software McMaster University
SWFR ENG 4TE3 (6TE3) COMP SCI 4TE3 (6TE3) Continuous Optimization Algorithm Convex Optimization Computing and Software McMaster University General NLO problem (NLO : Non Linear Optimization) (N LO) min
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationECE580 Fall 2015 Solution to Midterm Exam 1 October 23, Please leave fractions as fractions, but simplify them, etc.
ECE580 Fall 2015 Solution to Midterm Exam 1 October 23, 2015 1 Name: Solution Score: /100 This exam is closed-book. You must show ALL of your work for full credit. Please read the questions carefully.
More informationInterior-Point Methods for Linear Optimization
Interior-Point Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More information