More First-Order Optimization Algorithms
|
|
- Norah Green
- 5 years ago
- Views:
Transcription
1 More First-Order Optimization Algorithms Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. yyye Chapters 3, 8, 3
2 The SDM for Unconstrained Convex Lipschitz Optimization Here we consider f(x) being convex and differentiable everywhere and satisfying the (first-order) β-lipschitz condition. Given the knowledge β, we again adopt the fixed step-size rule: x k+ = x k β f(xk ). () Theorem For convex Lipschitz optimization the Steepest Descent Method generates a sequence of solutions such that f(x k+ ) f(x ) β k + 2 x0 x 2 and where x is a minimizer of the problem. min l=0,...,k f(xl ) 2 = 4β 2 (k + )(k + 2) x0 x 2, Proof: For simplicity, we let δ k = f(x k ) f(x )( 0), g k = f(x k ), and k = x k x in the rest of proof. As we have proved for general Lipschitz optimization δ k+ δ k = f(x k+ ) f(x k ) 2β gk 2, that is δ k δ k+ 2β gk 2. (2) 2
3 Furthermore, from the convexity, δ k = f(x ) f(x k ) (g k ) T (x x k ) = (g k ) T k, that is δ k (g k ) T k. (3) Thus, from (2) and (3) δ k+ = δ k+ δ k + δ k 2β gk 2 + (g k ) T k = β 2 xk+ x k 2 β(x k+ x k ) k, (using ()) = β 2 ( xk+ x k 2 + 2(x k+ x k ) T k ) = β 2 ( k+ k 2 + 2( k+ k ) T k ) = β 2 ( k 2 k+ 2 ). (4) Sum up (4) from to k +, we have k+ l= δ l β 2 ( 0 2 k+ 2 ) β
4 From the proof of the Corollary of last lecture, we have δ 0 β Thus, and k+ k+ l=0 δ l β 0 2, (5) l=0 δl = k+ l=0 (l + l)δl = k+ l=0 (l + )δl k+ l=0 lδl = k+2 l= lδl k+ l= lδl = (k + 2)δ k+ + k+ l= lδl k+ l= lδl = (k + 2)δ k+ + k+ l= l(δl δ l ) (k + 2)δ k+ + k+ l= l 2β gl 2, where the first inequality comes from (2). Let g = min l=0,...,k g l. Then we finally have (k + 2)δ k+ + (k + )(k + 2)/2 g 2 β 0 2, (6) 2β which completes the proof. 4
5 The Accelerated Steepest Descent Method (ASDM) There is an accelerated steepest descent method (Nesterov 83) that works as follows: λ 0 = 0, λ k+ = + + 4(λ k ) 2 2, α k = λk, (7) λk+ x k+ = x k β f(xk ), x k+ = ( α k ) x k+ + α k x k. (8) Note that (λ k ) 2 = λ k+ (λ k+ ), λ k > k/2 and α k 0. One can prove: Theorem 2 f( x k+ ) f(x ) 2β k 2 x0 x 2, k. 5
6 Convergence Analysis of ASDM Again for simplification, we let k = λ k x k (λ k ) x k x, g k = f(x k ) and δ k = f( x k ) f(x )( 0) in the following. Applying Lemma for x = x k+ and y = x k, convexity of f and (8) we have δ k+ δ k = f( x k+ ) f(x k ) + f(x k ) f( x k ) β 2 xk+ x k 2 + f(x k ) f( x k ) β 2 xk+ x k 2 + (g k ) T (x k x k ) = β 2 xk+ x k 2 β( x k+ x k ) T (x k x k ). (9) Applying Lemma for x = x k+ and y = x, convexity of f and (8) we have δ k+ = f( x k+ ) f(x k ) + f(x k ) f(x ) β 2 xk+ x k 2 + f(x k ) f(x ) β 2 xk+ x k 2 + (g k ) T (x k x ) = β 2 xk+ x k 2 β( x k+ x k ) T (x k x ). (0) 6
7 Multiplying (9) by λ k (λ k ) and (0) by λ k respectively, and summing the two, we have (λ k ) 2 δ k+ (λ k ) 2 δ k (λ k ) 2 β 2 xk+ x k 2 λ k β( x k+ x k ) T k Using (7) and (8) we can derive = β 2 ((λk ) 2 x k+ x k 2 + 2λ k ( x k+ x k ) T k ) = β 2 ( λk x k+ (λ k ) x k x 2 k 2 ) = β 2 ( k 2 λ k x k+ (λ k ) x k x 2 ). λ k x k+ (λ k ) x k = λ k+ x k+ (λ k+ ) x k+. Thus, Sum up () from to k we have (λ k ) 2 δ k+ (λ k ) 2 δ k β 2 ( k 2 k+ 2.) () δ k+ β 2(λ k ) 2 2 2β k since λ k k/2 and 0. 7
8 First-Order Algorithms for Simply Constrained Optimization Consider the conic nonlinear optimization problem: min f(x) s.t. x K. Nonnegative Linear Regression: given data A R m n and b R m min f(x) = 2 Ax b 2 s.t. x 0; where f(x) = A T (Ax b). Semidefinite Linear Regression: given data A i S n for i =,..., m and b R m min f(x) = 2 AX b 2 s.t. X 0; where f(x) = A T (AX b). AX = A X... A m X and AT y = i= y i A i. 8
9 I: The Logarithmic Barrier Method One may add a barrier regularization: min ϕ(x) = f(x) + µb(x), where B(x) (or B(X)) is a barrier function keeping x (or X) in the interior of cone K and µ is a small positive barrier parameter. K = R n + : B(x) = j log(x j ), B(x) = x... x n Rn. K = S n + : B(X) = log(det(x)), B(X) = X S n. 9
10 Optimality Conditions for Optimization with Logarithmic Barrier Optimization in Nonnegative Cone (x > 0): ϕ(x) = f(x) µx e = 0, X = diag(x), and the scaled gradient vector: X ϕ(x) = X f(x) µe and it converges to zero implies f(x) > 0 for any positive µ. In practice, we set µ = ϵ tolerance. Optimization in Semidefinite Cone (X 0): ϕ(x) = f(x) µx = 0, and the scaled gradient matrix: X /2 ϕ(x)x /2 = X /2 f(x)x /2 µi and it converges to zero implies f(x) 0 for any positive µ (Suggested Project I) 0
11 II: SDM Followed by Feasible-Region-Projection ˆx k+ = x k β f(xk ) x k+ = Proj K (ˆx k+ ) For examples: if K = {x : x 0}, then x k+ = Proj K (ˆx k+ ) = max{0, ˆx k+ }. K = {X : X 0}, then factorize ˆX k+ = V T ΛV and let X k+ = Proj K ( ˆX k+ ) = V T max{0, Λ}V. The drawback is that the eigenvalue-factorization may be costly in each iteration (using L T DL factorization in Suggested Project I?)
12 Value-Iteration for MDP as Gradient-Ascent and Feasible-Region-Projection Let y R m represent the cost-to-go values of the m states, ith entry for ith state, of a given policy. The MDP problem entails choosing the optimal value vector y such that it satisfies: y i = min j A i {c j + γp T j y }, i, which can be casted as an optimal solution of the linear program max y b T y, s.t. y i c j + γp T j y, j A i, i, where b is any fixed positive vector. The Value-Iteration (VI) Method is, starting from any y 0, y k+ i = min j A i {c j + γp T j y k }, i. If the initial y 0 is strictly feasible for state i, that is, y 0 i < c j + γp T j y0, j A i, then y k i increasing in the VI iteration for all i and k. would be On the other hand, if any of the inequalities is violated, then we have to decrease y i min j Ai {c j + γp T j y0 }. at least to 2
13 Convergence of Value-Iteration for MDP Theorem 3 Let the VI algorithm mapping be A(v) i = min j Ai {c j + γp T j v, i}. Then, for any two value vectors u R m and v R m and every state i: A(u) i A(v) i γ u v, which implies A(u) i A(v) i γ u v Let j u and j v be the two arg min actions for value vectors u and v, respectively. Assume that A(u) i A(v) i 0 where the other case can be proved similarly. 0 A(u) i A(v) i = (c ju + γp T j u u) (c jv + γp T j v v) (c jv + γp T j v u) (c jv + γp T j v v) = γp T j v (u v) γ u v. where the first inequality is from that j u is the arg min action for value vector u, and the last inequality follows from the fact that the elements in p jv are non-negative and sum-up to. Many research issues in suggested Project II. 3
14 III: SDM with Affine Scaling Let an iterate solution x k > 0 (for simplicity let each entry of x k be bounded above by ). Then, we can scale it to e, the vector of all ones, by x = (X k ) x where X k is the diagonal matrix of vector x k. f (x ) = f(x k x ) where f (e) = X k f(x k ), the new SDM iterate would be x = e α k X k f(x k ) and after scaling back x k+ = x k α k (X k ) 2 f(x k ). If function f is β-lipschitz, then so is f with β x k 2, and we can fix α k = β x k 2 keep x k+ > 0) so that f(x k+ ) f(x k ) 2β x k 2 X k f(x k ) 2, (or smaller to so that the scaled gradient/complementarity vector X k f(x k )/ x k converging to zero. 4
15 Affine Scaling for SDP Cone Let an iterate solution X k 0. Then, one can scale it to I, the identity matrix, by X = (X k ) /2 X(X k ) /2. Now consider the objective function after scaling f (X ) = f((x k ) /2 X (X k ) /2 ) where f (I) = (X k ) /2 f(x k )(X k ) /2, the new SDM iterate would be X = I α k (X k ) /2 f(x k )(X k ) /2 and after scaling back X k+ = X k α k X k f(x k )X k. If function f is β-lipschitz, then so is f with with β X k 2,, and we can fix α k = β X k 2 to keep X k+ 0) so that f(x k+ ) f(x k ) 2β X k 2 (X k ) /2 f(x k )(X k ) /2 2, (or smaller so that the scaled gradient/complementarity matrix converging to zero... 5
16 IV: Gradient-Projection Method for Conic Optimization Consider the nonnegative cone. At any iterate solution x k 0, we project the gradient vector to the feasible direction space at x k : g k j = f(x k ) j if x k j > 0 or f(xk ) j < 0, 0 otherwise. Then we apply the SDM with the projected gradient vector g k, that is, take a largest stepsize α such that x k+ = x k αg k 0, 0 α β. Does it converge? What s the relation with the type II method? What is the convergence speed? (Consider it a bonus question to Problem 6 of HW3.) 6
17 V: Reduced Gradient Method the Simplex Algorithm for LP LP: min c T x s.t. Ax = b, x 0, where A R m n has a full row rank m. Theorem 4 (The Fundamental Theorem of LP in Algebraic form) Given (LP) and (LD) where A has full row rank m, i) if there is a feasible solution, there is a basic feasible solution (Carathéodory s theorem); ii) if there is an optimal solution, there is an optimal basic solution. High-Level Idea:. Initialization Start at a BSF or corner point of the feasible polyhedron. 2. Test for Optimality. Compute the reduced gradient vector at the corner. If no descent and feasible direction can be found, stop and claim optimality at the current corner point; otherwise, select a new corner point and go to Step 2. 7
18 LP theorems depicted in two variable space c a a2 a2 a3 a3 a4 a4 Objective contour a5 Figure : The LP Simplex Method 8
19 When a Basic Feasible Solution is Optimal Suppose the basis of a basic feasible solution is A B and the rest is A N. One can transform the equality constraint to A B Ax = A B b, so that x B = A B b A B A Nx N. That is, we express x B in terms of x N, the non-basic variables are are active for constraints x 0. Then the objective function equivalently becomes c T x = c T Bx B + c T Nx N = c T BA B b ct BA B A N x N + c T Nx N = c T BA B b + (ct N c T BA B A N )x N. Vector r T = c T c T B A B A is called the Reduced Gradient/Cost Vector where r B = 0 always. Theorem 5 If Reduced Gradient Vector r T = c T c T B A B A 0, then the BFS is optimal. Proof: Let y T = c T B A B (called Shadow Price Vector), then y is a dual feasible solution (r = c A T y 0) and c T x = c T B x B = c T B A B b = yt b, that is, the duality gap is zero. 9
20 The Simplex Algorithm Procedures 0. Initialize Start a BFS with basic index set B and let N denote the complementary index set.. Test for Optimality: Compute the Reduced Gradient Vector r at the current BFS and let If r e 0, stop the current BFS is optimal. r e = min j N {r j }. 2. Determine the Replacement: Increase x e while keep all other non-basic variables at the zero value (inactive) and maintain the equality constraints: x B = A B b A B A.ex e ( 0). If x e can be increased to, stop the problem is unbounded below. Otherwise, let the basic variable x o be the one first becoming Update basis: update B with x o being replaced by x e, and return to Step. 20
21 A Toy Example minimize x 2x 2 subject to x +x 3 = A = x 2 +x 4 = x +x 2 +x 5 =.5., b =.5, ct = ( ). Consider initial BFS with basic variables B = {3, 4, 5} and N = {, 2}. Iteration :. A B = I, A B = I, yt = (0 0 0) and r N = ( 2) it s NOT optimal. Let e = 2. 2
22 2. Increase x 2 while x B = A B b A B A.2x 2 =.5 0 x 2. We see x 4 becomes 0 first. 3. The new basic variables are B = {3, 2, 5} and N = {, 4}. Iteration 2:. A B = , A B = , y T = (0 2 0) and r N = ( 2) it s NOT optimal. Let e =. 22
23 2. Increase x while x B = A B b A B A.x = x. We see x 5 becomes 0 first. 3. The new basic variables are B = {3, 2, } and N = {4, 5}. Iteration 3:. A B = , A B = 0 0 0, y T = (0 ) and r N = ( ) it s Optimal. Is the Simplex Method always convergent to a minimizer? Which condition of the Global Convergence Theorem failed? 23
Yinyu Ye, MS&E, Stanford MS&E310 Lecture Note #06. The Simplex Method
The Simplex Method Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye (LY, Chapters 2.3-2.5, 3.1-3.4) 1 Geometry of Linear
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationConic Linear Optimization and its Dual. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #04 1 Conic Linear Optimization and its Dual Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.
More informationMidterm Review. Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.
Midterm Review Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye (LY, Chapter 1-4, Appendices) 1 Separating hyperplane
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationSecond Order Optimization Algorithms I
Second Order Optimization Algorithms I Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 7, 8, 9 and 10 1 The
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationLecture 15 Newton Method and Self-Concordance. October 23, 2008
Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications
More informationNumerical Optimization
Linear Programming - Interior Point Methods Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Example 1 Computational Complexity of Simplex Algorithm
More informationA Review of Linear Programming
A Review of Linear Programming Instructor: Farid Alizadeh IEOR 4600y Spring 2001 February 14, 2001 1 Overview In this note we review the basic properties of linear programming including the primal simplex
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationPrimal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization
Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationLecture: Algorithms for LP, SOCP and SDP
1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More informationInterior Point Methods for Mathematical Programming
Interior Point Methods for Mathematical Programming Clóvis C. Gonzaga Federal University of Santa Catarina, Florianópolis, Brazil EURO - 2013 Roma Our heroes Cauchy Newton Lagrange Early results Unconstrained
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationMathematical Optimization Models and Applications
Mathematical Optimization Models and Applications Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapters 1, 2.1-2,
More informationSemidefinite Programming
Chapter 2 Semidefinite Programming 2.0.1 Semi-definite programming (SDP) Given C M n, A i M n, i = 1, 2,..., m, and b R m, the semi-definite programming problem is to find a matrix X M n for the optimization
More informationInterior Point Methods in Mathematical Programming
Interior Point Methods in Mathematical Programming Clóvis C. Gonzaga Federal University of Santa Catarina, Brazil Journées en l honneur de Pierre Huard Paris, novembre 2008 01 00 11 00 000 000 000 000
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationOperations Research Lecture 4: Linear Programming Interior Point Method
Operations Research Lecture 4: Linear Programg Interior Point Method Notes taen by Kaiquan Xu@Business School, Nanjing University April 14th 2016 1 The affine scaling algorithm one of the most efficient
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More information6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE. Three Alternatives/Remedies for Gradient Projection
6.252 NONLINEAR PROGRAMMING LECTURE 10 ALTERNATIVES TO GRADIENT PROJECTION LECTURE OUTLINE Three Alternatives/Remedies for Gradient Projection Two-Metric Projection Methods Manifold Suboptimization Methods
More information1. Algebraic and geometric treatments Consider an LP problem in the standard form. x 0. Solutions to the system of linear equations
The Simplex Method Most textbooks in mathematical optimization, especially linear programming, deal with the simplex method. In this note we study the simplex method. It requires basically elementary linear
More informationA Distributed Newton Method for Network Utility Maximization, II: Convergence
A Distributed Newton Method for Network Utility Maximization, II: Convergence Ermin Wei, Asuman Ozdaglar, and Ali Jadbabaie October 31, 2012 Abstract The existing distributed algorithms for Network Utility
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationECE580 Exam 1 October 4, Please do not write on the back of the exam pages. Extra paper is available from the instructor.
ECE580 Exam 1 October 4, 2012 1 Name: Solution Score: /100 You must show ALL of your work for full credit. This exam is closed-book. Calculators may NOT be used. Please leave fractions as fractions, etc.
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationLecture 7: September 17
10-725: Optimization Fall 2013 Lecture 7: September 17 Lecturer: Ryan Tibshirani Scribes: Serim Park,Yiming Gu 7.1 Recap. The drawbacks of Gradient Methods are: (1) requires f is differentiable; (2) relatively
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationThe Simplex and Policy Iteration Methods are Strongly Polynomial for the Markov Decision Problem with Fixed Discount
The Simplex and Policy Iteration Methods are Strongly Polynomial for the Markov Decision Problem with Fixed Discount Yinyu Ye Department of Management Science and Engineering and Institute of Computational
More informationLecture: Introduction to LP, SDP and SOCP
Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement:
More informationOn self-concordant barriers for generalized power cones
On self-concordant barriers for generalized power cones Scott Roy Lin Xiao January 30, 2018 Abstract In the study of interior-point methods for nonsymmetric conic optimization and their applications, Nesterov
More informationPart 4: Active-set methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Active-set methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationPenalty and Barrier Methods. So we again build on our unconstrained algorithms, but in a different way.
AMSC 607 / CMSC 878o Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 3: Penalty and Barrier Methods Dianne P. O Leary c 2008 Reference: N&S Chapter 16 Penalty and Barrier
More informationNotes taken by Graham Taylor. January 22, 2005
CSC4 - Linear Programming and Combinatorial Optimization Lecture : Different forms of LP. The algebraic objects behind LP. Basic Feasible Solutions Notes taken by Graham Taylor January, 5 Summary: We first
More informationLinear Programming: Simplex
Linear Programming: Simplex Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Linear Programming: Simplex IMA, August 2016
More informationA Second-Order Path-Following Algorithm for Unconstrained Convex Optimization
A Second-Order Path-Following Algorithm for Unconstrained Convex Optimization Yinyu Ye Department is Management Science & Engineering and Institute of Computational & Mathematical Engineering Stanford
More informationWritten Examination
Division of Scientific Computing Department of Information Technology Uppsala University Optimization Written Examination 202-2-20 Time: 4:00-9:00 Allowed Tools: Pocket Calculator, one A4 paper with notes
More information3 The Simplex Method. 3.1 Basic Solutions
3 The Simplex Method 3.1 Basic Solutions In the LP of Example 2.3, the optimal solution happened to lie at an extreme point of the feasible set. This was not a coincidence. Consider an LP in general form,
More informationDiscrete Optimization
Prof. Friedrich Eisenbrand Martin Niemeier Due Date: April 15, 2010 Discussions: March 25, April 01 Discrete Optimization Spring 2010 s 3 You can hand in written solutions for up to two of the exercises
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More informationOptimization methods
Lecture notes 3 February 8, 016 1 Introduction Optimization methods In these notes we provide an overview of a selection of optimization methods. We focus on methods which rely on first-order information,
More informationAnalytic Center Cutting-Plane Method
Analytic Center Cutting-Plane Method S. Boyd, L. Vandenberghe, and J. Skaf April 14, 2011 Contents 1 Analytic center cutting-plane method 2 2 Computing the analytic center 3 3 Pruning constraints 5 4 Lower
More informationChap6 Duality Theory and Sensitivity Analysis
Chap6 Duality Theory and Sensitivity Analysis The rationale of duality theory Max 4x 1 + x 2 + 5x 3 + 3x 4 S.T. x 1 x 2 x 3 + 3x 4 1 5x 1 + x 2 + 3x 3 + 8x 4 55 x 1 + 2x 2 + 3x 3 5x 4 3 x 1 ~x 4 0 If we
More information1 Review Session. 1.1 Lecture 2
1 Review Session Note: The following lists give an overview of the material that was covered in the lectures and sections. Your TF will go through these lists. If anything is unclear or you have questions
More informationA priori bounds on the condition numbers in interior-point methods
A priori bounds on the condition numbers in interior-point methods Florian Jarre, Mathematisches Institut, Heinrich-Heine Universität Düsseldorf, Germany. Abstract Interior-point methods are known to be
More informationProject Discussions: SNL/ADMM, MDP/Randomization, Quadratic Regularization, and Online Linear Programming
Project Discussions: SNL/ADMM, MDP/Randomization, Quadratic Regularization, and Online Linear Programming Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305,
More informationConvex Optimization and l 1 -minimization
Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationUses of duality. Geoff Gordon & Ryan Tibshirani Optimization /
Uses of duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Remember conjugate functions Given f : R n R, the function is called its conjugate f (y) = max x R n yt x f(x) Conjugates appear
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primal-dual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More information3E4: Modelling Choice. Introduction to nonlinear programming. Announcements
3E4: Modelling Choice Lecture 7 Introduction to nonlinear programming 1 Announcements Solutions to Lecture 4-6 Homework will be available from http://www.eng.cam.ac.uk/~dr241/3e4 Looking ahead to Lecture
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More information10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation
More informationCONSTRAINED NONLINEAR PROGRAMMING
149 CONSTRAINED NONLINEAR PROGRAMMING We now turn to methods for general constrained nonlinear programming. These may be broadly classified into two categories: 1. TRANSFORMATION METHODS: In this approach
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationF 1 F 2 Daily Requirement Cost N N N
Chapter 5 DUALITY 5. The Dual Problems Every linear programming problem has associated with it another linear programming problem and that the two problems have such a close relationship that whenever
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationIE 5531 Midterm #2 Solutions
IE 5531 Midterm #2 s Prof. John Gunnar Carlsson November 9, 2011 Before you begin: This exam has 9 pages and a total of 5 problems. Make sure that all pages are present. To obtain credit for a problem,
More informationSummary of the simplex method
MVE165/MMG630, The simplex method; degeneracy; unbounded solutions; infeasibility; starting solutions; duality; interpretation Ann-Brith Strömberg 2012 03 16 Summary of the simplex method Optimality condition:
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationReview Solutions, Exam 2, Operations Research
Review Solutions, Exam 2, Operations Research 1. Prove the weak duality theorem: For any x feasible for the primal and y feasible for the dual, then... HINT: Consider the quantity y T Ax. SOLUTION: To
More informationNOTES ON FIRST-ORDER METHODS FOR MINIMIZING SMOOTH FUNCTIONS. 1. Introduction. We consider first-order methods for smooth, unconstrained
NOTES ON FIRST-ORDER METHODS FOR MINIMIZING SMOOTH FUNCTIONS 1. Introduction. We consider first-order methods for smooth, unconstrained optimization: (1.1) minimize f(x), x R n where f : R n R. We assume
More informationInterior Point Methods: Second-Order Cone Programming and Semidefinite Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:
More informationLecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.
MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.
More informationLecture 5. The Dual Cone and Dual Problem
IE 8534 1 Lecture 5. The Dual Cone and Dual Problem IE 8534 2 For a convex cone K, its dual cone is defined as K = {y x, y 0, x K}. The inner-product can be replaced by x T y if the coordinates of the
More informationPart 3: Trust-region methods for unconstrained optimization. Nick Gould (RAL)
Part 3: Trust-region methods for unconstrained optimization Nick Gould (RAL) minimize x IR n f(x) MSc course on nonlinear optimization UNCONSTRAINED MINIMIZATION minimize x IR n f(x) where the objective
More informationCSCI 1951-G Optimization Methods in Finance Part 01: Linear Programming
CSCI 1951-G Optimization Methods in Finance Part 01: Linear Programming January 26, 2018 1 / 38 Liability/asset cash-flow matching problem Recall the formulation of the problem: max w c 1 + p 1 e 1 = 150
More informationLecture 9 Sequential unconstrained minimization
S. Boyd EE364 Lecture 9 Sequential unconstrained minimization brief history of SUMT & IP methods logarithmic barrier function central path UMT & SUMT complexity analysis feasibility phase generalized inequalities
More information3. THE SIMPLEX ALGORITHM
Optimization. THE SIMPLEX ALGORITHM DPK Easter Term. Introduction We know that, if a linear programming problem has a finite optimal solution, it has an optimal solution at a basic feasible solution (b.f.s.).
More informationPrimal-Dual Interior-Point Methods for Linear Programming based on Newton s Method
Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationCubic regularization of Newton s method for convex problems with constraints
CORE DISCUSSION PAPER 006/39 Cubic regularization of Newton s method for convex problems with constraints Yu. Nesterov March 31, 006 Abstract In this paper we derive efficiency estimates of the regularized
More informationGlobal Optimization of Polynomials
Semidefinite Programming Lecture 9 OR 637 Spring 2008 April 9, 2008 Scribe: Dennis Leventhal Global Optimization of Polynomials Recall we were considering the problem min z R n p(z) where p(z) is a degree
More informationLecture 17: Primal-dual interior-point methods part II
10-725/36-725: Convex Optimization Spring 2015 Lecture 17: Primal-dual interior-point methods part II Lecturer: Javier Pena Scribes: Pinchao Zhang, Wei Ma Note: LaTeX template courtesy of UC Berkeley EECS
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationTIM 206 Lecture 3: The Simplex Method
TIM 206 Lecture 3: The Simplex Method Kevin Ross. Scribe: Shane Brennan (2006) September 29, 2011 1 Basic Feasible Solutions Have equation Ax = b contain more columns (variables) than rows (constraints),
More informationHomework 4. Convex Optimization /36-725
Homework 4 Convex Optimization 10-725/36-725 Due Friday November 4 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationAdditional Exercises for Introduction to Nonlinear Optimization Amir Beck March 16, 2017
Additional Exercises for Introduction to Nonlinear Optimization Amir Beck March 16, 2017 Chapter 1 - Mathematical Preliminaries 1.1 Let S R n. (a) Suppose that T is an open set satisfying T S. Prove that
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 42 Subgradients Assumptions
More informationOn well definedness of the Central Path
On well definedness of the Central Path L.M.Graña Drummond B. F. Svaiter IMPA-Instituto de Matemática Pura e Aplicada Estrada Dona Castorina 110, Jardim Botânico, Rio de Janeiro-RJ CEP 22460-320 Brasil
More informationarxiv: v1 [math.oc] 1 Jul 2016
Convergence Rate of Frank-Wolfe for Non-Convex Objectives Simon Lacoste-Julien INRIA - SIERRA team ENS, Paris June 8, 016 Abstract arxiv:1607.00345v1 [math.oc] 1 Jul 016 We give a simple proof that the
More informationUnconstrained minimization
CSCI5254: Convex Optimization & Its Applications Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions 1 Unconstrained
More informationMultidisciplinary System Design Optimization (MSDO)
Multidisciplinary System Design Optimization (MSDO) Numerical Optimization II Lecture 8 Karen Willcox 1 Massachusetts Institute of Technology - Prof. de Weck and Prof. Willcox Today s Topics Sequential
More informationLecture Note 18: Duality
MATH 5330: Computational Methods of Linear Algebra 1 The Dual Problems Lecture Note 18: Duality Xianyi Zeng Department of Mathematical Sciences, UTEP The concept duality, just like accuracy and stability,
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Algorithms for Constrained Nonlinear Optimization Problems) Tamás TERLAKY Computing and Software McMaster University Hamilton, November 2005 terlaky@mcmaster.ca
More informationIntroduction and Math Preliminaries
Introduction and Math Preliminaries Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Appendices A, B, and C, Chapter
More informationChapter 6 Interior-Point Approach to Linear Programming
Chapter 6 Interior-Point Approach to Linear Programming Objectives: Introduce Basic Ideas of Interior-Point Methods. Motivate further research and applications. Slide#1 Linear Programming Problem Minimize
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationLine Search Methods for Unconstrained Optimisation
Line Search Methods for Unconstrained Optimisation Lecture 8, Numerical Linear Algebra and Optimisation Oxford University Computing Laboratory, MT 2007 Dr Raphael Hauser (hauser@comlab.ox.ac.uk) The Generic
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationConic Linear Programming. Yinyu Ye
Conic Linear Programming Yinyu Ye December 2004, revised January 2015 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture
More information