Convex Optimization : Conic Versus Functional Form
|
|
- Abigayle Fitzgerald
- 5 years ago
- Views:
Transcription
1 Convex Optimization : Conic Versus Functional Form Erling D. Andersen MOSEK ApS, Fruebjergvej 3, Box 16, DK 2100 Copenhagen, Blog: Linkedin: e.d.andersen@mosek.com WWW:
2 2 / 33
3 Topic Topic Considerations about the forms Conic optimization. A recap Basic cone types Comments Given a convex optimization problem then is the classical form min c(x) st a(x) 0 or the conic form min c T x st Ax = b, x K. preferable from an user perspective? Assumptions c(x) is convex anda(x) is concave. K is a convex cone. We can only solve convex problems efficiently. 3 / 33
4 Considerations about the forms Topic Considerations about the forms Conic optimization. A recap Basic cone types Comments Robustness. Accuracy of the solution. How sensitive is the optimizer to the problem data. Solution time. Does the form help the user to build a good. Does it prevent nonconvex problems because checking convexity is hard. Ease of use. 4 / 33
5 Conic optimization. A recap Topic Considerations about the forms Conic optimization. A recap Basic cone types Comments Primal problem: (CO P ) min c T x st Ax = b, x K 1 K 2 K k, where eachk k is a cone (closed, pointed and solid). Dual problem: (CO P ) max b T y st A T y +s = c, s K1 K 2 K k, where eachk k is the dual convex cone i.e. K k {s : x T s 0, x K}. 5 / 33
6 Basic cone types Topic Considerations about the forms Conic optimization. A recap Basic cone types Comments Linear: Quadratic: K q := K l := {x R : x 0} x Rn : x 1 n j=2 x 2 j 6 / 33
7 Comments Topic Considerations about the forms Conic optimization. A recap Basic cone types Comments We will only deal with the above 2 self-dual cones. A quote from Stephen Boyd of Stanford: Almost all convex optimization problem can be formulated using a combination of linear, quadratic, semi-definite and the exponential cones. See also the book of Ben-Tal and Nemirovski for conic quadratic representable sets. 7 / 33
8 8 / 33
9 Convex quadratic optimization Convex quadratic optimization continued Practical issues Functional formulation: min 1 2 xt Qx+c T x st Ax = b, x 0. Given convexity then there exists a H such that Q = HH T. Conic reformulation min 1 2 t+ct x st Ax = b, H T x y = 0, s = 1, y 2 2st, x 0. 9 / 33
10 continued Convex quadratic optimization continued Practical issues Conic reformulation requires more constraints and variables. The addition is highly structured and sparse. Conic form is convex by construction. Q is frequently negative semi-definite due to rounding errors or mistakes. How to deal with: [ ǫ ]? In practiceqis frequently low rank and dense. Implying the conic reformulation requires much less space to store. 10 / 33
11 Practical issues Convex quadratic optimization continued Practical issues Many users of the nonlinear capabilities in MOSEK: Do believe their nonlinear problems are convex. Do believe convexity is not important. Almost always make mistakes. (Errors in function or gradient computations). Checking and verifying convexity is an issue even for. Checking convexity and smoothness for arbitrary black box functions are hard. Data for the conic form is very simple. Just vectors, matrices and list of indexes. 11 / 33
12 12 / 33
13 The conic s The available s determines which form is preferable. Only primal-dual interior-point s are considered. They are efficient in practice. Polynomial complexity holds: For some. All conic. Claim: The conic form is more efficient and robust than functional form. Justification of claim: Follows. 13 / 33
14 functional form The conic s Problem: Assumptions: 1. x R n min c(x) st a i (x) 0, i, x c is convex and twice differentiable. 3. a i is concave and twice differentiable. 14 / 33
15 The conic s Lagrange function wherey 0. The dual problem: s R n. Alternatively: L(x,y) c(x) y T a(x), max L(x,y) x T s st (x) L(x,y) T = s, x,y,s 0, max L(x,y) x L(x,y)x st x L(x,y) T = s, x,y,s / 33
16 The conic s (MCP) min (x;y) T (s;z) st a(x) = z, st x L(x,y) T = s, x,z,y,s 0. wherez R m is slack variables. Note: The KKT is a monotone complementarity problem. Primal-dual interior-point type. Difficulties with detecting infeasibility! Feasibility constraints are potentially highly nonlinear! 16 / 33
17 The conic s denoted (HMCP): min (x;τ) T (s;κ) st τ a(x/τ) = z, primal feasibility τ x L(x/τ,y/τ) T = s, dual feasibility, x T x L(x/τ,y/τ) T y T a(x/τ) = κ, zero gap, x,z,τ,s,y,κ 0. Has two additional variablesτ,κ and one additional constraint. (HMCP) is a homogeneous monotone complementarity problem. Always has solution (0). 17 / 33
18 The conic s Theorem: (Andersen and Ye 95) Let (x,z,y,τ,s,κ ) be a maximal complementarity solution to (HMCP). Then (MCP) has a solution if and only if τ > 0. In this case, (x,z,y,s )/τ is a complementarity solution for(mcp). Conclusion: Compute a maximal solution to (HMCP). 18 / 33
19 The conic s Algorithm: Apply a primal-dual interior-point method to (HMCP) i.e. apply Newtons method to the perturbed optimality : τ x L(x/τ,y/τ) T = s, τa(x/τ) = z, x T x L(x/τ,y/τ) T y T a(x/τ) = κ, Xs = µe, Zy = µe, τκ = µ, x,z,τ,s,y,κ > 0. µ > 0: Is a parameter. Primal feasibility cond. is nonlinear. Dual feasibility cond. is nonlinear. Perturbed complementarity cond. is nonlinear. 19 / 33
20 The Step 0: Choose (x;z;τ;y;s;κ) > 0. Step 1: Choose parameters. Step 2: Compute the residuals: The where r r D r G r P s τg κ+x T g y T a(x/τ) z τa(x/τ) w (x;z;τ;y;s;κ) J a(x/τ), g x L(x/τ,y/τ) T H 2 xl(x/τ,y/τ)., conic s 20 / 33
21 The conic s Step 3: Solve the Newton equation system: and where H v2 J T (v 1 ) T H G (v 3 ) T J v 3 0 v 1 v 2 v 3 H g d x d τ d y d s d κ d z Sd x +Xd s = Xs+γµe, Yd z +Zd y = Zy +γµe, κd τ +τd κ = τκ+γµ, c(x/τ) T Hx/τ, c(x/τ) T Hx/τ, a(x/τ) Jx/τ, x/τ T H(x/τ), µ (x;z;τ) T (s;y;κ)/(n+m+1). = ηr 21 / 33
22 The conic s η = (γ 1) whereγ [0,1] is an ic parameters. Step 4: Update. Update of slacks: or where (x + ;τ + ;y + ;) = (x;τ;y)+α(d x ;d τ ;d y ) (z + ;s + ;κ + ) = (z +αd z ;s+αd s ;κ+αd κ ) z + = ((1+αη)r P +τ + a(x + /τ + ), s + = ((1+αη)r D +g + ), κ + = ((1+αη)r G +g + κ, g + τ + x L(x + /τ +,y + /τ + ) T g κ + (x + ) T x L(x + /τ +,y + /τ + ) T (y + ) T a(x + /τ + ) 22 / 33
23 The conic s Step 5: Repeat step 2 to 4. Issues: Gap and residuals are reduced by the factor(1+αη) in every iteration (given the nonlinear ). Polynomial complexity holds under certain assumptions. OBSERVE: z + = ((1+αη)r P +τ + a(x + /τ + ), s + = ((1+αη)r D +g + ). κ + = ((1+αη)r G +g + κ). Does not always work well!!! 23 / 33
24 The conic s Alternative. A linear z + = x+αd z, s + = s+αd s κ + = κ+αd κ but then a merit function is required e.g. two norm of the residuals. Works most of the time. Can lead to very slow convergence and robustness problems. A major weakness. Nonlinear primal and dual feasibility constraints are the cause of the problems. 24 / 33
25 conic quadratic problems The conic s : (H) Ax bτ = 0, (primal feasibility) A T y +s cτ = 0, (dual feasibility) c T x+b T y κ = 0, (zero gap) (x;τ) K,(s;κ) K In linear case suggested suggested Goldman and Tucker. Later rediscovered by Mizuno, Todd and Ye. All solutions satisfies: x T s+τκ = 0. If and only ifτ > 0, then an optimal solution exists. Find a solution such that τ +κ > 0. Linear primal and dual feasibility!!! Algorithmic framework: The Nesterov-Todd direction. 25 / 33
26 The conic s NT : Ad x bd τ A T d y +d s cd τ c T d x +b T d y d κ XT(ΘW) 1 d s + STΘWd x τd κ +κd τ whereη := γ 1 andγ [0,1). New iterate: x + x τ + τ y + = y s + s κ + κ = η(ax bτ), = η(a T y +s cτ), = η( c T x+b T y κ), = X Se+γµe, = τκ+γµ. +α d x d τ d y d s d κ. 26 / 33
27 The conic s Fact: Ax + bτ + = (1+αη)(Ax bτ), A T y + +s + cτ + = (1+αη)(A T y +s cτ), c T x + +b T y + κ + = (1+αη)( c T x+b T y κ), d T xd T s +d τ d κ = 0, (x + ) T s + +τ + κ + = (1+αη)((x) T s+τκ). Observations: NT can be seen as Newtons method. The method is symmetric! The complementarity gap is reduced by a factor of (1+αη) [0,1). The infeasibility is reduced by the same factor. No merit function is required. 27 / 33
28 s The conic s Both s are called primal-dual interior-point. Only in the conic form are the feasibility constraints linear. The functional one is more like primal the KKT system The conic is truly symmetric i.e. you can flip the primal and dual problem. Due to issues with the merit function then the functional form is not as robust as conic one. 28 / 33
29 The conic s Conic form: Functional form: min st min st y x = y x y, x R y x = y x 2 y y, y 0, x,y R 29 / 33
30 MOSEK v6 on the conic formulation The conic s ITE PFEAS DFEAS KAP/TAU POBJ DOBJ MU 0 1.0e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e / 33
31 MOSEK v6 on the ulation The conic s ITE PFEAS DFEAS KAP/TAU POBJ DOBJ MU 0 1.0e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e-012 Performance on the ulation is bad. Merit function forces the alg. to take small steps. Behavior seen occasionally on large instances. Other interior-point optimizers may do better of course. 31 / 33
32 Summary The conic s The (restricted) conic formulation of nonlinear problems is advantageous because The problem data is simpler to deal in a software sense and no blackbox functions are required. It is impossible to formulate nonconvex problems. The available is much better. It easy to dualize the conic formulation. Caveats: The conic formulation is usually bigger in terms of number of variables and constraints but also highly structured. Usually the storage and time spend per iteration will be same or less for the conic formulation. 32 / 33
33 Conclusion The conic form is better than the! The conic s 33 / 33
On implementing a primal-dual interior-point method for conic quadratic optimization
On implementing a primal-dual interior-point method for conic quadratic optimization E. D. Andersen, C. Roos, and T. Terlaky December 18, 2000 Abstract Conic quadratic optimization is the problem of minimizing
More information15. Conic optimization
L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone
More informationConic Linear Optimization and its Dual. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #04 1 Conic Linear Optimization and its Dual Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.
More informationLecture 5. Theorems of Alternatives and Self-Dual Embedding
IE 8534 1 Lecture 5. Theorems of Alternatives and Self-Dual Embedding IE 8534 2 A system of linear equations may not have a solution. It is well known that either Ax = c has a solution, or A T y = 0, c
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationIMPLEMENTATION OF INTERIOR POINT METHODS
IMPLEMENTATION OF INTERIOR POINT METHODS IMPLEMENTATION OF INTERIOR POINT METHODS FOR SECOND ORDER CONIC OPTIMIZATION By Bixiang Wang, Ph.D. A Thesis Submitted to the School of Graduate Studies in Partial
More informationNew stopping criteria for detecting infeasibility in conic optimization
Optimization Letters manuscript No. (will be inserted by the editor) New stopping criteria for detecting infeasibility in conic optimization Imre Pólik Tamás Terlaky Received: March 21, 2008/ Accepted:
More informationSolving conic optimization problems with MOSEK
Solving conic optimization problems with MOSEK Erling D. Andersen MOSEK ApS, Fruebjergvej 3, Box 16, 2100 Copenhagen, Denmark. WWW: http://www.mosek.com August 8, 2013 http://www.mosek.com 2 / 42 Outline
More informationInterior-Point Methods
Interior-Point Methods Stephen Wright University of Wisconsin-Madison Simons, Berkeley, August, 2017 Wright (UW-Madison) Interior-Point Methods August 2017 1 / 48 Outline Introduction: Problems and Fundamentals
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationWarmstarting the Homogeneous and Self-Dual Interior Point Method for Linear and Conic Quadratic Problems
Mathematical Programming Computation manuscript No. (will be inserted by the editor) Warmstarting the Homogeneous and Self-Dual Interior Point Method for Linear and Conic Quadratic Problems Anders Skajaa
More informationAn Infeasible Interior-Point Algorithm with full-newton Step for Linear Optimization
An Infeasible Interior-Point Algorithm with full-newton Step for Linear Optimization H. Mansouri M. Zangiabadi Y. Bai C. Roos Department of Mathematical Science, Shahrekord University, P.O. Box 115, Shahrekord,
More informationA Generalized Homogeneous and Self-Dual Algorithm. for Linear Programming. February 1994 (revised December 1994)
A Generalized Homogeneous and Self-Dual Algorithm for Linear Programming Xiaojie Xu Yinyu Ye y February 994 (revised December 994) Abstract: A generalized homogeneous and self-dual (HSD) infeasible-interior-point
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationInterior Point Methods for Mathematical Programming
Interior Point Methods for Mathematical Programming Clóvis C. Gonzaga Federal University of Santa Catarina, Florianópolis, Brazil EURO - 2013 Roma Our heroes Cauchy Newton Lagrange Early results Unconstrained
More information18. Primal-dual interior-point methods
L. Vandenberghe EE236C (Spring 213-14) 18. Primal-dual interior-point methods primal-dual central path equations infeasible primal-dual method primal-dual method for self-dual embedding 18-1 Symmetric
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationSolving conic optimization problems via self-dual embedding and facial reduction: a unified approach
Solving conic optimization problems via self-dual embedding and facial reduction: a unified approach Frank Permenter Henrik A. Friberg Erling D. Andersen August 18, 216 Abstract We establish connections
More informationNewton s Method. Ryan Tibshirani Convex Optimization /36-725
Newton s Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, Properties and examples: f (y) = max x
More informationSemidefinite Optimization using MOSEK
Semidefinite Optimization using MOSEK Joachim Dahl ISMP Berlin, August 23, 2012 http://www.mose.com Introduction Introduction Semidefinite optimization Algorithms Results and examples Summary MOSEK is
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLecture 15 Newton Method and Self-Concordance. October 23, 2008
Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationPrimal-Dual Interior-Point Methods. Javier Peña Convex Optimization /36-725
Primal-Dual Interior-Point Methods Javier Peña Convex Optimization 10-725/36-725 Last time: duality revisited Consider the problem min x subject to f(x) Ax = b h(x) 0 Lagrangian L(x, u, v) = f(x) + u T
More informationA full-newton step infeasible interior-point algorithm for linear programming based on a kernel function
A full-newton step infeasible interior-point algorithm for linear programming based on a kernel function Zhongyi Liu, Wenyu Sun Abstract This paper proposes an infeasible interior-point algorithm with
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book:
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationLecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming
Lecture 11 and 12: Penalty methods and augmented Lagrangian methods for nonlinear programming Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lecture 11 and
More informationInterior Point Methods in Mathematical Programming
Interior Point Methods in Mathematical Programming Clóvis C. Gonzaga Federal University of Santa Catarina, Brazil Journées en l honneur de Pierre Huard Paris, novembre 2008 01 00 11 00 000 000 000 000
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationA Second-Order Path-Following Algorithm for Unconstrained Convex Optimization
A Second-Order Path-Following Algorithm for Unconstrained Convex Optimization Yinyu Ye Department is Management Science & Engineering and Institute of Computational & Mathematical Engineering Stanford
More informationORIE 6326: Convex Optimization. Quasi-Newton Methods
ORIE 6326: Convex Optimization Quasi-Newton Methods Professor Udell Operations Research and Information Engineering Cornell April 10, 2017 Slides on steepest descent and analysis of Newton s method adapted
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationA path following interior-point algorithm for semidefinite optimization problem based on new kernel function. djeffal
Journal of Mathematical Modeling Vol. 4, No., 206, pp. 35-58 JMM A path following interior-point algorithm for semidefinite optimization problem based on new kernel function El Amir Djeffal a and Lakhdar
More informationBarrier Method. Javier Peña Convex Optimization /36-725
Barrier Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: Newton s method For root-finding F (x) = 0 x + = x F (x) 1 F (x) For optimization x f(x) x + = x 2 f(x) 1 f(x) Assume f strongly
More informationAnalytic Center Cutting-Plane Method
Analytic Center Cutting-Plane Method S. Boyd, L. Vandenberghe, and J. Skaf April 14, 2011 Contents 1 Analytic center cutting-plane method 2 2 Computing the analytic center 3 3 Pruning constraints 5 4 Lower
More informationNewton s Method. Javier Peña Convex Optimization /36-725
Newton s Method Javier Peña Convex Optimization 10-725/36-725 1 Last time: dual correspondences Given a function f : R n R, we define its conjugate f : R n R, f ( (y) = max y T x f(x) ) x Properties and
More informationResearch Note. A New Infeasible Interior-Point Algorithm with Full Nesterov-Todd Step for Semi-Definite Optimization
Iranian Journal of Operations Research Vol. 4, No. 1, 2013, pp. 88-107 Research Note A New Infeasible Interior-Point Algorithm with Full Nesterov-Todd Step for Semi-Definite Optimization B. Kheirfam We
More informationPrimal-Dual Interior-Point Methods for Linear Programming based on Newton s Method
Primal-Dual Interior-Point Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationMidterm Review. Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.
Midterm Review Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye (LY, Chapter 1-4, Appendices) 1 Separating hyperplane
More informationminimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1,
4 Duality 4.1 Numerical perturbation analysis example. Consider the quadratic program with variables x 1, x 2, and parameters u 1, u 2. minimize x 2 1 +2x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationAdvances in Convex Optimization: Theory, Algorithms, and Applications
Advances in Convex Optimization: Theory, Algorithms, and Applications Stephen Boyd Electrical Engineering Department Stanford University (joint work with Lieven Vandenberghe, UCLA) ISIT 02 ISIT 02 Lausanne
More informationPrimal-Dual Interior-Point Methods. Ryan Tibshirani Convex Optimization /36-725
Primal-Dual Interior-Point Methods Ryan Tibshirani Convex Optimization 10-725/36-725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,...
More informationLecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.
MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.
More informationApolynomialtimeinteriorpointmethodforproblemswith nonconvex constraints
Apolynomialtimeinteriorpointmethodforproblemswith nonconvex constraints Oliver Hinder, Yinyu Ye Department of Management Science and Engineering Stanford University June 28, 2018 The problem I Consider
More informationLinear Programming: Simplex
Linear Programming: Simplex Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Linear Programming: Simplex IMA, August 2016
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationPrimal-Dual Interior-Point Methods. Ryan Tibshirani Convex Optimization
Primal-Dual Interior-Point Methods Ryan Tibshirani Convex Optimization 10-725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,... m are
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationA SHIFTED PRIMAL-DUAL PENALTY-BARRIER METHOD FOR NONLINEAR OPTIMIZATION
A SHIFTED PRIMAL-DUAL PENALTY-BARRIER METHOD FOR NONLINEAR OPTIMIZATION Philip E. Gill Vyacheslav Kungurtsev Daniel P. Robinson UCSD Center for Computational Mathematics Technical Report CCoM-19-3 March
More informationPenalty and Barrier Methods General classical constrained minimization problem minimize f(x) subject to g(x) 0 h(x) =0 Penalty methods are motivated by the desire to use unconstrained optimization techniques
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationDetecting Infeasibility in Infeasible-Interior-Point Methods for Optimization
Detecting Infeasibility in Infeasible-Interior-Point Methods for Optimization M. J. Todd January 16, 2003 Abstract We study interior-point methods for optimization problems in the case of infeasibility
More informationInfeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization
Infeasibility Detection and an Inexact Active-Set Method for Large-Scale Nonlinear Optimization Frank E. Curtis, Lehigh University involving joint work with James V. Burke, University of Washington Daniel
More information1 A HOMOGENIZED CUTTING PLANE
OPTIMIZATION METHODS AND APPLICATIONS Published by KLUWER 1999 1 A HOMOGENIZED CUTTING PLANE METHOD TO SOLVE THE CONVEX FEASIBILITY PROBLEM E.D.Andersen,J.E.Mitchell,C.Roos,andT.Terlaky. Faculty of ITS,
More informationCourse Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979)
Course Outline FRTN Multivariable Control, Lecture Automatic Control LTH, 6 L-L Specifications, models and loop-shaping by hand L6-L8 Limitations on achievable performance L9-L Controller optimization:
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primal-dual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More informationPattern Classification, and Quadratic Problems
Pattern Classification, and Quadratic Problems (Robert M. Freund) March 3, 24 c 24 Massachusetts Institute of Technology. 1 1 Overview Pattern Classification, Linear Classifiers, and Quadratic Optimization
More informationAlgorithms for nonlinear programming problems II
Algorithms for nonlinear programming problems II Martin Branda Charles University Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects of Optimization
More informationOn Two Measures of Problem Instance Complexity and their Correlation with the Performance of SeDuMi on Second-Order Cone Problems
2016 Springer International Publishing AG. Part of Springer Nature. http://dx.doi.org/10.1007/s10589-005-3911-0 On Two Measures of Problem Instance Complexity and their Correlation with the Performance
More informationInterior Point Methods. We ll discuss linear programming first, followed by three nonlinear problems. Algorithms for Linear Programming Problems
AMSC 607 / CMSC 764 Advanced Numerical Optimization Fall 2008 UNIT 3: Constrained Optimization PART 4: Introduction to Interior Point Methods Dianne P. O Leary c 2008 Interior Point Methods We ll discuss
More informationOn self-concordant barriers for generalized power cones
On self-concordant barriers for generalized power cones Scott Roy Lin Xiao January 30, 2018 Abstract In the study of interior-point methods for nonsymmetric conic optimization and their applications, Nesterov
More information10. Unconstrained minimization
Convex Optimization Boyd & Vandenberghe 10. Unconstrained minimization terminology and assumptions gradient descent method steepest descent method Newton s method self-concordant functions implementation
More informationProximal Newton Method. Ryan Tibshirani Convex Optimization /36-725
Proximal Newton Method Ryan Tibshirani Convex Optimization 10-725/36-725 1 Last time: primal-dual interior-point method Given the problem min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h
More informationNumerical Methods for PDE-Constrained Optimization
Numerical Methods for PDE-Constrained Optimization Richard H. Byrd 1 Frank E. Curtis 2 Jorge Nocedal 2 1 University of Colorado at Boulder 2 Northwestern University Courant Institute of Mathematical Sciences,
More informationLecture: Algorithms for LP, SOCP and SDP
1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationSequential Quadratic Programming Method for Nonlinear Second-Order Cone Programming Problems. Hirokazu KATO
Sequential Quadratic Programming Method for Nonlinear Second-Order Cone Programming Problems Guidance Professor Masao FUKUSHIMA Hirokazu KATO 2004 Graduate Course in Department of Applied Mathematics and
More informationPrimal-dual Subgradient Method for Convex Problems with Functional Constraints
Primal-dual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primal-dual
More informationLMI MODELLING 4. CONVEX LMI MODELLING. Didier HENRION. LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ. Universidad de Valladolid, SP March 2009
LMI MODELLING 4. CONVEX LMI MODELLING Didier HENRION LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ Universidad de Valladolid, SP March 2009 Minors A minor of a matrix F is the determinant of a submatrix
More informationTutorial on Convex Optimization: Part II
Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation
More informationPrimal-Dual Interior-Point Methods
Primal-Dual Interior-Point Methods Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 Outline Today: Primal-dual interior-point method Special case: linear programming
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More information1. Gradient method. gradient method, first-order methods. quadratic bounds on convex functions. analysis of gradient method
L. Vandenberghe EE236C (Spring 2016) 1. Gradient method gradient method, first-order methods quadratic bounds on convex functions analysis of gradient method 1-1 Approximate course outline First-order
More informationConstraint Reduction for Linear Programs with Many Constraints
Constraint Reduction for Linear Programs with Many Constraints André L. Tits Institute for Systems Research and Department of Electrical and Computer Engineering University of Maryland, College Park Pierre-Antoine
More informationInterior Point Methods for Convex Quadratic and Convex Nonlinear Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationConvex Optimization and l 1 -minimization
Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationLecture 14: Primal-Dual Interior-Point Method
CSE 599: Interplay between Convex Optimization and Geometry Winter 218 Lecturer: Yin Tat Lee Lecture 14: Primal-Dual Interior-Point Method Disclaimer: Please tell me any mistake you noticed. In this and
More informationLecture 9 Sequential unconstrained minimization
S. Boyd EE364 Lecture 9 Sequential unconstrained minimization brief history of SUMT & IP methods logarithmic barrier function central path UMT & SUMT complexity analysis feasibility phase generalized inequalities
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationDescent methods. min x. f(x)
Gradient Descent Descent methods min x f(x) 5 / 34 Descent methods min x f(x) x k x k+1... x f(x ) = 0 5 / 34 Gradient methods Unconstrained optimization min f(x) x R n. 6 / 34 Gradient methods Unconstrained
More informationThe Q Method for Symmetric Cone Programmin
The Q Method for Symmetric Cone Programming The Q Method for Symmetric Cone Programmin Farid Alizadeh and Yu Xia alizadeh@rutcor.rutgers.edu, xiay@optlab.mcma Large Scale Nonlinear and Semidefinite Progra
More information