Lecture 18: Optimization Programming


 Earl Mitchell
 1 years ago
 Views:
Transcription
1 Fall, 2016
2 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equalityconstrained Optimization Inequalityconstrained Optimization Mixtureconstrained Optimization 3 Quadratic Programming (QP), Linear Programming (LP)
3 Unconstrained Optimization Consider the minimization problem min x f (x), where x R d, and f is continuously differentiable. Necessary condition for local minimums If x is a local minimum of f (x), then f (x ) = 0. A point satisfying the necessary condition is called a stationary point, which can be a local minimum, local maximum, or saddle point. Sufficient conditions for x be a local minimum are: f (x ) = 0, 2 f (x ) is positive definite.
4 Equality Inequality Mixture Consider the minimization problem min x f (x) subject to g i (x) 0, i = 1,, m. h j (x) = 0, j = 1,, k. Assume x R d. f : R d R is called the objective function g i : R d R, i = 1,, m are inequality constraints h j : R d R, j = 1,, k are equality constraints. Assume f, g i s, h j s are all continuously differentiable.
5 Single Equality Constraint Equality Inequality Mixture Consider the minimization problem Introduce the Lagrange function: Lagrange extreme value theory: min x f (x) (1) subject to h(x) = 0. (2) L(x, λ) = f (x) + λh(x) If x is a local minimum under the constraint, then there exist some λ such that x L(x, λ ) = 0, λ L(x, λ ) = 0.
6 Interpretations Unconstrained Optimization Equality Inequality Mixture Above conditions imply that (x, λ ) is a stationary point of L. The above equations are equivalent to x f (x ) + λ x h(x ) = 0, h(x ) = 0. Note the first condition gives d equations.
7 Multiple Equality Constraints Equality Inequality Mixture Consider the minimization problem min x f (x) (3) subject to h j (x) = 0, j = 1,, k. (4) Introduce multiple Lagrange multipliers: λ = (λ 1,, λ k ) L(x, λ) = f (x) + k λ j h j (x) Lagrange extreme theory: If x is a local minimum under the constraint, then there exist some λ = (λ 1,, λ k ) such that j=1 x L(x, λ ) = 0, λ L(x, λ ) = 0. In other words, (x, λ ) is a stationary point of L.
8 Equality Inequality Mixture Example: Entropy Maximization Problem Consider the minimization problem n min p i log(p i ), subject to p 1,,p n i=1 i=1 n p i = 1. i=1 The Lagrange function is n n L(p, λ) = p i log(p i ) + λ( p i 1). L p i = log(p i ) λ = 0, log(p i ) = 1 λ, i = 1,, n. Together with n i=1 p i = 1, we have p 1 = = p n = 1 n. This problem is known as maximizing information entropy. i=1
9 Generalized Lagrange Methods Consider the minimization problem Equality Inequality Mixture min x f (x) subject to g i (x) 0, i = 1,, m. Introduce multiple nonnegative Lagrange multipliers: µ 1 0,, µ m 0. m L(x, µ) = f (x) µ i g i (x). µ i s are also called dual variables or slack variables. Correspondingly, we call x as primal variables. We would like to minimize L with respect to x and maximize L with respect to µ. i=1
10 KKT Necessary Optimality Conditions Equality Inequality Mixture If x is a local minimum under the above inequality constraints, then there exist some µ = (µ 1,, µ m) such that Stationarity: x L(x, µ ) = 0, i.e., x f (x ) m µ k xg i (x ) = 0. i=1 Primal feasibility: g i (x ) 0 for i = 1,, m. Dual feasibility: µ i 0 for i = 1,, m. Complementary slackness: KKT = KarushKuhnTucker µ i g i (x ) = 0, i = 1,, m.
11 Sufficient Optimality Conditions Equality Inequality Mixture The above necessary conditions are also sufficient for optimality, if f, g i s are continuously differentiable f and g i s are convex functions. Under this case, There exists a solution x satisfying the KKT condition, x is actually a global optimum. Examples: SVM problems
12 Alternative Constraints Equality Inequality Mixture Consider the minimization problem min x f (x) subject to g i (x) 0, i = 1,, m. Introduce multiple nonnegative Lagrange multipliers: µ 1 0,, µ m 0. m L(x, µ) = f (x) + µ i g i (x). µ i s are also called dual variables or slack variables. Correspondingly, we call x as primal variables. We would like to minimize L with respect to x and maximize L with respect to µ. i=1
13 KKT Necessary Optimality Conditions Equality Inequality Mixture If x is a local minimum under the above inequality constraints, then there exist some µ = (µ 1,, µ m) such that Stationarity: x L(x, µ ) = 0, i.e., x f (x ) + m µ k xg i (x ) = 0. i=1 Primal feasibility: g i (x ) 0 for i = 1,, m. Dual feasibility: µ i 0 for i = 1,, m. Complementary slackness: µ i g i (x ) = 0, i = 1,, m.
14 Generalized Lagrange Methods Consider the minimization problem Equality Inequality Mixture min x f (x) subject to g i (x) 0, i = 1,, m. h j (x) = 0, j = 1,, k. Introduce two sets of Lagrange multipliers: Define the Lagrange function: µ 1 0,, µ m 0, λ 1,, λ k. L(x, µ, λ) = f (x) m µ i g i (x) + i=1 k λ j h j (x) j=1 µ i s and λ j s are dual variables or slack variables.
15 KKT Necessary Optimality Conditions Equality Inequality Mixture If x is a local minimum under the above inequality constraints, then there exist some (µ, λ ) such that Stationarity: x L(x, µ, λ ) = 0, i.e., x f (x ) Primal feasibility: m µ i x g i (x ) + i=1 k λ j x h j (x ) = 0. j=1 g i (x ) 0, i = 1,, m; h j (x ) = 0, j = 1,, k. Dual feasibility: µ i 0 for i = 1,, m. Complementary slackness: µ i g i (x ) = 0, i = 1,, m.
16 Sufficient Optimality Conditions Equality Inequality Mixture The above necessary conditions are also sufficient for optimality, if f, g i s are continuously differentiable. f and g i s are convex functions, h j s are linear constraints. Under this case, There exists a solution x satisfying the KKT condition, x is actually a global optimum.
17 Quadratic Programming (QP) Quadratic Programming (QP) The objective function is quadratic in x The constraints are linear in x. Consider the minimization problem min x f (x) = 1 2 xt Qx + c T x, subject to Ax b, Ex = d. where Q is a symmetric matrix of d d, A is a matrix of m d, E is a matrix of k d, c R d, b R m, and d R k. Example: SVM
18 Properties of QP Unconstrained Optimization Quadratic Programming (QP) If Q is a nonnegative definite matrix, then f (x) is convex. If Q is a nonnegative definite, the above QP has a global minimizer if there exists at least one x satisfying the constraints and f (x) is bounded below in the feasible region. If Q is a positive definite, the above QP has a unique global minimizer. If Q = 0, the problem becomes a linear programming (LP). From optimization theory, If Q is nonnegative definite, the necessary and sufficient condition for a point x to be a global minimizer is for it to satisfy the KKT condition.
19 Solving QP Unconstrained Optimization Quadratic Programming (QP) If there are only equality constraints, then the QP can be solved by a linear system. In general, a variety of methods for solving the QP are commonly used, including Methods: ellipsoid, interior point, active set, conjugate gradient, etc Software packages: Matlab, CPLEX, AMPL, GAMS, etc. For positive definite Q, the ellipsoid method solves the problem in polynomial time. In other words, the running time is upper bounded by a polynomial in the size of the input for the algorithm, i.e., T = O(d k ) for some k.
20 Dual Problem for QP Quadratic Programming (QP) Consider the minimization problem The Lagrange function. min f (x) = 1 x 2 xt Qx, subject to Ax b. L(x, λ) = f (x) + λ T (Ax b). We need to minimize L with respect to x and maximize L with respect to λ. The dual function is defined as a function of dual variables only by solving x first with any fixed λ: G(λ) = inf x L(x, λ).
21 Dual Problem Derivation for QP Quadratic Programming (QP) With fixed λ, we solve the equation L x (x, λ) = 0 and obtain x = Q 1 A T λ. Plug this into L, we get the dual function The dual problem of the QP is G(λ) = 1 2 λt AQ 1 A T b T λ. max G(λ) = 1 λ 2 λt AQ 1 A T b T λ, subject to λ 0. We first find λ = arg max λ G(λ), and then compute x = Q 1 A T λ. The idea is similar to the profile method.
5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationICSE4030 Kernel Methods in Machine Learning
ICSE4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt2016fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationSupport Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and FuJie Huang 1 Outline What is behind
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec  Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec  Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationOptimization. YuhJye Lee. March 28, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 40
Optimization YuhJye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 28, 2017 1 / 40 The Key Idea of Newton s Method Let f : R n R be a twice differentiable function
More informationCSE4830 Kernel Methods in Machine Learning
CSE4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationEE/AA 578, Univ of Washington, Fall Duality
7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationIntroduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module  5 Lecture  22 SVM: The Dual Formulation Good morning.
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationCE 191: Civil & Environmental Engineering Systems Analysis. LEC 17 : Final Review
CE 191: Civil & Environmental Engineering Systems Analysis LEC 17 : Final Review Professor Scott Moura Civil & Environmental Engineering University of California, Berkeley Fall 2014 Prof. Moura UC Berkeley
More informationConvex Optimization and Support Vector Machine
Convex Optimization and Support Vector Machine Problem 0. Consider a twoclass classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationThe KarushKuhnTucker conditions
Chapter 6 The KarushKuhnTucker conditions 6.1 Introduction In this chapter we derive the first order necessary condition known as KarushKuhnTucker (KKT) conditions. To this aim we introduce the alternative
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More informationConvex Optimization and SVM
Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence
More informationTutorial on Convex Optimization: Part II
Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470  Convex Optimization Fall 201718, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationKarushKuhnTucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36725
KarushKuhnTucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10725/36725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.14.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very wellwritten and a pleasure to read. The
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationLecture 2: Linear SVM in the Dual
Lecture 2: Linear SVM in the Dual Stéphane Canu stephane.canu@litislab.eu São Paulo 2015 July 22, 2015 Road map 1 Linear SVM Optimization in 10 slides Equality constraints Inequality constraints Dual formulation
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationSupport Vector Machines
Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationLecture Notes on Support Vector Machine
Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a ndimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is
More informationLecture 7: Convex Optimizations
Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LPproblem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationSupport vector machines
Support vector machines Guillaume Obozinski Ecole des Ponts  ParisTech SOCN course 2014 SVM, kernel methods and multiclass 1/23 Outline 1 Constrained optimization, Lagrangian duality and KKT 2 Support
More informationMore on Lagrange multipliers
More on Lagrange multipliers CE 377K April 21, 2015 REVIEW The standard form for a nonlinear optimization problem is min x f (x) s.t. g 1 (x) 0. g l (x) 0 h 1 (x) = 0. h m (x) = 0 The objective function
More informationLinear Support Vector Machine. Classification. Linear SVM. Huiping Cao. Huiping Cao, Slide 1/26
Huiping Cao, Slide 1/26 Classification Linear SVM Huiping Cao linear hyperplane (decision boundary) that will separate the data Huiping Cao, Slide 2/26 Support Vector Machines rt Vector Find a linear Machines
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationMachine Learning A Geometric Approach
Machine Learning A Geometric Approach CIML book Chap 7.7 Linear Classification: Support Vector Machines (SVM) Professor Liang Huang some slides from Alex Smola (CMU) Linear Separator Ham Spam From Perceptron
More information10701 Recitation 5 Duality and SVM. Ahmed Hefny
10701 Recitation 5 Duality and SVM Ahmed Hefny Outline Langrangian and Duality The Lagrangian Duality Eamples Support Vector Machines Primal Formulation Dual Formulation Soft Margin and Hinge Loss Lagrangian
More informationScientific Computing: Optimization
Scientific Computing: Optimization Aleksandar Donev Courant Institute, NYU 1 donev@courant.nyu.edu 1 Course MATHGA.2043 or CSCIGA.2112, Spring 2012 March 8th, 2011 A. Donev (Courant Institute) Lecture
More informationOptimization Problems with Constraints  introduction to theory, numerical Methods and applications
Optimization Problems with Constraints  introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationChap 2. Optimality conditions
Chap 2. Optimality conditions Version: 29092012 2.1 Optimality conditions in unconstrained optimization Recall the definitions of global, local minimizer. Geometry of minimization Consider for f C 1
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More informationMachine Learning. Support Vector Machines. Manfred Huber
Machine Learning Support Vector Machines Manfred Huber 2015 1 Support Vector Machines Both logistic regression and linear discriminant analysis learn a linear discriminant function to separate the data
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of WisconsinMadison May 2014 Wright (UWMadison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationMachine Learning. Lecture 6: Support Vector Machine. Feng Li.
Machine Learning Lecture 6: Support Vector Machine Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 Warm Up 2 / 80 Warm Up (Contd.)
More informationOn the Method of Lagrange Multipliers
On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should
More informationNumerical Optimization. Review: Unconstrained Optimization
Numerical Optimization Finding the best feasible solution Edward P. Gatzke Department of Chemical Engineering University of South Carolina Ed Gatzke (USC CHE ) Numerical Optimization ECHE 589, Spring 2011
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationNumerical Optimization
Constrained Optimization Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Constrained Optimization Constrained Optimization Problem: min h j (x) 0,
More informationLecture 13: Constrained optimization
20101203 Basic ideas A nonlinearly constrained problem must somehow be converted relaxed into a problem which we can solve (a linear/quadratic or unconstrained problem) We solve a sequence of such problems
More informationEE364a Review Session 5
EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phonein office hours: tuesdays 67pm (6507231156) 1 Complementary
More information12. Interiorpoint methods
12. Interiorpoint methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationSupport Vector Machine (SVM) and Kernel Methods
Support Vector Machine (SVM) and Kernel Methods CE717: Machine Learning Sharif University of Technology Fall 2016 Soleymani Outline Margin concept HardMargin SVM SoftMargin SVM Dual Problems of HardMargin
More informationPart 4: Activeset methods for linearly constrained optimization. Nick Gould (RAL)
Part 4: Activeset methods for linearly constrained optimization Nick Gould RAL fx subject to Ax b Part C course on continuoue optimization LINEARLY CONSTRAINED MINIMIZATION fx subject to Ax { } b where
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 function with f () on (, L) and that you have explicit formulae for
More informationSupport Vector Machines for Regression
COMP566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x
More informationTutorial on Convex Optimization for Engineers Part II
Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D98684 Ilmenau, Germany jens.steinwandt@tuilmenau.de
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationNonlinear Programming (Hillier, Lieberman Chapter 13) CHEME7155 Production Planning and Control
Nonlinear Programming (Hillier, Lieberman Chapter 13) CHEME7155 Production Planning and Control 19/4/2012 Lecture content Problem formulation and sample examples (ch 13.1) Theoretical background Graphical
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470  Convex Optimization Fall 201819, HKUST, Hong Kong Outline of Lecture Subgradients
More informationPart IB Optimisation
Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after
More informationThe Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:
HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 00 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLECTURE 7 Support vector machines
LECTURE 7 Support vector machines SVMs have been used in a multitude of applications and are one of the most popular machine learning algorithms. We will derive the SVM algorithm from two perspectives:
More informationAdditional Exercises for Introduction to Nonlinear Optimization Amir Beck March 16, 2017
Additional Exercises for Introduction to Nonlinear Optimization Amir Beck March 16, 2017 Chapter 1  Mathematical Preliminaries 1.1 Let S R n. (a) Suppose that T is an open set satisfying T S. Prove that
More informationIE 5531 Midterm #2 Solutions
IE 5531 Midterm #2 s Prof. John Gunnar Carlsson November 9, 2011 Before you begin: This exam has 9 pages and a total of 5 problems. Make sure that all pages are present. To obtain credit for a problem,
More informationCS711008Z Algorithm Design and Analysis
.. CS711008Z Algorithm Design and Analysis Lecture 9. Algorithm design technique: Linear programming and duality Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China
More informationLECTURE 10 LECTURE OUTLINE
LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationSupport Vector Machine (SVM) and Kernel Methods
Support Vector Machine (SVM) and Kernel Methods CE717: Machine Learning Sharif University of Technology Fall 2015 Soleymani Outline Margin concept HardMargin SVM SoftMargin SVM Dual Problems of HardMargin
More information1. f(β) 0 (that is, β is a feasible point for the constraints)
xvi 2. The lasso for linear models 2.10 Bibliographic notes Appendix Convex optimization with constraints In this Appendix we present an overview of convex optimization concepts that are particularly useful
More informationCheng Soon Ong & Christian Walder. Canberra February June 2018
Cheng Soon Ong & Christian Walder Research Group and College of Engineering and Computer Science Canberra February June 2018 Outlines Overview Introduction Linear Algebra Probability Linear Regression
More informationHomework Set #6  Solutions
EE 15  Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 111 Homework Set #6  Solutions 1 a The feasible set is the interval [ 4] The unique
More informationELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications
ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February
More informationPrimaldual Subgradient Method for Convex Problems with Functional Constraints
Primaldual Subgradient Method for Convex Problems with Functional Constraints Yurii Nesterov, CORE/INMA (UCL) Workshop on embedded optimization EMBOPT2014 September 9, 2014 (Lucca) Yu. Nesterov Primaldual
More informationQuiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006
Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 20182019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationPattern Classification, and Quadratic Problems
Pattern Classification, and Quadratic Problems (Robert M. Freund) March 3, 24 c 24 Massachusetts Institute of Technology. 1 1 Overview Pattern Classification, Linear Classifiers, and Quadratic Optimization
More informationSupport Vector Machine (SVM) & Kernel CE717: Machine Learning Sharif University of Technology. M. Soleymani Fall 2012
Support Vector Machine (SVM) & Kernel CE717: Machine Learning Sharif University of Technology M. Soleymani Fall 2012 Linear classifier Which classifier? x 2 x 1 2 Linear classifier Margin concept x 2
More informationMachine Learning and Data Mining. Support Vector Machines. Kalev Kask
Machine Learning and Data Mining Support Vector Machines Kalev Kask Linear classifiers Which decision boundary is better? Both have zero training error (perfect training accuracy) But, one of them seems
More informationWhat s New in ActiveSet Methods for Nonlinear Optimization?
What s New in ActiveSet Methods for Nonlinear Optimization? Philip E. Gill Advances in Numerical Computation, Manchester University, July 5, 2011 A Workshop in Honor of Sven Hammarling UCSD Center for
More informationLinear classifiers selecting hyperplane maximizing separation margin between classes (large margin classifiers)
Support vector machines In a nutshell Linear classifiers selecting hyperplane maximizing separation margin between classes (large margin classifiers) Solution only depends on a small subset of training
More informationDuality. Geoff Gordon & Ryan Tibshirani Optimization /
Duality Geoff Gordon & Ryan Tibshirani Optimization 10725 / 36725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider
More informationLinear classifiers selecting hyperplane maximizing separation margin between classes (large margin classifiers)
Support vector machines In a nutshell Linear classifiers selecting hyperplane maximizing separation margin between classes (large margin classifiers) Solution only depends on a small subset of training
More information