COM S 578X: Optimization for Machine Learning
|
|
- Brent Ramsey
- 5 years ago
- Views:
Transcription
1 COM S 578X: Optimization for Machine Learning Lecture Note 4: Duality Jia (Kevin) Liu Assistant Professor Department of Computer Science Iowa State University, Ames, Iowa, USA Fall 2018 JKL (CS@ISU) COM S 578X: Lecture 4 1/21
2 Outline In this lecture: Lagrange dual problem Weak and strong duality Geometric interpretation Examples in machine learning JKL COM S 578X: Lecture 4 2/21
3 The Lagrangian Standard optimization problem (may or may not be convex): convex Opt : Minimize f(x) it :t convex P : Vi dwdvarormk9471es convex fi g subject to g i (x) apple 0, i =1,,m Ui > o Hi hit 'AE h j (x) =0, j =1,,p Uj EIR 'tj variable Pis x convex 2 R n, opt hike? domain D, optimal value p Lagrangian: L: R n R m R p! R, withdom(l) =D R m R p : mx px L(x, u, v) =f(x)+ u i g i (x)+ v j h j (x) i=1 j=1 weighted sum of objective and constraint functions u i 0 is dual variable (Lagrangian multiplier) associated with g i (x) apple 0 v j 2 R is dual variable (Lagrangian multiplier) associated with h j (x) =0 JKL (CS@ISU) COM S 578X: Lecture 4 3/21
4 ' value ( Lagrangian Dual Function Lagrangian dual function: :R m R p! R: (u, v) = inf L(x, u, v) x2d 0 mx It = u i g i (x)+ (u, v) is concave in u, v, canbe 1 for some u, v bk its ' a pt Lower bound property: If u Proof If x 2Dand u 20<0 =O px t v j h j (x) A x2d i=1 j=1 we EO wise 0, then (u, v) apple p minimum Wirt y 1,1 ) opt of Primal problem by + to ", detyninj, t 4 0, then f( x) L( x, u, v) inf L(x, u, v) = (u, v) x2d Since this holds for any x 2D, minimizing over all x yields p 1 of altmetns DET Lagrangian dual (u, v) JKL (CS@ISU) COM S 578X: Lecture 4 4/21
5 Example: Leastnorm solution of linear equations Minimize x > x subject to Ax = b I EIR " Dual function: Lagrangian is: L(x, v) =x > x + v > (Ax b) To minimize L(x, v) over x, set gradient equal to zero r x L(x, v) =2x + A > v =0 =) x = 1 2 A> v Plug it in L to obtain : (v) =L( which is clearly a concave function of v it 1 2 A> v, v) = 1 4 v> AA > v b > v, NSD Eo Lower bound property: p 1 4 v> AA > v b > v for all v Largest LB JKL (CS@ISU) COM S 578X: Lecture 4 5/21
6 Example: Linear Programming Dual function: Lagrangian is: Minimize c > x subject to Ax = b, x 0 a a IEIR " YEIR "+ L is a ne in x, hence L(x, u, v) =c > x + v > (Ax b) u > x (u, v) =infl(x, u, v) = x = b > v +(c + A > v u) > x 3 w : ni, ( b > v A > v u + c =0 1 otherwise = = at is linear on a ne domain {(u, v) :A > v u + c = 0}, hence concave Lower bound property: p b > v if A > v + c 0 w win + case JKL (CS@ISU) COM S 578X: Lecture 4 6/21
7 Example: Equality Constrained Norm Minimization Minimize kxk subject to Ax = b Dual function: (v) =inf x (kxk v> Ax + b > v)= ( b > v ka > vk apple 1 1 otherwise = bit tenets { MI where kvk, sup kukapple1 u > v (referred to as dual Effeya } = EK see norm of k k), { I # a Proof It follows from the fact that inf x (kxk y > x)=0if kyk apple 1, 1 otherwise If kyk apple 1, thenkxk y > x 0, 8x, withequalityifx = 0 ( Cauchy schwatd, if kyk > 1, choose x = tu, whereu satisfies kuk apple1, u > y = kyk > 1: kxk y > x = t(kuk kyk )! 1 as t!1 Lower bound property: p b > v if ka > vk apple 1 T > 0 = E I > 1 we I EIR " thea JKL (CS@ISU) COM S 578X: Lecture 4 7/21
8 NP Example: Twoway Partitioning Minimize x > Wx = IEFe, c subject to x 2 i =1, i =1,,n Ii E { I, 13 A nonconvex problem; feasible set contains 2 n, ti Hard discrete points Interpretation: partition {1,,n} into two sets; (W) ij is cost of assigning i, j to the same set; (W) ij is cost of assigning to di erent sets Dual function: (v) =inf x (x> Wx + X i Lower bound property: p v i (x 2 i symmetric 1)) = inf x {x> W + Diag(v) x 1 > v} ( 1 > v W+ Diag{v} 0 = 1 otherwise 1 > v if W + Diag{v} 0 Example: v = min(w)1 gives nontrivial bound p n min (W) a " xilhflijnj diagonal matrix I ER " # ( l ;] : t atuitttfivnl =±y YII* ' JKL (CS@ISU) COM S 578X: Lecture 4 8/21
9 Lagrangian Dual and Conjugate Function Dual function: Minimize f(x) subject to Ax apple b, Cx = d erm+ * ytern (u, v) = inf x2dom(f) f(x)+(a > u + C > v) > x b > u d > v = f ( A > u C > v) b > u d > v Legendre Fenehelconjngatekonvex conjugate Definition of conjugate function: f (y) =sup x2dom(f) y > x f(x) Simplifies derivation of dual if conjugate of f is known Example: Entropy maximization f(x) = nx x i log x i, f (y) = i=1 nx exp(y i 1) i=1 JKL (CS@ISU) COM S 578X: Lecture 4 9/21
10 The Lagrangian Dual Problem Maximize (u, v) subject to u 0 if Min EQ OH, v ) Finds largest lower bound on p, obtained from Lagrangian dual function A convex optimization problem; optimal value denoted d u, v are dual feasible if u 0, (u, v) 2 dom( ) Often simplified by making implicit constraint u 0, (u, v) 2 dom( ) explicit Example: Standard form LP and its dual: Minimize c > x subject to Ax = b, x 0 Maximize b > v subject to A > v + c 0 More on this later JKL (CS@ISU) COM S 578X: Lecture 4 10 / 21
11 Weak and Strong Duality Weak duality: d apple p Always holds (for convex and nonconvex problems) Can be used to find nontrivial lower bounds for di cult problems For example, solving SDP semi definite Programming Maximize subject to 1 > v W + Diag{v} 0 yields a lower bound for the twoway partitioning problem Strong duality: d = p Does not hold in general Usually hold for convex problems mild ( under certaihhonditions ) Conditions that guarantee strong duality in convex problems are called constraint qualifications ( CQ ) JKL (CS@ISU) COM S 578X: Lecture 4 11 / 21
12 Slater s Constraint Qualification Strong duality holds for a convex problem Minimize f(x) subject to g i (x) apple 0, Ax = b i =1,,m if it is strictly feasible, ie, 9x 2 intd : g i (x) < 0, i =1,, m, Ax = b Also guarantees that the dual optimum is attained (if p > 1) Can be further relaxed: eg, can replace intd with relintd (interior relative to a ne hull); linear inequalities do not need to hold with strict inequality, There exist many other types of constraint qualifications (see [BSS, Ch 5]) JKL (CS@ISU) COM S 578X: Lecture 4 12 / 21
13 Other WellKnown CQs LICQ (Linear independence CQ): Gradients of active inequality constraints and gradients of equality constraints are linearly independent at x MFCQ (ManasarianFromovitz CQ): Gradients of equality constraints are LI at x, 9d 2 R n such that rg > i (x )d < 0 for active inequality constraints, rh > j (x )d =0for equality constraints CRCQ (Constant rank CQ): For each subset of gradients of active inequality constraints & gradients of equality constraints, the rank at x s vicinity is constant CPLD (Constant positive linear dependence CQ): For each subset of gradients of active inequality constraints & gradients of equality constraints, if it s positivelinear dep at x then it s positivelinear dependent in x s vicinity QNCQ (Quasinormality CQ): If gradients of active inequality constraints and gradients of equality constraints are positivelylinearly dependent at x with duals u i for inequalities and v j for equalities, then there is no sequence x k! x such that v j 6=0) v j h j (x k ) > 0 and u i 6=0) u i g i (x k ) < 0 JKL (CS@ISU) COM S 578X: Lecture 4 13 / 21
14 value 9 " value Geometric Interpretation Mi?fg function " c Bssr op g) peg, sit, p*=pco, For ease of exposition, consider problem with one constraint g(x) apple 0 Interpretation of dual function P sent lines I Perturbed (u) = inf (uq + p), where G = {(g(x),f(x)) x 2D} (q,p)2g t ) p hp =p # obj T q pcobj ppc ) E HI, ptxaicm I * y " c RAS ) q uq + p = (u) is an supporting hyperplane to set G with slope u Hyperplane intersects paxis at p = (u) JKL (CS@ISU) COM S 578X: Lecture 4 14 / 21
15 Geometric Interpretation Epigraph variation: Same interpretation if G is replaced with A = {(q, p) g(x) apple q, f(x) apple p for some x 2D} v : Strong duality Holds if there is a nonvertical supporting hyperplane to A at (0,p ) For convex problem, A is convex, hence has supporting hyperplane at (0,p ) Slater s condition: If there exist ( q, p) 2Awith q <0, then supporting hyperplanes at (0,p ) must be nonvertical JKL (CS@ISU) COM S 578X: Lecture 4 15 / 21
16 vertical Strong Duality Geometric Interpretation p I A Existence i " " of Teo forcing the appointing hyperplane to be non clearly, Slater 's condition does NOT hold JKL (CS@ISU) COM S 578X: Lecture 4 16 / 21 q
17 e Example: Inequality Form of LP Primal problem: Dual function: Dual problem: (u) =inf x Minimize c > x subject to Ax apple b T = LCE, 4) Eta t a I A a I s ZE so to ( (c + A > u) > x b > u = :* Ta : e Maximize b > u Is E subject to A > u + c = 0, u 0 From Slater s condition: p = d if A x < b for some x b > u A > u + c = 0 1 otherwise In fact, p = d except when primal and dual are infeasible I 's ) L JKL (CS@ISU) COM S 578X: Lecture 4 17 / 21
18 , set to O LES of Example: Quadratic Program Primal problem (P 2 S n ++): Minimize x > Px subject to Ax < b 130 Dual function: ( )=inf x Dual problem: take x > Px + u > (Ax b) = 1 4 u> AP 1 A > u b > u = quadratic fu k = e der ZAZ the Maximize 1 4 u> AP 1 A > u b > u subject to u 0 From Slater s condition: p = d if A x < b for some x In fact, p = d always JKL (CS@ISU) COM S 578X: Lecture 4 18 / 21
19 obj! large Example: Support Vector Machine " i ::* i itb s ),, FITYi III Given labels y 2 { 1, 1} n,featurevectorsx 1,,x m LetX,[x 1,,x m ] > hint be I Recall from Lecture 1 that the support vector machine problem: 1 mx Minimize w,b, 2 w> w + C i guess : I=o b = 0, i=1 choose E suft subject to y i (w > x i + b) 1 i, i =1,,m i 0, i =1,,m Ui 30 Vi 30 Introducing dual variables u, v 0 to obtain the Lagrangian: L(w,b,, u, v) = 1 mx nx 2 kwk2 2 + C i + u i (1 i y i (w > x i + b)) i=1 i=1 I orig penalty Minimizing over w,b, yields the Lagrangian dual function terms nx v i i i=1 JKL (CS@ISU) COM S 578X: Lecture 4 19 / 21
20 Example: Support Vector Machine (u, v) = ( 1 2 u> X> Xu + 1 > u if u = C1 v, u > y =0 1 otherwise where X, XDiag{y 1,,y m } The dual problem, after eliminating v, becomes: 1 Minimize 2 u> X> Xu 1 > u subject to u > y =0, 0 apple u i apple C, i =1,, m Slater s condition is satisfied ) strong duality In ML literature, more common to work with the dual: Quadratic having PSD Hessian with simple bounds, plus a single linear & simple box constraints At optimality, we can verify that w = Xu (this is not a coincidence! will be proved by KKT conditions in next class) e WEE g, b o, e sufficiently large JKL (CS@ISU) COM S 578X: Lecture 4 20 / 21
21 Next Class Optimality Conditions JKL COM S 578X: Lecture 4 21 / 21
5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationEE/AA 578, Univ of Washington, Fall Duality
7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationWHY DUALITY? Gradient descent Newton s method Quasi-newton Conjugate gradients. No constraints. Non-differentiable ???? Constrained problems? ????
DUALITY WHY DUALITY? No constraints f(x) Non-differentiable f(x) Gradient descent Newton s method Quasi-newton Conjugate gradients etc???? Constrained problems? f(x) subject to g(x) apple 0???? h(x) =0
More informationsubject to (x 2)(x 4) u,
Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the
More informationCOM S 672: Advanced Topics in Computational Models of Learning Optimization for Learning
COM S 672: Advanced Topics in Computational Models of Learning Optimization for Learning Lecture Note 4: Optimality Conditions Jia (Kevin) Liu Assistant Professor Department of Computer Science Iowa State
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationCOM S 578X: Optimization for Machine Learning
COM S 578X: Optimization for Machine Learning Lecture Note 5: Optimality Conditions Jia (Kevin) Liu Assistant Professor Department of Computer Science Iowa State University Ames Iowa USA Fall 2018 JKL
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More informationTutorial on Convex Optimization for Engineers Part II
Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de
More informationSolutions Chapter 5. The problem of finding the minimum distance from the origin to a line is written as. min 1 2 kxk2. subject to Ax = b.
Solutions Chapter 5 SECTION 5.1 5.1.4 www Throughout this exercise we will use the fact that strong duality holds for convex quadratic problems with linear constraints (cf. Section 3.4). The problem of
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationConvex Optimization M2
Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial
More informationOptimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
More informationConvex Optimization and Support Vector Machine
Convex Optimization and Support Vector Machine Problem 0. Consider a two-class classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We
More informationICS-E4030 Kernel Methods in Machine Learning
ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationQuiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006
Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationEE 227A: Convex Optimization and Applications October 14, 2008
EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationConstrained Optimization Theory
Constrained Optimization Theory Stephen J. Wright 1 2 Computer Sciences Department, University of Wisconsin-Madison. IMA, August 2016 Stephen Wright (UW-Madison) Constrained Optimization Theory IMA, August
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationCONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS
CONSTRAINT QUALIFICATIONS, LAGRANGIAN DUALITY & SADDLE POINT OPTIMALITY CONDITIONS A Dissertation Submitted For The Award of the Degree of Master of Philosophy in Mathematics Neelam Patel School of Mathematics
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko
More informationLecture 7: Convex Optimizations
Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1
More informationSupport Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and Fu-Jie Huang 1 Outline What is behind
More informationLECTURE 13 LECTURE OUTLINE
LECTURE 13 LECTURE OUTLINE Problem Structures Separable problems Integer/discrete problems Branch-and-bound Large sum problems Problems with many constraints Conic Programming Second Order Cone Programming
More informationConstrained Optimization
1 / 22 Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University March 30, 2015 2 / 22 1. Equality constraints only 1.1 Reduced gradient 1.2 Lagrange
More informationMaximum likelihood estimation
Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization
More informationAgenda. 1 Duality for LP. 2 Theorem of alternatives. 3 Conic Duality. 4 Dual cones. 5 Geometric view of cone programs. 6 Conic duality theorem
Agenda 1 Duality for LP 2 Theorem of alternatives 3 Conic Duality 4 Dual cones 5 Geometric view of cone programs 6 Conic duality theorem 7 Examples Lower bounds on LPs By eliminating variables (if needed)
More informationminimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1,
4 Duality 4.1 Numerical perturbation analysis example. Consider the quadratic program with variables x 1, x 2, and parameters u 1, u 2. minimize x 2 1 +2x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x
More informationFinite Dimensional Optimization Part III: Convex Optimization 1
John Nachbar Washington University March 21, 2017 Finite Dimensional Optimization Part III: Convex Optimization 1 1 Saddle points and KKT. These notes cover another important approach to optimization,
More informationDuality Theory of Constrained Optimization
Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients
More informationThe Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:
HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course
More informationEE364a Review Session 5
EE364a Review Session 5 EE364a Review announcements: homeworks 1 and 2 graded homework 4 solutions (check solution to additional problem 1) scpd phone-in office hours: tuesdays 6-7pm (650-723-1156) 1 Complementary
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More information10-725/ Optimization Midterm Exam
10-725/36-725 Optimization Midterm Exam November 6, 2012 NAME: ANDREW ID: Instructions: This exam is 1hr 20mins long Except for a single two-sided sheet of notes, no other material or discussion is permitted
More informationDuality. Geoff Gordon & Ryan Tibshirani Optimization /
Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider
More informationConvex Optimization and SVM
Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence
More informationHomework Set #6 - Solutions
EE 15 - Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 11-1 Homework Set #6 - Solutions 1 a The feasible set is the interval [ 4] The unique
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationCS , Fall 2011 Assignment 2 Solutions
CS 94-0, Fall 20 Assignment 2 Solutions (8 pts) In this question we briefly review the expressiveness of kernels (a) Construct a support vector machine that computes the XOR function Use values of + and
More informationMachine Learning. Lecture 6: Support Vector Machine. Feng Li.
Machine Learning Lecture 6: Support Vector Machine Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 Warm Up 2 / 80 Warm Up (Contd.)
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationOutline. Roadmap for the NPP segment: 1 Preliminaries: role of convexity. 2 Existence of a solution
Outline Roadmap for the NPP segment: 1 Preliminaries: role of convexity 2 Existence of a solution 3 Necessary conditions for a solution: inequality constraints 4 The constraint qualification 5 The Lagrangian
More informationLecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,
More informationOptimization Theory. Lectures 4-6
Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:
More informationUC Berkeley Department of Electrical Engineering and Computer Science. EECS 227A Nonlinear and Convex Optimization. Solutions 5 Fall 2009
UC Berkeley Department of Electrical Engineering and Computer Science EECS 227A Nonlinear and Convex Optimization Solutions 5 Fall 2009 Reading: Boyd and Vandenberghe, Chapter 5 Solution 5.1 Note that
More informationLecture Notes on Support Vector Machine
Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a n-dimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More informationConvex Optimization. Lecture 12 - Equality Constrained Optimization. Instructor: Yuanzhang Xiao. Fall University of Hawaii at Manoa
Convex Optimization Lecture 12 - Equality Constrained Optimization Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 19 Today s Lecture 1 Basic Concepts 2 for Equality Constrained
More informationConstrained optimization
Constrained optimization DS-GA 1013 / MATH-GA 2824 Optimization-based Data Analysis http://www.cims.nyu.edu/~cfgranda/pages/obda_fall17/index.html Carlos Fernandez-Granda Compressed sensing Convex constrained
More informationSufficient Conditions for Finite-variable Constrained Minimization
Lecture 4 It is a small de tour but it is important to understand this before we move to calculus of variations. Sufficient Conditions for Finite-variable Constrained Minimization ME 256, Indian Institute
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationN. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:
0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 1-3 line supporting sentence
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationEnhanced Fritz John Optimality Conditions and Sensitivity Analysis
Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained
More information6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC
6-1 The Positivstellensatz P. Parrilo and S. Lall, ECC 2003 2003.09.02.10 6. The Positivstellensatz Basic semialgebraic sets Semialgebraic sets Tarski-Seidenberg and quantifier elimination Feasibility
More informationLECTURE 10 LECTURE OUTLINE
LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,
More informationSupport Vector Machines
Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal
More informationLecture Support Vector Machine (SVM) Classifiers
Introduction to Machine Learning Lecturer: Amir Globerson Lecture 6 Fall Semester Scribe: Yishay Mansour 6.1 Support Vector Machine (SVM) Classifiers Classification is one of the most important tasks in
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More informationConvex Functions. Pontus Giselsson
Convex Functions Pontus Giselsson 1 Today s lecture lower semicontinuity, closure, convex hull convexity preserving operations precomposition with affine mapping infimal convolution image function supremum
More informationDeterminant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University
Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.-P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition
More informationRecita,on: Loss, Regulariza,on, and Dual*
10-701 Recita,on: Loss, Regulariza,on, and Dual* Jay- Yoon Lee 02/26/2015 *Adopted figures from 10725 lecture slides and from the book Elements of Sta,s,cal Learning Loss and Regulariza,on Op,miza,on problem
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More information5 Handling Constraints
5 Handling Constraints Engineering design optimization problems are very rarely unconstrained. Moreover, the constraints that appear in these problems are typically nonlinear. This motivates our interest
More informationOn the interior of the simplex, we have the Hessian of d(x), Hd(x) is diagonal with ith. µd(w) + w T c. minimize. subject to w T 1 = 1,
Math 30 Winter 05 Solution to Homework 3. Recognizing the convexity of g(x) := x log x, from Jensen s inequality we get d(x) n x + + x n n log x + + x n n where the equality is attained only at x = (/n,...,
More information4. Algebra and Duality
4-1 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: non-convex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
More information