CS711008Z Algorithm Design and Analysis


 Kellie Watkins
 1 years ago
 Views:
Transcription
1 CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31
2 Outline Brief history of interior point method Basic idea of interior point method 2 / 31
3 A brief history of linear program In 1949, G B Dantzig proposed the simplex algorithm; In 1971, Klee and Minty gave a counterexample to show that simplex is not a polynomialtime algorithm In 1975, L V Kantorovich and T C Koopmans, Nobel prize, application of linear programming in resource distribution; In 1979, L G Khanchian proposed a polynomialtime ellipsoid method; In 1984, N Karmarkar proposed another polynomialtime interiorpoint method; In 2001, D Spielman and S Teng proposed smoothed complexity to prove the efficiency of simplex algorithm 3 / 31
4 In 1979, L G Khanchian proposed a polynomialtime ellipsoid method for LP Figure: Leonid G Khanchian 4 / 31
5 In 1984, N Karmarkar proposed a new polynomialtime algorithm for LP 5 / 31
6 Basic idea of interior point method 6 / 31
7 KKT conditions for LP Let s consider a linear program in slack form and its dual problem, ie Primal problem: min c T x st Ax = b x 0 Dual problem: max b T y st A T y c KKT condition: Primal feasibility: Ax = b, x 0 Dual feasibility: A T y c Complementary slackness: x i = 0, or a T i y = c i for any i = 1,, m 7 / 31
8 Rewriting KKT condition Let s consider a linear program in slack form and its dual problem, ie Primal problem: min c T x st Ax = b x 0 Dual problem: max b T y st A T y c KKT condition: Primal feasibility: Ax = b, x 0; Dual feasibility: A T y + σ = c, σ 0; Complementary slackness: x i σ i = 0 These conditions consists of m + 2n equations over m + 2n variables plus nonnegative constraints 8 / 31
9 Rewrite KKT conditions further Let s define diagonal matrix: x σ X = 0 x 2 0, Σ = 0 σ x σ 3 and rewrite the complementary slackness as: x 1 σ x 1 σ 1 XΣe = 0 x 2 σ = 0 = x 2 σ x 3 σ x 3 σ 3 KKT condition: Primal feasibility: Ax = b, x 0; Dual feasibility: A T y + σ = c, σ 0; Complementary slackness: XΣe = 0 We call (x, y, σ) interior point if x > 0 and σ > 0 Question: How to find (x, y, σ) satisfy the KKT conditions? 9 / 31
10 A simple interiorpoint method Basic idea: Assume we have already known ( x, ȳ, σ) that are both primal and dual feasible, ie, A x = b, x > 0 A T ȳ + σ = c, σ > 0 and try to improve to another point (x, y, σ ) to make complementary slackness hold Improvement strategy: Starting from ( x, ȳ, σ), we follow a direction ( x, y, σ) such that ( x + x, ȳ + y, σ + σ) is a better solution, ie, it comes closer to satisfying the complementary slackness To find such a direction, we substitute ( x + x, ȳ + y, σ + σ) into the KKT conditions: Primal feasibility: A x + x = b, x + x 0; Dual feasibility: A T (ȳ + y) + ( σ + σ) = c, σ + σ 0; Complementary slackness: ( X + X)( Σ + Σ)e = 0 10 / 31
11 Since we start from ( x, ȳ, σ) such that A x = b, x > 0 A T ȳ + σ = c, σ > 0, the above conditions change into A x = 0, x + x 0; A T y + σ = 0, σ + σ 0; X Σ + Σ X = X Σe X Σe Note that when x and σ are small, the final nonlinear term X Σe is small relative to X Σe and can thus be dropped out, generating a linear system A x = 0, x + x 0; A T y + σ = 0, σ + σ 0; X Σ + Σ X = X Σe The solution are: σ = X 1 ( X Σ Σ x) = σ X 1 Σ x ( ) ( ) ) Σ X 1 A T x ( σ = A 0 y 0 11 / 31
12 1: Set initial solution x such that A x = b, x > 0; 2: Set initial solution ȳ, σ such that A T ȳ + σ = c, σ > 0; 3: while TRUE do 4: Solve ( X 1 Σ A T A 0 ) ( ) x y = ) ( σ 0 5: Set σ = X 1 ( X Σ Σ x) = σ X 1 Σ x; 6: Calculate θ to make x + θ x 0, and σ + θ σ 0; 7: Update x = x + θ x, ȳ = ȳ + θ y, σ = σ + θ σ; 8: if x i σ i ϵ for all i = 1,, m then 9: break; 10: end if 11: end while 12: return (x, y, σ); 12 / 31
13 An example min 2x 1 +15x 2 st 12x 1 +24x x 1 +16x x 1 +12x x 1 15 x 2 15 x 1 0 x / 31
14 An execution Starting point 1: x = (10, 10) x 1 x / 31
15 Another execution Starting point 2: x = (14, 1) x 1 x / 31
16 Centered interiorpoint method To avoid the poor performance, we need a way to keep the iterate away from the boundary until the solution approaches the optimum An efficient way to achieve this goal is to relax complementary slackness: XΣe = 0 into XΣe = 1 (t > 0) t We start from a small t, and gradually increase it as the algorithm proceeds At each iteration, we execute the previous affine interior point method 16 / 31
17 Running center path algorithm Starting point 1: x = (10, 10) x 1 x / 31
18 Running center path algorithm Starting point 2: x = (14, 1) x 1 x / 31
19 Another viewpoint Actually, the relaxed complementary slackness XΣe = 1 t (t > 0) corresponds to the following linear program: min c T x 1 t n i=1 log x i st Ax = b x 0 19 / 31
20 Consider a general optimization problem Suppose we are trying to solve a convex program with inequality constraints: min f(x) st Ax = b g(x) 0 where f(x) and g(x) are convex, and twice differentiable 20 / 31
21 Let s start from equality constrained quadratic programming Suppose we are trying to solve a QP with equality constraints : min st 1 2 xt P x + Q T x + r Ax = b Applying Lagrangian conditions, we have Ax = b, and P x + Q + A T λ = 0 Thus, the optimum point x can be solved as follows: [ ] [ ] [ ] P A T x Q = A 0 λ b 21 / 31
22 Then how to minimize f(x) with equality constraints? Newton s method Suppose we are trying to minimize a convex function f(x), which is not a QP min st f(x) Ax = b Basic idea: let s try to improve from a feasible solution x At x, we write the Taylor extension, and use quadratic approximation to f(x + x) = f(x) + f(x) xt 2 f(x) x min st f(x) A(x + x) = b Thus, the optimum point x can be solved as follows: [ 2 f(x) A T ] [ ] [ ] x f(x) = A 0 λ b Since this is just an approximation to f(x), we need to iteratively perform line search 22 / 31
23 Then how to minimize f(x) with inequality constraints? Suppose we are trying to solve a convex program with inequality constraints: min st f(x) Ax = b g(x) 0 where f(x) and g(x) are convex, and twice differentiable Basic idea: 1 Transform it into a series of equality constrainted optimisation problems; 2 Solve each equality constrainted optimisation problem using Newton s method 23 / 31
24 Log barrier function: transform into equality constraints 24 / 31
25 Log barrier function Suppose we are trying to solve a convex program with inequality constraints: min st f(x) Ax = b g(x) 0 where f(x) and g(x) are convex, and twice differentiable This is equivalent to the following optimization problem min f(x) + I_(g(x)) st Ax = b { 0 if x 0 where I_(x) = otherwise But indicator function is not differentiable 25 / 31
26 Log barrier function: smooth approximation to the indicator function Suppose we are trying to solve a convex program with inequality constraints: min st f(x) Ax = b g(x) 0 where f(x) and g(x) are convex, and twice differentiable This is equivalent to the following optimization problem min st f(x) 1 t log( g(x)) Ax = b Basic idea: a smooth approximation to indicator function 1 t log( x), and this approximation improves as t 26 / 31
27 Log barrier function approximates indicator function 1 t log( x) approximates the indicator function, and the approximation improves as t For each setting of t, we obtain an approximation to the original optimisation problem 27 / 31
28 Interior method and central path Solve a sequence of optimization problem: min st f(x) 1 t log( g(x)) Ax = b For any t > 0, we define x (t) as the optimal solution t increases step by step t should not be too large initially, as it is not easy to solve it using Newton s method Central path: {x (t) t > 0} 28 / 31
29 Interior point method for LP Consider the following LP min c T x st Ax b x 0 29 / 31
30 Central path For any t > 0, we define x (t) as the optimal solution to: min f(x) 1 t log( g(x)) st Ax = b x (t) is not optimal solution to the original problem The duality gap is bounded by m t 30 / 31
31 Interior point algorithm 31 / 31
PrimalDual InteriorPoint Methods
PrimalDual InteriorPoint Methods Lecturer: Aarti Singh Coinstructor: Pradeep Ravikumar Convex Optimization 10725/36725 Outline Today: Primaldual interiorpoint method Special case: linear programming
More informationPrimalDual InteriorPoint Methods. Ryan Tibshirani Convex Optimization
PrimalDual InteriorPoint Methods Ryan Tibshirani Convex Optimization 10725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,... m are
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equalityconstrained Optimization Inequalityconstrained Optimization Mixtureconstrained Optimization 3 Quadratic Programming
More informationInterior Point Algorithms for Constrained Convex Optimization
Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems
More informationNumerical Optimization
Linear Programming  Interior Point Methods Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Example 1 Computational Complexity of Simplex Algorithm
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More informationPrimalDual InteriorPoint Methods. Ryan Tibshirani Convex Optimization /36725
PrimalDual InteriorPoint Methods Ryan Tibshirani Convex Optimization 10725/36725 Given the problem Last time: barrier method min x subject to f(x) h i (x) 0, i = 1,... m Ax = b where f, h i, i = 1,...
More information12. Interiorpoint methods
12. Interiorpoint methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationBarrier Method. Javier Peña Convex Optimization /36725
Barrier Method Javier Peña Convex Optimization 10725/36725 1 Last time: Newton s method For rootfinding F (x) = 0 x + = x F (x) 1 F (x) For optimization x f(x) x + = x 2 f(x) 1 f(x) Assume f strongly
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationPrimalDual InteriorPoint Methods. Javier Peña Convex Optimization /36725
PrimalDual InteriorPoint Methods Javier Peña Convex Optimization 10725/36725 Last time: duality revisited Consider the problem min x subject to f(x) Ax = b h(x) 0 Lagrangian L(x, u, v) = f(x) + u T
More informationCS711008Z Algorithm Design and Analysis
.. CS711008Z Algorithm Design and Analysis Lecture 9. Algorithm design technique: Linear programming and duality Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China
More informationInteriorPoint Methods for Linear Optimization
InteriorPoint Methods for Linear Optimization Robert M. Freund and Jorge Vera March, 204 c 204 Robert M. Freund and Jorge Vera. All rights reserved. Linear Optimization with a Logarithmic Barrier Function
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More information12. Interiorpoint methods
12. Interiorpoint methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationConvex Optimization and SVM
Convex Optimization and SVM Problem 0. Cf lecture notes pages 12 to 18. Problem 1. (i) A slab is an intersection of two half spaces, hence convex. (ii) A wedge is an intersection of two half spaces, hence
More informationPrimalDual InteriorPoint Methods for Linear Programming based on Newton s Method
PrimalDual InteriorPoint Methods for Linear Programming based on Newton s Method Robert M. Freund March, 2004 2004 Massachusetts Institute of Technology. The Problem The logarithmic barrier approach
More informationLecture 15: October 15
10725: Optimization Fall 2012 Lecturer: Barnabas Poczos Lecture 15: October 15 Scribes: Christian Kroer, Fanyi Xiao Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have
More informationCSCI 1951G Optimization Methods in Finance Part 09: Interior Point Methods
CSCI 1951G Optimization Methods in Finance Part 09: Interior Point Methods March 23, 2018 1 / 35 This material is covered in S. Boyd, L. Vandenberge s book Convex Optimization https://web.stanford.edu/~boyd/cvxbook/.
More informationSupport Vector Machines: Maximum Margin Classifiers
Support Vector Machines: Maximum Margin Classifiers Machine Learning and Pattern Recognition: September 16, 2008 Piotr Mirowski Based on slides by Sumit Chopra and FuJie Huang 1 Outline What is behind
More information2.098/6.255/ Optimization Methods Practice True/False Questions
2.098/6.255/15.093 Optimization Methods Practice True/False Questions December 11, 2009 Part I For each one of the statements below, state whether it is true or false. Include a 13 line supporting sentence
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.14.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationCSCI 1951G Optimization Methods in Finance Part 01: Linear Programming
CSCI 1951G Optimization Methods in Finance Part 01: Linear Programming January 26, 2018 1 / 38 Liability/asset cashflow matching problem Recall the formulation of the problem: max w c 1 + p 1 e 1 = 150
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 function with f () on (, L) and that you have explicit formulae for
More informationCSC373: Algorithm Design, Analysis and Complexity Fall 2017 DENIS PANKRATOV NOVEMBER 1, 2017
CSC373: Algorithm Design, Analysis and Complexity Fall 2017 DENIS PANKRATOV NOVEMBER 1, 2017 Linear Function f: R n R is linear if it can be written as f x = a T x for some a R n Example: f x 1, x 2 =
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationICSE4030 Kernel Methods in Machine Learning
ICSE4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationLinear programming II
Linear programming II Review: LP problem 1/33 The standard form of LP problem is (primal problem): max z = cx s.t. Ax b, x 0 The corresponding dual problem is: min b T y s.t. A T y c T, y 0 Strong Duality
More informationAdvanced Mathematical Programming IE417. Lecture 24. Dr. Ted Ralphs
Advanced Mathematical Programming IE417 Lecture 24 Dr. Ted Ralphs IE417 Lecture 24 1 Reading for This Lecture Sections 11.211.2 IE417 Lecture 24 2 The Linear Complementarity Problem Given M R p p and
More informationMath 5593 Linear Programming Week 1
University of Colorado Denver, Fall 2013, Prof. Engau 1 ProblemSolving in Operations Research 2 Brief History of Linear Programming 3 Review of Basic Linear Algebra Linear Programming  The Story About
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LPproblem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470  Convex Optimization Fall 201718, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 Minimum volume ellipsoid around a set LöwnerJohn ellipsoid
More informationOperations Research Lecture 4: Linear Programming Interior Point Method
Operations Research Lecture 4: Linear Programg Interior Point Method Notes taen by Kaiquan Xu@Business School, Nanjing University April 14th 2016 1 The affine scaling algorithm one of the most efficient
More informationConvex Optimization and Support Vector Machine
Convex Optimization and Support Vector Machine Problem 0. Consider a twoclass classification problem. The training data is L n = {(x 1, t 1 ),..., (x n, t n )}, where each t i { 1, 1} and x i R p. We
More informationIE 5531 Midterm #2 Solutions
IE 5531 Midterm #2 s Prof. John Gunnar Carlsson November 9, 2011 Before you begin: This exam has 9 pages and a total of 5 problems. Make sure that all pages are present. To obtain credit for a problem,
More informationConvex Optimization and l 1 minimization
Convex Optimization and l 1 minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l
More informationISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints
ISM206 Lecture Optimization of Nonlinear Objective with Linear Constraints Instructor: Prof. Kevin Ross Scribe: Nitish John October 18, 2011 1 The Basic Goal The main idea is to transform a given constrained
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 00 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Nonlinear programming In case of LP, the goal
More informationTheory and Internet Protocols
Game Lecture 2: Linear Programming and Zero Sum Nash Equilibrium Xiaotie Deng AIMS Lab Department of Computer Science Shanghai Jiaotong University September 26, 2016 1 2 3 4 Standard Form (P) Outline
More informationFollowing The Central Trajectory Using The Monomial Method Rather Than Newton's Method
Following The Central Trajectory Using The Monomial Method Rather Than Newton's Method YiChih Hsieh and Dennis L. Bricer Department of Industrial Engineering The University of Iowa Iowa City, IA 52242
More informationLecture 9 Sequential unconstrained minimization
S. Boyd EE364 Lecture 9 Sequential unconstrained minimization brief history of SUMT & IP methods logarithmic barrier function central path UMT & SUMT complexity analysis feasibility phase generalized inequalities
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470  Convex Optimization Fall 201819, HKUST, Hong Kong Outline of Lecture Subgradients
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More informationCSE4830 Kernel Methods in Machine Learning
CSE4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationIntroduction to Nonlinear Stochastic Programming
School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS
More informationConvex Optimization. Prof. Nati Srebro. Lecture 12: InfeasibleStart Newton s Method Interior Point Methods
Convex Optimization Prof. Nati Srebro Lecture 12: InfeasibleStart Newton s Method Interior Point Methods Equality Constrained Optimization f 0 (x) s. t. A R p n, b R p Using access to: 2 nd order oracle
More informationLecture 16: October 22
0725/36725: Conve Optimization Fall 208 Lecturer: Ryan Tibshirani Lecture 6: October 22 Scribes: Nic Dalmasso, Alan Mishler, Benja LeRoy Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationOptimisation and Operations Research
Optimisation and Operations Research Lecture 22: Linear Programming Revisited Matthew Roughan http://www.maths.adelaide.edu.au/matthew.roughan/ Lecture_notes/OORII/ School
More informationConvex Optimization Lecture 13
Convex Optimization Lecture 13 Today: InteriorPoint (continued) Central Path method for SDP Feasibility and Phase I Methods From Central Path to Primal/Dual Central'Path'Log'Barrier'Method Init: Feasible&#
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More informationThe Qparametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Springmass System
The Qparametrization (Youla) Lecture 3: Synthesis by Convex Optimization controlled variables z Plant distubances w Example: Springmass system measurements y Controller control inputs u Idea for lecture
More informationSupport Vector Machines for Regression
COMP566 Rohan Shah (1) Support Vector Machines for Regression Provided with n training data points {(x 1, y 1 ), (x 2, y 2 ),, (x n, y n )} R s R we seek a function f for a fixed ɛ > 0 such that: f(x
More informationKarushKuhnTucker Conditions. Lecturer: Ryan Tibshirani Convex Optimization /36725
KarushKuhnTucker Conditions Lecturer: Ryan Tibshirani Convex Optimization 10725/36725 1 Given a minimization problem Last time: duality min x subject to f(x) h i (x) 0, i = 1,... m l j (x) = 0, j =
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec  Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec  Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationInterior Point Methods for LP
11.1 Interior Point Methods for LP Katta G. Murty, IOE 510, LP, U. Of Michigan, Ann Arbor, Winter 1997. Simplex Method  A Boundary Method: Starting at an extreme point of the feasible set, the simplex
More informationSF2822 Applied nonlinear optimization, final exam Wednesday June
SF2822 Applied nonlinear optimization, final exam Wednesday June 3 205 4.00 9.00 Examiner: Anders Forsgren, tel. 08790 7 27. Allowed tools: Pen/pencil, ruler and eraser. Note! Calculator is not allowed.
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationSF2822 Applied nonlinear optimization, final exam Saturday December
SF2822 Applied nonlinear optimization, final exam Saturday December 5 27 8. 3. Examiner: Anders Forsgren, tel. 79 7 27. Allowed tools: Pen/pencil, ruler and rubber; plus a calculator provided by the department.
More informationLecture: Introduction to LP, SDP and SOCP
Lecture: Introduction to LP, SDP and SOCP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2015.html wenzw@pku.edu.cn Acknowledgement:
More informationComputational Finance
Department of Mathematics at University of California, San Diego Computational Finance Optimization Techniques [Lecture 2] Michael Holst January 9, 2017 Contents 1 Optimization Techniques 3 1.1 Examples
More informationLecture 10: Linear programming duality and sensitivity 00
Lecture 10: Linear programming duality and sensitivity 00 The canonical primal dual pair 1 A R m n, b R m, and c R n maximize z = c T x (1) subject to Ax b, x 0 n and minimize w = b T y (2) subject to
More informationAgenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primaldual path following algorithms
Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primaldual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point
More informationLINEAR AND NONLINEAR PROGRAMMING
LINEAR AND NONLINEAR PROGRAMMING Stephen G. Nash and Ariela Sofer George Mason University The McGrawHill Companies, Inc. New York St. Louis San Francisco Auckland Bogota Caracas Lisbon London Madrid Mexico
More informationIntroduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Introduction to Machine Learning Prof. Sudeshna Sarkar Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Module  5 Lecture  22 SVM: The Dual Formulation Good morning.
More informationOutline. Outline. Outline DMP204 SCHEDULING, TIMETABLING AND ROUTING. 1. Scheduling CPM/PERT Resource Constrained Project Scheduling Model
Outline DMP204 SCHEDULING, TIMETABLING AND ROUTING Lecture 3 and Mixed Integer Programg Marco Chiarandini 1. Resource Constrained Project Model 2. Mathematical Programg 2 Outline Outline 1. Resource Constrained
More informationResource Constrained Project Scheduling Linear and Integer Programming (1)
DM204, 2010 SCHEDULING, TIMETABLING AND ROUTING Lecture 3 Resource Constrained Project Linear and Integer Programming (1) Marco Chiarandini Department of Mathematics & Computer Science University of Southern
More informationInterior Point Methods for Mathematical Programming
Interior Point Methods for Mathematical Programming Clóvis C. Gonzaga Federal University of Santa Catarina, Florianópolis, Brazil EURO  2013 Roma Our heroes Cauchy Newton Lagrange Early results Unconstrained
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationOptimization (168) Lecture 789
Optimization (168) Lecture 789 Jesús De Loera UC Davis, Mathematics Wednesday, April 2, 2012 1 DEGENERACY IN THE SIMPLEX METHOD 2 DEGENERACY z =2x 1 x 2 + 8x 3 x 4 =1 2x 3 x 5 =3 2x 1 + 4x 2 6x 3 x 6
More informationIntroduction to optimization
Introduction to optimization Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 24 The plan 1. The basic concepts 2. Some useful tools (linear programming = linear optimization)
More informationHomework 4. Convex Optimization /36725
Homework 4 Convex Optimization 10725/36725 Due Friday November 4 at 5:30pm submitted to Christoph Dann in Gates 8013 (Remember to a submit separate writeup for each problem, with your name at the top)
More informationFRTN10 Multivariable Control, Lecture 13. Course outline. The Qparametrization (Youla) Example: Springmass System
FRTN Multivariable Control, Lecture 3 Anders Robertsson Automatic Control LTH, Lund University Course outline The Qparametrization (Youla) LL5 Purpose, models and loopshaping by hand L6L8 Limitations
More informationLecture 1 Introduction
L. Vandenberghe EE236A (Fall 201314) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation
More informationOptimization. YuhJye Lee. March 28, Data Science and Machine Intelligence Lab National Chiao Tung University 1 / 40
Optimization YuhJye Lee Data Science and Machine Intelligence Lab National Chiao Tung University March 28, 2017 1 / 40 The Key Idea of Newton s Method Let f : R n R be a twice differentiable function
More informationInteriorpoint methods Optimization Geoff Gordon Ryan Tibshirani
Interiorpoint methods 10725 Optimization Geoff Gordon Ryan Tibshirani SVM duality Review min v T v/2 + 1 T s s.t. Av yd + s 1 0 s 0 max 1 T α α T Kα/2 s.t. y T α = 0 0 α 1 Gram matrix K Interpretation
More informationNonlinear Optimization for Optimal Control
Nonlinear Optimization for Optimal Control Pieter Abbeel UC Berkeley EECS Many slides and figures adapted from Stephen Boyd [optional] Boyd and Vandenberghe, Convex Optimization, Chapters 9 11 [optional]
More informationPart IB Optimisation
Part IB Optimisation Theorems Based on lectures by F. A. Fischer Notes taken by Dexter Chua Easter 2015 These notes are not endorsed by the lecturers, and I have modified them (often significantly) after
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plusproperties and Funiqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FPI book:
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationOptimization: Then and Now
Optimization: Then and Now Optimization: Then and Now Optimization: Then and Now Why would a dynamicist be interested in linear programming? Linear Programming (LP) max c T x s.t. Ax b αi T x b i for i
More informationTopics in Theoretical Computer Science April 08, Lecture 8
Topics in Theoretical Computer Science April 08, 204 Lecture 8 Lecturer: Ola Svensson Scribes: David Leydier and Samuel Grütter Introduction In this lecture we will introduce Linear Programming. It was
More informationAn upper bound for the number of different solutions generated by the primal simplex method with any selection rule of entering variables
An upper bound for the number of different solutions generated by the primal simplex method with any selection rule of entering variables Tomonari Kitahara and Shinji Mizuno February 2012 Abstract Kitahara
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationSupport vector machines
Support vector machines Guillaume Obozinski Ecole des Ponts  ParisTech SOCN course 2014 SVM, kernel methods and multiclass 1/23 Outline 1 Constrained optimization, Lagrangian duality and KKT 2 Support
More informationPenalty and Barrier Methods General classical constrained minimization problem minimize f(x) subject to g(x) 0 h(x) =0 Penalty methods are motivated by the desire to use unconstrained optimization techniques
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 1 Minimum volume ellipsoid around a set LöwnerJohn ellipsoid
More informationSolution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark
Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationLecture 15 Newton Method and SelfConcordance. October 23, 2008
Newton Method and SelfConcordance October 23, 2008 Outline Lecture 15 Selfconcordance Notion Selfconcordant Functions Operations Preserving Selfconcordance Properties of Selfconcordant Functions Implications
More information1. Gradient method. gradient method, firstorder methods. quadratic bounds on convex functions. analysis of gradient method
L. Vandenberghe EE236C (Spring 2016) 1. Gradient method gradient method, firstorder methods quadratic bounds on convex functions analysis of gradient method 11 Approximate course outline Firstorder
More informationConvexification of MixedInteger Quadratically Constrained Quadratic Programs
Convexification of MixedInteger Quadratically Constrained Quadratic Programs Laura Galli 1 Adam N. Letchford 2 Lancaster, April 2011 1 DEIS, University of Bologna, Italy 2 Department of Management Science,
More informationDuality revisited. Javier Peña Convex Optimization /36725
Duality revisited Javier Peña Conve Optimization 10725/36725 1 Last time: barrier method Main idea: approimate the problem f() + I C () with the barrier problem f() + 1 t φ() tf() + φ() where t > 0 and
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More information