Introduction to Nonlinear Stochastic Programming

Size: px
Start display at page:

Download "Introduction to Nonlinear Stochastic Programming"

Transcription

1 School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio J.Gondzio@ed.ac.uk URL: SPS 2007, Bergamo, April

2 Outline Convexity convex sets, convex functions local optimum, global optimum Duality Lagrange duality Wolfe duality primal-dual pairs of LPs and QPs Nonlinear Stochastic Program with Recourse Interior Point Methods for Optimization unified view of Linear, Quadratic and Nonlinear programming SPS 2007, Bergamo, April

3 Consider the general optimization problem min f(x) s.t. g(x) 0, where x R n, and f : R n R and g : R n R m are convex, twice differentiable. Basic Assumptions: f and g are convex If there exists a local minimum then it is a global one. f and g are twice differentiable We can use the second order Taylor approximations of them. SPS 2007, Bergamo, April

4 Convexity Reading: Bertsekas, D., Nonlinear Programming, Athena Scientific, Massachusetts, ISBN SPS 2007, Bergamo, April

5 Convexity is a key property in optimization. Def. A set C R n is convex if λx + (1 λ)y C, x, y C, λ [0, 1]. y z x z y x Convex set Nonconvex set Def. Let C be a convex subset of R n. A function f : C R is convex if f(λx + (1 λ)y) λf(x) + (1 λ)f(y), x, y C, λ [0, 1]. x z y x z Convex function Nonconvex function y SPS 2007, Bergamo, April

6 Convexity and Optimization Consider a problem minimize f(x) subject to x X, where X is a set of feasible solutions and f : X R is an objective function. Def. A vector ˆx is a local minimum of f if ɛ > 0 such that f(ˆx) f(x), x x ˆx < ɛ. Def. A vector ˆx is a global minimum of f if f(ˆx) f(x), x X. SPS 2007, Bergamo, April

7 Lemma. If X is a convex set and f : X R is a convex function, then a local minimum is a global minimum. Proof. Suppose that x is a local minimum, but not a global one. Then y x such that f(y)<f(x). From convexity of f, for all λ [0, 1], we have f((1 λ)x+λy) (1 λ)f(x)+λf(y) < (1 λ)f(x)+λf(x) = f(x). In particular, for a sufficiently small λ, the point z = (1 λ)x+λy lies in the ɛ-neighbourhood of x and f(z) < f(x). This contradicts the assumption that x is a local minimum. SPS 2007, Bergamo, April

8 Useful properties 1. For any collection {C i i I} of convex sets, the intersection i I C i is convex. 2. If C is a convex set and f : C R is a convex function, the level sets {x C f(x) α} and {x C f(x) < α} are convex for all scalars α. 3. Let C R n be a convex set and f : C R be differentiable over C. (a) The function f is convex if and only if f(y) f(x) + T f(x)(y x), x, y C. (b) If the inequality is strict for x y, then f is strictly convex. 4. Let C R n be a convex set and f : C R be twice continuously differentiable over C. (a) If 2 f(x) is positive semidefinite for all x C, then f is convex. (b) If 2 f(x) is positive definite for all x C, then f is strictly convex. (c) If f is convex, then 2 f(x) is positive semidefinite for all x C. SPS 2007, Bergamo, April

9 Lemma. If f : R n R and g : R n R m are convex, then the following general optimization problem min f(x) s.t. g(x) 0 is convex. Proof. Since the objective function f is convex, we only need to prove that the feasible set of the above problem is convex. Define for i = 1, 2,..., m From Property 2, X i is convex for all i. We observe that X = {x R n : g(x) 0} X i = {x R n : g i (x) 0}. X = {x R n : g i (x) 0, i = 1..m} = X i. i i.e., X is an intersection of convex sets and from Property 1, X is a convex set. SPS 2007, Bergamo, April

10 Duality Reading: Bertsekas, D., Nonlinear Programming, Athena Scientific, Massachusetts, ISBN SPS 2007, Bergamo, April

11 Consider a general optimization problem min f(x) s.t. g(x) 0, (1) x X R n, where f : R n R and g : R n R m. The set X is arbitrary; it may include, for example, an integrality constraint. Let ˆx be an optimal solution of (1) and define ˆf = f(ˆx). Introduce the Lagrange multiplier y i 0 for every inequality constraint g i (x) 0. Define y = (y 1,..., y m ) T and the Lagrangian y are also called dual variables. L(x, y) = f(x) + y T g(x), SPS 2007, Bergamo, April

12 Consider the problem L D (y) = min x L(x, y) s.t. x X R n. Its optimal solution x depends on y and so does the optimal objective L D (y). Lemma. For any y 0, L D (y) is a lower bound on ˆf (the optimal solution of (1)), i.e., ˆf L D (y) y 0. Proof. ˆf = min {f(x) g(x) 0, x X} min { f(x) + y T g(x) g(x) 0, y 0, x X } min { f(x) + y T g(x) y 0, x X } = L D (y). Corollary. ˆf max L D(y), i.e., ˆf max min L(x, y). y 0 y 0 x X SPS 2007, Bergamo, April

13 Lagrangian Duality If i g i (x) > 0, then max y 0 L(x, y) = + (we let the corresponding y i grow to + ). If i g i (x) 0, then L(x, y) = f(x), max y 0 because i y i g i (x) 0 and the maximum is attained when y i g i (x) = 0, i = 1, 2,..., m. Hence the problem (1) is equivalent to the following MinMax problem which could also be written as follows: min max L(x, y), x X y 0 ˆf = min max L(x, y). x X y 0 SPS 2007, Bergamo, April

14 Consider the following problem min {f(x) g(x) 0, x X}, where f, g and X are arbitrary. With this problem we associate the Lagrangian L(x, y) = f(x) + y T g(x), y are dual variables (Lagrange multipliers). The weak duality always holds: min max L(x, y) max x X y 0 y 0 min x X L(x, y). We have not made any assumption about functions f and g and set X. If f and g are convex, X is convex and certain regularity conditions are satisfied, then min max L(x, y) = max min L(x, y). x X y 0 y 0 x X This is called the strong duality. SPS 2007, Bergamo, April

15 Duality and Convexity The weak duality holds regardless of the form of functions f, g and set X: min max L(x, y) max x X y 0 y 0 min x X L(x, y). What do we need for the inequality in the weak duality to become an equation? If X R n is convex; f and g are convex; optimal solution is finite; some mysterious regularity conditions hold, then strong duality holds. That is min max L(x, y) = max x X y 0 y 0 An example of regularity conditions: x int(x) such that g(x) < 0. min x X L(x, y). SPS 2007, Bergamo, April

16 Lagrange duality does not need differentiability. Suppose f and g are convex and differentiable. Suppose X is convex. The dual function L D (y) = min x X requires minimization with respect to x. Instead of minimization with respect to x, we ask for a stationarity with respect to x: L(x, y). x L(x, y) = 0. Lagrange dual problem: max y 0 Wolfe dual problem: L D(y) ( i.e., max y 0 min x X ) L(x, y). max L(x, y) s.t. x L(x, y) = 0 y 0. SPS 2007, Bergamo, April

17 Dual Linear Program Consider a linear program min c T x s.t. Ax = b, x 0, where c, x R n, b R m, A R m n. We associate Lagrange multipliers y R m and s R n (s 0) with the constraints Ax = b and x 0, and write the Lagrangian L(x, y, s) = c T x y T (Ax b) s T x. To determine the Lagrangian dual L D (y, s) = min L(x, y, s) x X we need stationarity with respect to x: x L(x, y, s) = c A T y s = 0. SPS 2007, Bergamo, April

18 Hence and the dual LP has a form: where y R m and s R n. L D (y, s) = c T x y T (Ax b) s T x = b T y + x T (c A T y s) = b T y. max b T y s.t. A T y + s = c, y free, s 0, Primal Problem Dual Problem min c T x max b T y s.t. Ax = b, s.t. A T y + s = c, x 0; s 0. SPS 2007, Bergamo, April

19 Dual Quadratic Program Consider a quadratic program min c T x xt Q x s.t. Ax = b, x 0, where c, x R n, b R m, A R m n, Q R n n. We associate Lagrange multipliers y R m and s R n (s 0) with the constraints Ax = b and x 0, and write the Lagrangian L(x, y, s) = c T x xt Q x y T (Ax b) s T x. To determine the Lagrangian dual L D (y, s) = min L(x, y, s) x X we need stationarity with respect to x: x L(x, y, s) = c + Qx A T y s = 0. SPS 2007, Bergamo, April

20 Hence L D (y, s) = c T x xt Q x y T (Ax b) s T x = b T y + x T (c + Qx A T y s) 1 2 xt Q x = b T y 1 2 xt Q x, and the dual QP has the form: max b T y 1 2 xt Q x s.t. A T y + s Qx = c, x, s 0, where y R m and x, s R n. Primal Problem Dual Problem min c T x xt Qx max b T y 1 2 xt Qx s.t. Ax = b, s.t. A T y + s Qx = c, x 0; s 0. SPS 2007, Bergamo, April

21 General Stochastic Program with Recourse Reading: Kall P. and S.W. Wallace., Stochastic Programming, John Wiley & Sons, Chichester 1994, UK. SPS 2007, Bergamo, April

22 General Stochastic Program with Recourse Consider a deterministic problem minimize f(x) subject to g(x) 0, x X, where f : R n R is an objective function, and g i : R n R, i = 1...m are constraints. Consider its stochastic analogue minimize f(x, ξ) subject to g(x, ξ) 0, x X, where ξ is a random vector varying over a set Ξ R k, f : R n Ξ R is an objective function, and g i : R n Ξ R, i = 1...m are constraints. SPS 2007, Bergamo, April

23 Define and a nonlinear recourse function g + i (x, ξ) = { 0 if gi (x, ξ) 0, g i (x, ξ) otherwise. Q(x, ξ) = min{q(y) u i (y) g + i (x, ξ), i = 1..m, y Y R n }, where q : R n R and u i : R n R are given. Replace original stochastic program by stochastic program with recourse min x X E ξ{f(x, ξ) + Q(x, ξ)}. Convexity of Recourse Problems Lemma. If the functions q(.) and g i (., ξ), i = 1..m are convex and the functions u i (.), i = 1..m are concave, then the nonlinear recourse function Q(x, ξ) = min{q(y) u i (y) g + i (x, ξ), i = 1..m, y Y R n } (2) is convex with respect to its first argument. SPS 2007, Bergamo, April

24 Proof. Let y 1 and y 2 be the optimal solutions of the recourse problems for x 1 and x 2, respectively. By the convexity of g i (., ξ) and the concavity of u i (.) we have for any λ [0, 1]: g i (λx 1 + (1 λ)x 2, ξ) λg i (x 1, ξ) + (1 λ)g i (x 2, ξ) λu i (y 1 ) + (1 λ)u i (y 2 ) u i (λy 1 + (1 λ)y 2 ). Hence ȳ = λy 1 + (1 λ)y 2 is feasible in (2) for x = λx 1 + (1 λ)x 2 and by convexity of q(.) which completes the proof. Q( x, ξ) q(ȳ) λq(y 1 ) + (1 λ)q(y 2 ) λq(x 1, ξ) + (1 λ)q(x 2, ξ), SPS 2007, Bergamo, April

25 Lemma. If f(., ξ) and Q(., ξ) are convex in x ξ Ξ, and if X is a convex set, then E ξ{f(x, ξ) + Q(x, ξ)}. min x X is a convex program. Proof. Take x, y X, λ [0, 1] and z = λx+(1 λ)y. From convexity of f(., ξ) and Q(., ξ) with respect to their first argument, ξ Ξ, we have: and f(z, ξ) λf(x, ξ) + (1 λ)f(y, ξ) Q(z, ξ) λq(x, ξ) + (1 λ)q(y, ξ), respectively. Adding these inequalities, we obtain ξ Ξ implying f(z, ξ) + Q(z, ξ) λ[f(x, ξ) + Q(x, ξ)] + (1 λ)[f(y, ξ) + Q(y, ξ)], E ξ {f(z, ξ)+q(z, ξ)} λe ξ {f(x, ξ)+q(x, ξ)}+(1 λ)e ξ {f(y, Hence the objective in stochastic program with recourse is convex. ξ)+q(y, ξ)}. SPS 2007, Bergamo, April

26 Interior Point Methods from LP via QP to NLP Reading: Wright S., Primal-Dual Interior-Point Methods, SIAM, Nocedal J. & S. Wright, Numerical Optimization, Springer-Verlag, Conn A., N. Gould & Ph. Toint, Trust-Region Methods, SIAM, SPS 2007, Bergamo, April

27 Logarithmic barrier replaces the inequality x j 0. ln x j ln x 1 x Observe that min e n j=1 ln x j max n j=1 x j The minimization of n j=1 ln x j is equivalent to the maximization of the product of distances from all hyperplanes defining the positive orthant: it prevents all x j from approaching zero. SPS 2007, Bergamo, April

28 Use Logarithmic Barrier Primal Problem Dual Problem min c T x max b T y s.t. Ax = b, s.t. A T y + s = c, x 0; s 0. Primal Barrier Problem Dual Barrier Problem min c T x n ln x j max b T y + n j=1 j=1 ln s j s.t. Ax = b, s.t. A T y + s = c, SPS 2007, Bergamo, April

29 Primal Barrier Problem: min c T x µ n j=1 ln x j s.t. Ax = b. n Lagrangian: L(x, y, µ) = c T x y T (Ax b) µ ln x j, Stationarity: x L(x, y, µ) = c A T y µx 1 e = 0 j=1 Denote: s = µx 1 e, i.e. XSe = µe. The First Order Optimality Conditions are: Ax = b, A T y + s = c, XSe = µe (x, s) > 0. SPS 2007, Bergamo, April

30 First Order Optimality Conditions Simplex Method: Interior Point Method: Ax = b A T y + s = c XSe = 0 x, s 0. Ax = b A T y + s = c XSe = µe x, s 0. s s s s x Basic: x > 0, s = 0 Nonbasic: x = 0, s > 0 x x "Basic": x > 0, s = 0 "Nonbasic": x = 0, s > 0 x Theory: IPMs converge in O( n) or O(n) iterations Practice: IPMs converge in O(log n) iterations... but one iteration may be expensive! SPS 2007, Bergamo, April

31 Newton Method We use Newton Method to find a stationary point of the barrier problem. z f(x) Find a root of a nonlinear equation k f(x ) k z-f(x ) = f(x )(x-x ) k k A tangent line f(x) = 0. k+1 f(x ) k+2 f(x ) x k x k+1 x k+2 x z f(x k ) = f(x k ) (x x k ) is a local approximation of the graph of the function f(x). Substite z = 0 to get a new point x k+1 = x k ( f(x k )) 1 f(x k ). SPS 2007, Bergamo, April

32 Newton Method The first order optimality conditions for the barrier problem form a large system of nonlinear equations F (x, y, s) = 0, where F : R 2n+m R 2n+m is an application defined as follows: Ax b F (x, y, s) = A T y + s c. XSe µe Actually, the first two terms of it are linear; only the last one, corresponding to the complementarity condition, is nonlinear. For a given point (x, y, s) we find the Newton direction ( x, y, s) by solving the system of linear equations: A 0 0 x b Ax 0 A T I y = c A T y s. S 0 X s µe XSe SPS 2007, Bergamo, April

33 IPM for LP min c T x min c T x µ n j=1 ln x j s.t. Ax = b, s.t. Ax = b, x 0. The first order conditions (for the barrier problem) Ax = b, A T y + s = c, XSe = µe. Newton direction A A T I S 0 X x y s = ξ p ξ d ξ µ = b Ax c A T y s µe XSe. Augmented system [ Θ 1 A T A 0 ] [ x y ] = [ ξd X 1 ξ µ ξ p ]. SPS 2007, Bergamo, April

34 IPM for QP min c T x xt Q x min c T x xt Q x µ n j=1 ln x j s.t. Ax = b, s.t. Ax = b, x 0. The first order conditions (for the barrier problem) Ax = b, A T y + s Qx = c, XSe = µe. Newton direction A 0 0 Q A T I S 0 X x y s = ξ p ξ d ξ µ = b Ax c A T y s+qx µe XSe. Augmented system [ Q Θ 1 A T A 0 ] [ x y ] = [ ξd X 1 ξ µ ξ p ]. SPS 2007, Bergamo, April

35 IPM for NLP min f(x) min f(x) µ m s.t. g(x) + z = 0 s.t. g(x) + z = 0, z 0. Lagrangian: L(x, y, z, µ) = f(x) + y T (g(x) + z) µ m ln z i. The first order conditions (for the barrier problem) f(x) + g(x) T y = 0, g(x) + z = 0, Y Ze = µe. Newton direction Q(x, y) A(x) T 0 A(x) 0 I 0 Z Y Augmented system [ ] [ ] Q(x, y) A(x) T x A(x) ZY 1 = y x y = z [ f(x) A(x) T y g(x) µy 1 e i=1 f(x) A(x) T y g(x) z µe Y Ze ] where. i=1 ln z i A(x) = g Q(x, y) = 2 xxl SPS 2007, Bergamo, April

36 Generic Interior-Point NLP Algorithm Initialize k = 0 (x 0, y 0, z 0 ) such that y 0 > 0 and z 0 > 0, µ 0 = 1 m (y0 ) T z 0 Repeat until optimality k = k + 1 µ k = σµ k 1, where σ (0, 1) Compute A(x) and Q(x, y) = Newton direction towards µ-center Ratio test: α 1 := max {α > 0 : y + α y 0}, α 2 := max {α > 0 : z + α z 0}. Choose the step: α min {α 1, α 2 } (use trust region or line search) Make step: x k+1 = x k + α x, y k+1 = y k + α y, z k+1 = z k + α z. SPS 2007, Bergamo, April

Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming

Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

More information

Interior Point Methods for Linear Programming: Motivation & Theory

Interior Point Methods for Linear Programming: Motivation & Theory School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Linear Programming: Motivation & Theory Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

More information

Lecture 3. Optimization Problems and Iterative Algorithms

Lecture 3. Optimization Problems and Iterative Algorithms Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex

More information

Numerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen

Numerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen

More information

4TE3/6TE3. Algorithms for. Continuous Optimization

4TE3/6TE3. Algorithms for. Continuous Optimization 4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality

More information

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness. CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

More information

Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

More information

Duality Theory of Constrained Optimization

Duality Theory of Constrained Optimization Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive

More information

UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems

UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction

More information

Applications of Linear Programming

Applications of Linear Programming Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal

More information

Examination paper for TMA4180 Optimization I

Examination paper for TMA4180 Optimization I Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted

More information

Nonlinear Programming

Nonlinear Programming Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week

More information

CS711008Z Algorithm Design and Analysis

CS711008Z Algorithm Design and Analysis CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief

More information

Lectures 9 and 10: Constrained optimization problems and their optimality conditions

Lectures 9 and 10: Constrained optimization problems and their optimality conditions Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained

More information

Algorithms for nonlinear programming problems II

Algorithms for nonlinear programming problems II Algorithms for nonlinear programming problems II Martin Branda Charles University Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects of Optimization

More information

Nonlinear Programming Models

Nonlinear Programming Models Nonlinear Programming Models Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Nonlinear Programming Models p. Introduction Nonlinear Programming Models p. NLP problems minf(x) x S R n Standard form:

More information

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE

LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization

More information

Algorithms for nonlinear programming problems II

Algorithms for nonlinear programming problems II Algorithms for nonlinear programming problems II Martin Branda Charles University in Prague Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects

More information

Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark

Solution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT

More information

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

More information

Lecture 18: Optimization Programming

Lecture 18: Optimization Programming Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

More information

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010 I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

More information

Algorithms for constrained local optimization

Algorithms for constrained local optimization Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained

More information

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

More information

10 Numerical methods for constrained problems

10 Numerical methods for constrained problems 10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside

More information

SF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren

SF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren SF2822 Applied Nonlinear Optimization Lecture 9: Sequential quadratic programming Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH / 24 Lecture 9, 207/208 Preparatory question. Try to solve theory

More information

Solution Methods for Stochastic Programs

Solution Methods for Stochastic Programs Solution Methods for Stochastic Programs Huseyin Topaloglu School of Operations Research and Information Engineering Cornell University ht88@cornell.edu August 14, 2010 1 Outline Cutting plane methods

More information

Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version

Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

More information

Optimization for Machine Learning

Optimization for Machine Learning Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

More information

Lecture Note 5: Semidefinite Programming for Stability Analysis

Lecture Note 5: Semidefinite Programming for Stability Analysis ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State

More information

Convex Optimization M2

Convex Optimization M2 Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

More information

Convex Functions and Optimization

Convex Functions and Optimization Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized

More information

5. Duality. Lagrangian

5. Duality. Lagrangian 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Constrained Optimization and Lagrangian Duality

Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

More information

Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming

Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

More information

Second-order cone programming

Second-order cone programming Outline Second-order cone programming, PhD Lehigh University Department of Industrial and Systems Engineering February 10, 2009 Outline 1 Basic properties Spectral decomposition The cone of squares The

More information

Math 273a: Optimization Basic concepts

Math 273a: Optimization Basic concepts Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize

More information

Operations Research Lecture 4: Linear Programming Interior Point Method

Operations Research Lecture 4: Linear Programming Interior Point Method Operations Research Lecture 4: Linear Programg Interior Point Method Notes taen by Kaiquan Xu@Business School, Nanjing University April 14th 2016 1 The affine scaling algorithm one of the most efficient

More information

Optimisation in Higher Dimensions

Optimisation in Higher Dimensions CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained

More information

12. Interior-point methods

12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

More information

Lagrange duality. The Lagrangian. We consider an optimization program of the form

Lagrange duality. The Lagrangian. We consider an optimization program of the form Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

More information

Linear and non-linear programming

Linear and non-linear programming Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

More information

Duality. Geoff Gordon & Ryan Tibshirani Optimization /

Duality. Geoff Gordon & Ryan Tibshirani Optimization / Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider

More information

Two hours. To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER. xx xxxx 2017 xx:xx xx.

Two hours. To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER. xx xxxx 2017 xx:xx xx. Two hours To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER CONVEX OPTIMIZATION - SOLUTIONS xx xxxx 27 xx:xx xx.xx Answer THREE of the FOUR questions. If

More information

LP. Kap. 17: Interior-point methods

LP. Kap. 17: Interior-point methods LP. Kap. 17: Interior-point methods the simplex algorithm moves along the boundary of the polyhedron P of feasible solutions an alternative is interior-point methods they find a path in the interior of

More information

Nonlinear Optimization: What s important?

Nonlinear Optimization: What s important? Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global

More information

1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016

1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016 AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the

More information

minimize x subject to (x 2)(x 4) u,

minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

More information

Convex Optimization Boyd & Vandenberghe. 5. Duality

Convex Optimization Boyd & Vandenberghe. 5. Duality 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Lecture 8. Strong Duality Results. September 22, 2008

Lecture 8. Strong Duality Results. September 22, 2008 Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation

More information

Additional Homework Problems

Additional Homework Problems Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.

More information

Optimization Theory. Lectures 4-6

Optimization Theory. Lectures 4-6 Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:

More information

Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers

Optimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex

More information

Lecture 2: Convex Sets and Functions

Lecture 2: Convex Sets and Functions Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are

More information

SF2822 Applied nonlinear optimization, final exam Wednesday June

SF2822 Applied nonlinear optimization, final exam Wednesday June SF2822 Applied nonlinear optimization, final exam Wednesday June 3 205 4.00 9.00 Examiner: Anders Forsgren, tel. 08-790 7 27. Allowed tools: Pen/pencil, ruler and eraser. Note! Calculator is not allowed.

More information

Optimization. Charles J. Geyer School of Statistics University of Minnesota. Stat 8054 Lecture Notes

Optimization. Charles J. Geyer School of Statistics University of Minnesota. Stat 8054 Lecture Notes Optimization Charles J. Geyer School of Statistics University of Minnesota Stat 8054 Lecture Notes 1 One-Dimensional Optimization Look at a graph. Grid search. 2 One-Dimensional Zero Finding Zero finding

More information

Applied Lagrange Duality for Constrained Optimization

Applied Lagrange Duality for Constrained Optimization Applied Lagrange Duality for Constrained Optimization February 12, 2002 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual

More information

Miscellaneous Nonlinear Programming Exercises

Miscellaneous Nonlinear Programming Exercises Miscellaneous Nonlinear Programming Exercises Henry Wolkowicz 2 08 21 University of Waterloo Department of Combinatorics & Optimization Waterloo, Ontario N2L 3G1, Canada Contents 1 Numerical Analysis Background

More information

Linear and Combinatorial Optimization

Linear and Combinatorial Optimization Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality

More information

Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization

Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department

More information

Generalization to inequality constrained problem. Maximize

Generalization to inequality constrained problem. Maximize Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum

More information

Pacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT

Pacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT Pacific Journal of Optimization Vol., No. 3, September 006) PRIMAL ERROR BOUNDS BASED ON THE AUGMENTED LAGRANGIAN AND LAGRANGIAN RELAXATION ALGORITHMS A. F. Izmailov and M. V. Solodov ABSTRACT For a given

More information

Lecture: Algorithms for LP, SOCP and SDP

Lecture: Algorithms for LP, SOCP and SDP 1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:

More information

LECTURE 10 LECTURE OUTLINE

LECTURE 10 LECTURE OUTLINE LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,

More information

Convergence Analysis of Inexact Infeasible Interior Point Method. for Linear Optimization

Convergence Analysis of Inexact Infeasible Interior Point Method. for Linear Optimization Convergence Analysis of Inexact Infeasible Interior Point Method for Linear Optimization Ghussoun Al-Jeiroudi Jacek Gondzio School of Mathematics The University of Edinburgh Mayfield Road, Edinburgh EH9

More information

Symmetric Matrices and Eigendecomposition

Symmetric Matrices and Eigendecomposition Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions

More information

Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs

Introduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following

More information

LMI Methods in Optimal and Robust Control

LMI Methods in Optimal and Robust Control LMI Methods in Optimal and Robust Control Matthew M. Peet Arizona State University Lecture 02: Optimization (Convex and Otherwise) What is Optimization? An Optimization Problem has 3 parts. x F f(x) :

More information

Optimization Problems with Constraints - introduction to theory, numerical Methods and applications

Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)

More information

BASICS OF CONVEX ANALYSIS

BASICS OF CONVEX ANALYSIS BASICS OF CONVEX ANALYSIS MARKUS GRASMAIR 1. Main Definitions We start with providing the central definitions of convex functions and convex sets. Definition 1. A function f : R n R + } is called convex,

More information

We describe the generalization of Hazan s algorithm for symmetric programming

We describe the generalization of Hazan s algorithm for symmetric programming ON HAZAN S ALGORITHM FOR SYMMETRIC PROGRAMMING PROBLEMS L. FAYBUSOVICH Abstract. problems We describe the generalization of Hazan s algorithm for symmetric programming Key words. Symmetric programming,

More information

Convex Optimization & Lagrange Duality

Convex Optimization & Lagrange Duality Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

More information

Optimality Conditions for Constrained Optimization

Optimality Conditions for Constrained Optimization 72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

More information

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014 Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

More information

Interior-Point Methods

Interior-Point Methods Interior-Point Methods Stephen Wright University of Wisconsin-Madison Simons, Berkeley, August, 2017 Wright (UW-Madison) Interior-Point Methods August 2017 1 / 48 Outline Introduction: Problems and Fundamentals

More information

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016

Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,

More information

arxiv: v1 [math.oc] 1 Jul 2016

arxiv: v1 [math.oc] 1 Jul 2016 Convergence Rate of Frank-Wolfe for Non-Convex Objectives Simon Lacoste-Julien INRIA - SIERRA team ENS, Paris June 8, 016 Abstract arxiv:1607.00345v1 [math.oc] 1 Jul 016 We give a simple proof that the

More information

Chapter 2 Convex Analysis

Chapter 2 Convex Analysis Chapter 2 Convex Analysis The theory of nonsmooth analysis is based on convex analysis. Thus, we start this chapter by giving basic concepts and results of convexity (for further readings see also [202,

More information

Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

More information

You should be able to...

You should be able to... Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set

More information

Computational Optimization. Mathematical Programming Fundamentals 1/25 (revised)

Computational Optimization. Mathematical Programming Fundamentals 1/25 (revised) Computational Optimization Mathematical Programming Fundamentals 1/5 (revised) If you don t know where you are going, you probably won t get there. -from some book I read in eight grade If you do get there,

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

More information

A Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions

A Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions A Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions Angelia Nedić and Asuman Ozdaglar April 16, 2006 Abstract In this paper, we study a unifying framework

More information

Lecture 3 Interior Point Methods and Nonlinear Optimization

Lecture 3 Interior Point Methods and Nonlinear Optimization Lecture 3 Interior Point Methods and Nonlinear Optimization Robert J. Vanderbei April 16, 2012 Machine Learning Summer School La Palma http://www.princeton.edu/ rvdb Example: Basis Pursuit Denoising L

More information

March 5, 2012 MATH 408 FINAL EXAM SAMPLE

March 5, 2012 MATH 408 FINAL EXAM SAMPLE March 5, 202 MATH 408 FINAL EXAM SAMPLE Partial Solutions to Sample Questions (in progress) See the sample questions for the midterm exam, but also consider the following questions. Obviously, a final

More information

Using interior point methods for optimization in training very large scale Support Vector Machines. Interior Point Methods for QP. Elements of the IPM

Using interior point methods for optimization in training very large scale Support Vector Machines. Interior Point Methods for QP. Elements of the IPM School of Mathematics T H E U N I V E R S I T Y O H Using interior point methods for optimization in training very large scale Support Vector Machines F E D I N U R Part : Interior Point Methods for QP

More information

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation

Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu

More information

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006

Quiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006 Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in

More information

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44 Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

More information

Enhanced Fritz John Optimality Conditions and Sensitivity Analysis

Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained

More information

Maximum likelihood estimation

Maximum likelihood estimation Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization

More information

CS-E4830 Kernel Methods in Machine Learning

CS-E4830 Kernel Methods in Machine Learning CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

More information

Lagrange Relaxation and Duality

Lagrange Relaxation and Duality Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler

More information

Lagrangian Duality Theory

Lagrangian Duality Theory Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual

More information

IE 521 Convex Optimization Homework #1 Solution

IE 521 Convex Optimization Homework #1 Solution IE 521 Convex Optimization Homework #1 Solution your NAME here your NetID here February 13, 2019 Instructions. Homework is due Wednesday, February 6, at 1:00pm; no late homework accepted. Please use the

More information

Algorithms for Constrained Optimization

Algorithms for Constrained Optimization 1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic

More information

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

More information

Mathematical Economics. Lecture Notes (in extracts)

Mathematical Economics. Lecture Notes (in extracts) Prof. Dr. Frank Werner Faculty of Mathematics Institute of Mathematical Optimization (IMO) http://math.uni-magdeburg.de/ werner/math-ec-new.html Mathematical Economics Lecture Notes (in extracts) Winter

More information

Lecture: Duality.

Lecture: Duality. Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

More information