Introduction to Nonlinear Stochastic Programming
|
|
- Bartholomew Barrett
- 5 years ago
- Views:
Transcription
1 School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio J.Gondzio@ed.ac.uk URL: SPS 2007, Bergamo, April
2 Outline Convexity convex sets, convex functions local optimum, global optimum Duality Lagrange duality Wolfe duality primal-dual pairs of LPs and QPs Nonlinear Stochastic Program with Recourse Interior Point Methods for Optimization unified view of Linear, Quadratic and Nonlinear programming SPS 2007, Bergamo, April
3 Consider the general optimization problem min f(x) s.t. g(x) 0, where x R n, and f : R n R and g : R n R m are convex, twice differentiable. Basic Assumptions: f and g are convex If there exists a local minimum then it is a global one. f and g are twice differentiable We can use the second order Taylor approximations of them. SPS 2007, Bergamo, April
4 Convexity Reading: Bertsekas, D., Nonlinear Programming, Athena Scientific, Massachusetts, ISBN SPS 2007, Bergamo, April
5 Convexity is a key property in optimization. Def. A set C R n is convex if λx + (1 λ)y C, x, y C, λ [0, 1]. y z x z y x Convex set Nonconvex set Def. Let C be a convex subset of R n. A function f : C R is convex if f(λx + (1 λ)y) λf(x) + (1 λ)f(y), x, y C, λ [0, 1]. x z y x z Convex function Nonconvex function y SPS 2007, Bergamo, April
6 Convexity and Optimization Consider a problem minimize f(x) subject to x X, where X is a set of feasible solutions and f : X R is an objective function. Def. A vector ˆx is a local minimum of f if ɛ > 0 such that f(ˆx) f(x), x x ˆx < ɛ. Def. A vector ˆx is a global minimum of f if f(ˆx) f(x), x X. SPS 2007, Bergamo, April
7 Lemma. If X is a convex set and f : X R is a convex function, then a local minimum is a global minimum. Proof. Suppose that x is a local minimum, but not a global one. Then y x such that f(y)<f(x). From convexity of f, for all λ [0, 1], we have f((1 λ)x+λy) (1 λ)f(x)+λf(y) < (1 λ)f(x)+λf(x) = f(x). In particular, for a sufficiently small λ, the point z = (1 λ)x+λy lies in the ɛ-neighbourhood of x and f(z) < f(x). This contradicts the assumption that x is a local minimum. SPS 2007, Bergamo, April
8 Useful properties 1. For any collection {C i i I} of convex sets, the intersection i I C i is convex. 2. If C is a convex set and f : C R is a convex function, the level sets {x C f(x) α} and {x C f(x) < α} are convex for all scalars α. 3. Let C R n be a convex set and f : C R be differentiable over C. (a) The function f is convex if and only if f(y) f(x) + T f(x)(y x), x, y C. (b) If the inequality is strict for x y, then f is strictly convex. 4. Let C R n be a convex set and f : C R be twice continuously differentiable over C. (a) If 2 f(x) is positive semidefinite for all x C, then f is convex. (b) If 2 f(x) is positive definite for all x C, then f is strictly convex. (c) If f is convex, then 2 f(x) is positive semidefinite for all x C. SPS 2007, Bergamo, April
9 Lemma. If f : R n R and g : R n R m are convex, then the following general optimization problem min f(x) s.t. g(x) 0 is convex. Proof. Since the objective function f is convex, we only need to prove that the feasible set of the above problem is convex. Define for i = 1, 2,..., m From Property 2, X i is convex for all i. We observe that X = {x R n : g(x) 0} X i = {x R n : g i (x) 0}. X = {x R n : g i (x) 0, i = 1..m} = X i. i i.e., X is an intersection of convex sets and from Property 1, X is a convex set. SPS 2007, Bergamo, April
10 Duality Reading: Bertsekas, D., Nonlinear Programming, Athena Scientific, Massachusetts, ISBN SPS 2007, Bergamo, April
11 Consider a general optimization problem min f(x) s.t. g(x) 0, (1) x X R n, where f : R n R and g : R n R m. The set X is arbitrary; it may include, for example, an integrality constraint. Let ˆx be an optimal solution of (1) and define ˆf = f(ˆx). Introduce the Lagrange multiplier y i 0 for every inequality constraint g i (x) 0. Define y = (y 1,..., y m ) T and the Lagrangian y are also called dual variables. L(x, y) = f(x) + y T g(x), SPS 2007, Bergamo, April
12 Consider the problem L D (y) = min x L(x, y) s.t. x X R n. Its optimal solution x depends on y and so does the optimal objective L D (y). Lemma. For any y 0, L D (y) is a lower bound on ˆf (the optimal solution of (1)), i.e., ˆf L D (y) y 0. Proof. ˆf = min {f(x) g(x) 0, x X} min { f(x) + y T g(x) g(x) 0, y 0, x X } min { f(x) + y T g(x) y 0, x X } = L D (y). Corollary. ˆf max L D(y), i.e., ˆf max min L(x, y). y 0 y 0 x X SPS 2007, Bergamo, April
13 Lagrangian Duality If i g i (x) > 0, then max y 0 L(x, y) = + (we let the corresponding y i grow to + ). If i g i (x) 0, then L(x, y) = f(x), max y 0 because i y i g i (x) 0 and the maximum is attained when y i g i (x) = 0, i = 1, 2,..., m. Hence the problem (1) is equivalent to the following MinMax problem which could also be written as follows: min max L(x, y), x X y 0 ˆf = min max L(x, y). x X y 0 SPS 2007, Bergamo, April
14 Consider the following problem min {f(x) g(x) 0, x X}, where f, g and X are arbitrary. With this problem we associate the Lagrangian L(x, y) = f(x) + y T g(x), y are dual variables (Lagrange multipliers). The weak duality always holds: min max L(x, y) max x X y 0 y 0 min x X L(x, y). We have not made any assumption about functions f and g and set X. If f and g are convex, X is convex and certain regularity conditions are satisfied, then min max L(x, y) = max min L(x, y). x X y 0 y 0 x X This is called the strong duality. SPS 2007, Bergamo, April
15 Duality and Convexity The weak duality holds regardless of the form of functions f, g and set X: min max L(x, y) max x X y 0 y 0 min x X L(x, y). What do we need for the inequality in the weak duality to become an equation? If X R n is convex; f and g are convex; optimal solution is finite; some mysterious regularity conditions hold, then strong duality holds. That is min max L(x, y) = max x X y 0 y 0 An example of regularity conditions: x int(x) such that g(x) < 0. min x X L(x, y). SPS 2007, Bergamo, April
16 Lagrange duality does not need differentiability. Suppose f and g are convex and differentiable. Suppose X is convex. The dual function L D (y) = min x X requires minimization with respect to x. Instead of minimization with respect to x, we ask for a stationarity with respect to x: L(x, y). x L(x, y) = 0. Lagrange dual problem: max y 0 Wolfe dual problem: L D(y) ( i.e., max y 0 min x X ) L(x, y). max L(x, y) s.t. x L(x, y) = 0 y 0. SPS 2007, Bergamo, April
17 Dual Linear Program Consider a linear program min c T x s.t. Ax = b, x 0, where c, x R n, b R m, A R m n. We associate Lagrange multipliers y R m and s R n (s 0) with the constraints Ax = b and x 0, and write the Lagrangian L(x, y, s) = c T x y T (Ax b) s T x. To determine the Lagrangian dual L D (y, s) = min L(x, y, s) x X we need stationarity with respect to x: x L(x, y, s) = c A T y s = 0. SPS 2007, Bergamo, April
18 Hence and the dual LP has a form: where y R m and s R n. L D (y, s) = c T x y T (Ax b) s T x = b T y + x T (c A T y s) = b T y. max b T y s.t. A T y + s = c, y free, s 0, Primal Problem Dual Problem min c T x max b T y s.t. Ax = b, s.t. A T y + s = c, x 0; s 0. SPS 2007, Bergamo, April
19 Dual Quadratic Program Consider a quadratic program min c T x xt Q x s.t. Ax = b, x 0, where c, x R n, b R m, A R m n, Q R n n. We associate Lagrange multipliers y R m and s R n (s 0) with the constraints Ax = b and x 0, and write the Lagrangian L(x, y, s) = c T x xt Q x y T (Ax b) s T x. To determine the Lagrangian dual L D (y, s) = min L(x, y, s) x X we need stationarity with respect to x: x L(x, y, s) = c + Qx A T y s = 0. SPS 2007, Bergamo, April
20 Hence L D (y, s) = c T x xt Q x y T (Ax b) s T x = b T y + x T (c + Qx A T y s) 1 2 xt Q x = b T y 1 2 xt Q x, and the dual QP has the form: max b T y 1 2 xt Q x s.t. A T y + s Qx = c, x, s 0, where y R m and x, s R n. Primal Problem Dual Problem min c T x xt Qx max b T y 1 2 xt Qx s.t. Ax = b, s.t. A T y + s Qx = c, x 0; s 0. SPS 2007, Bergamo, April
21 General Stochastic Program with Recourse Reading: Kall P. and S.W. Wallace., Stochastic Programming, John Wiley & Sons, Chichester 1994, UK. SPS 2007, Bergamo, April
22 General Stochastic Program with Recourse Consider a deterministic problem minimize f(x) subject to g(x) 0, x X, where f : R n R is an objective function, and g i : R n R, i = 1...m are constraints. Consider its stochastic analogue minimize f(x, ξ) subject to g(x, ξ) 0, x X, where ξ is a random vector varying over a set Ξ R k, f : R n Ξ R is an objective function, and g i : R n Ξ R, i = 1...m are constraints. SPS 2007, Bergamo, April
23 Define and a nonlinear recourse function g + i (x, ξ) = { 0 if gi (x, ξ) 0, g i (x, ξ) otherwise. Q(x, ξ) = min{q(y) u i (y) g + i (x, ξ), i = 1..m, y Y R n }, where q : R n R and u i : R n R are given. Replace original stochastic program by stochastic program with recourse min x X E ξ{f(x, ξ) + Q(x, ξ)}. Convexity of Recourse Problems Lemma. If the functions q(.) and g i (., ξ), i = 1..m are convex and the functions u i (.), i = 1..m are concave, then the nonlinear recourse function Q(x, ξ) = min{q(y) u i (y) g + i (x, ξ), i = 1..m, y Y R n } (2) is convex with respect to its first argument. SPS 2007, Bergamo, April
24 Proof. Let y 1 and y 2 be the optimal solutions of the recourse problems for x 1 and x 2, respectively. By the convexity of g i (., ξ) and the concavity of u i (.) we have for any λ [0, 1]: g i (λx 1 + (1 λ)x 2, ξ) λg i (x 1, ξ) + (1 λ)g i (x 2, ξ) λu i (y 1 ) + (1 λ)u i (y 2 ) u i (λy 1 + (1 λ)y 2 ). Hence ȳ = λy 1 + (1 λ)y 2 is feasible in (2) for x = λx 1 + (1 λ)x 2 and by convexity of q(.) which completes the proof. Q( x, ξ) q(ȳ) λq(y 1 ) + (1 λ)q(y 2 ) λq(x 1, ξ) + (1 λ)q(x 2, ξ), SPS 2007, Bergamo, April
25 Lemma. If f(., ξ) and Q(., ξ) are convex in x ξ Ξ, and if X is a convex set, then E ξ{f(x, ξ) + Q(x, ξ)}. min x X is a convex program. Proof. Take x, y X, λ [0, 1] and z = λx+(1 λ)y. From convexity of f(., ξ) and Q(., ξ) with respect to their first argument, ξ Ξ, we have: and f(z, ξ) λf(x, ξ) + (1 λ)f(y, ξ) Q(z, ξ) λq(x, ξ) + (1 λ)q(y, ξ), respectively. Adding these inequalities, we obtain ξ Ξ implying f(z, ξ) + Q(z, ξ) λ[f(x, ξ) + Q(x, ξ)] + (1 λ)[f(y, ξ) + Q(y, ξ)], E ξ {f(z, ξ)+q(z, ξ)} λe ξ {f(x, ξ)+q(x, ξ)}+(1 λ)e ξ {f(y, Hence the objective in stochastic program with recourse is convex. ξ)+q(y, ξ)}. SPS 2007, Bergamo, April
26 Interior Point Methods from LP via QP to NLP Reading: Wright S., Primal-Dual Interior-Point Methods, SIAM, Nocedal J. & S. Wright, Numerical Optimization, Springer-Verlag, Conn A., N. Gould & Ph. Toint, Trust-Region Methods, SIAM, SPS 2007, Bergamo, April
27 Logarithmic barrier replaces the inequality x j 0. ln x j ln x 1 x Observe that min e n j=1 ln x j max n j=1 x j The minimization of n j=1 ln x j is equivalent to the maximization of the product of distances from all hyperplanes defining the positive orthant: it prevents all x j from approaching zero. SPS 2007, Bergamo, April
28 Use Logarithmic Barrier Primal Problem Dual Problem min c T x max b T y s.t. Ax = b, s.t. A T y + s = c, x 0; s 0. Primal Barrier Problem Dual Barrier Problem min c T x n ln x j max b T y + n j=1 j=1 ln s j s.t. Ax = b, s.t. A T y + s = c, SPS 2007, Bergamo, April
29 Primal Barrier Problem: min c T x µ n j=1 ln x j s.t. Ax = b. n Lagrangian: L(x, y, µ) = c T x y T (Ax b) µ ln x j, Stationarity: x L(x, y, µ) = c A T y µx 1 e = 0 j=1 Denote: s = µx 1 e, i.e. XSe = µe. The First Order Optimality Conditions are: Ax = b, A T y + s = c, XSe = µe (x, s) > 0. SPS 2007, Bergamo, April
30 First Order Optimality Conditions Simplex Method: Interior Point Method: Ax = b A T y + s = c XSe = 0 x, s 0. Ax = b A T y + s = c XSe = µe x, s 0. s s s s x Basic: x > 0, s = 0 Nonbasic: x = 0, s > 0 x x "Basic": x > 0, s = 0 "Nonbasic": x = 0, s > 0 x Theory: IPMs converge in O( n) or O(n) iterations Practice: IPMs converge in O(log n) iterations... but one iteration may be expensive! SPS 2007, Bergamo, April
31 Newton Method We use Newton Method to find a stationary point of the barrier problem. z f(x) Find a root of a nonlinear equation k f(x ) k z-f(x ) = f(x )(x-x ) k k A tangent line f(x) = 0. k+1 f(x ) k+2 f(x ) x k x k+1 x k+2 x z f(x k ) = f(x k ) (x x k ) is a local approximation of the graph of the function f(x). Substite z = 0 to get a new point x k+1 = x k ( f(x k )) 1 f(x k ). SPS 2007, Bergamo, April
32 Newton Method The first order optimality conditions for the barrier problem form a large system of nonlinear equations F (x, y, s) = 0, where F : R 2n+m R 2n+m is an application defined as follows: Ax b F (x, y, s) = A T y + s c. XSe µe Actually, the first two terms of it are linear; only the last one, corresponding to the complementarity condition, is nonlinear. For a given point (x, y, s) we find the Newton direction ( x, y, s) by solving the system of linear equations: A 0 0 x b Ax 0 A T I y = c A T y s. S 0 X s µe XSe SPS 2007, Bergamo, April
33 IPM for LP min c T x min c T x µ n j=1 ln x j s.t. Ax = b, s.t. Ax = b, x 0. The first order conditions (for the barrier problem) Ax = b, A T y + s = c, XSe = µe. Newton direction A A T I S 0 X x y s = ξ p ξ d ξ µ = b Ax c A T y s µe XSe. Augmented system [ Θ 1 A T A 0 ] [ x y ] = [ ξd X 1 ξ µ ξ p ]. SPS 2007, Bergamo, April
34 IPM for QP min c T x xt Q x min c T x xt Q x µ n j=1 ln x j s.t. Ax = b, s.t. Ax = b, x 0. The first order conditions (for the barrier problem) Ax = b, A T y + s Qx = c, XSe = µe. Newton direction A 0 0 Q A T I S 0 X x y s = ξ p ξ d ξ µ = b Ax c A T y s+qx µe XSe. Augmented system [ Q Θ 1 A T A 0 ] [ x y ] = [ ξd X 1 ξ µ ξ p ]. SPS 2007, Bergamo, April
35 IPM for NLP min f(x) min f(x) µ m s.t. g(x) + z = 0 s.t. g(x) + z = 0, z 0. Lagrangian: L(x, y, z, µ) = f(x) + y T (g(x) + z) µ m ln z i. The first order conditions (for the barrier problem) f(x) + g(x) T y = 0, g(x) + z = 0, Y Ze = µe. Newton direction Q(x, y) A(x) T 0 A(x) 0 I 0 Z Y Augmented system [ ] [ ] Q(x, y) A(x) T x A(x) ZY 1 = y x y = z [ f(x) A(x) T y g(x) µy 1 e i=1 f(x) A(x) T y g(x) z µe Y Ze ] where. i=1 ln z i A(x) = g Q(x, y) = 2 xxl SPS 2007, Bergamo, April
36 Generic Interior-Point NLP Algorithm Initialize k = 0 (x 0, y 0, z 0 ) such that y 0 > 0 and z 0 > 0, µ 0 = 1 m (y0 ) T z 0 Repeat until optimality k = k + 1 µ k = σµ k 1, where σ (0, 1) Compute A(x) and Q(x, y) = Newton direction towards µ-center Ratio test: α 1 := max {α > 0 : y + α y 0}, α 2 := max {α > 0 : z + α z 0}. Choose the step: α min {α 1, α 2 } (use trust region or line search) Make step: x k+1 = x k + α x, y k+1 = y k + α y, z k+1 = z k + α z. SPS 2007, Bergamo, April
Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Convex Quadratic and Convex Nonlinear Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationInterior Point Methods for Linear Programming: Motivation & Theory
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods for Linear Programming: Motivation & Theory Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationNumerisches Rechnen. (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang. Institut für Geometrie und Praktische Mathematik RWTH Aachen
Numerisches Rechnen (für Informatiker) M. Grepl P. Esser & G. Welper & L. Zhang Institut für Geometrie und Praktische Mathematik RWTH Aachen Wintersemester 2011/12 IGPM, RWTH Aachen Numerisches Rechnen
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More informationDuality Theory of Constrained Optimization
Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More informationApplications of Linear Programming
Applications of Linear Programming lecturer: András London University of Szeged Institute of Informatics Department of Computational Optimization Lecture 9 Non-linear programming In case of LP, the goal
More informationExamination paper for TMA4180 Optimization I
Department of Mathematical Sciences Examination paper for TMA4180 Optimization I Academic contact during examination: Phone: Examination date: 26th May 2016 Examination time (from to): 09:00 13:00 Permitted
More informationNonlinear Programming
Nonlinear Programming Kees Roos e-mail: C.Roos@ewi.tudelft.nl URL: http://www.isa.ewi.tudelft.nl/ roos LNMB Course De Uithof, Utrecht February 6 - May 8, A.D. 2006 Optimization Group 1 Outline for week
More informationCS711008Z Algorithm Design and Analysis
CS711008Z Algorithm Design and Analysis Lecture 8 Linear programming: interior point method Dongbo Bu Institute of Computing Technology Chinese Academy of Sciences, Beijing, China 1 / 31 Outline Brief
More informationLectures 9 and 10: Constrained optimization problems and their optimality conditions
Lectures 9 and 10: Constrained optimization problems and their optimality conditions Coralia Cartis, Mathematical Institute, University of Oxford C6.2/B2: Continuous Optimization Lectures 9 and 10: Constrained
More informationAlgorithms for nonlinear programming problems II
Algorithms for nonlinear programming problems II Martin Branda Charles University Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects of Optimization
More informationNonlinear Programming Models
Nonlinear Programming Models Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Nonlinear Programming Models p. Introduction Nonlinear Programming Models p. NLP problems minf(x) x S R n Standard form:
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationAlgorithms for nonlinear programming problems II
Algorithms for nonlinear programming problems II Martin Branda Charles University in Prague Faculty of Mathematics and Physics Department of Probability and Mathematical Statistics Computational Aspects
More informationSolution Methods. Richard Lusby. Department of Management Engineering Technical University of Denmark
Solution Methods Richard Lusby Department of Management Engineering Technical University of Denmark Lecture Overview (jg Unconstrained Several Variables Quadratic Programming Separable Programming SUMT
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationLecture 18: Optimization Programming
Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming
More informationI.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationAlgorithms for constrained local optimization
Algorithms for constrained local optimization Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Algorithms for constrained local optimization p. Feasible direction methods Algorithms for constrained
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More information10 Numerical methods for constrained problems
10 Numerical methods for constrained problems min s.t. f(x) h(x) = 0 (l), g(x) 0 (m), x X The algorithms can be roughly divided the following way: ˆ primal methods: find descent direction keeping inside
More informationSF2822 Applied Nonlinear Optimization. Preparatory question. Lecture 9: Sequential quadratic programming. Anders Forsgren
SF2822 Applied Nonlinear Optimization Lecture 9: Sequential quadratic programming Anders Forsgren SF2822 Applied Nonlinear Optimization, KTH / 24 Lecture 9, 207/208 Preparatory question. Try to solve theory
More informationSolution Methods for Stochastic Programs
Solution Methods for Stochastic Programs Huseyin Topaloglu School of Operations Research and Information Engineering Cornell University ht88@cornell.edu August 14, 2010 1 Outline Cutting plane methods
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationLecture Note 5: Semidefinite Programming for Stability Analysis
ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationConvex Functions and Optimization
Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationInterior Point Methods: Second-Order Cone Programming and Semidefinite Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationSecond-order cone programming
Outline Second-order cone programming, PhD Lehigh University Department of Industrial and Systems Engineering February 10, 2009 Outline 1 Basic properties Spectral decomposition The cone of squares The
More informationMath 273a: Optimization Basic concepts
Math 273a: Optimization Basic concepts Instructor: Wotao Yin Department of Mathematics, UCLA Spring 2015 slides based on Chong-Zak, 4th Ed. Goals of this lecture The general form of optimization: minimize
More informationOperations Research Lecture 4: Linear Programming Interior Point Method
Operations Research Lecture 4: Linear Programg Interior Point Method Notes taen by Kaiquan Xu@Business School, Nanjing University April 14th 2016 1 The affine scaling algorithm one of the most efficient
More informationOptimisation in Higher Dimensions
CHAPTER 6 Optimisation in Higher Dimensions Beyond optimisation in 1D, we will study two directions. First, the equivalent in nth dimension, x R n such that f(x ) f(x) for all x R n. Second, constrained
More information12. Interior-point methods
12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationDuality. Geoff Gordon & Ryan Tibshirani Optimization /
Duality Geoff Gordon & Ryan Tibshirani Optimization 10-725 / 36-725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider
More informationTwo hours. To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER. xx xxxx 2017 xx:xx xx.
Two hours To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER CONVEX OPTIMIZATION - SOLUTIONS xx xxxx 27 xx:xx xx.xx Answer THREE of the FOUR questions. If
More informationLP. Kap. 17: Interior-point methods
LP. Kap. 17: Interior-point methods the simplex algorithm moves along the boundary of the polyhedron P of feasible solutions an alternative is interior-point methods they find a path in the interior of
More informationNonlinear Optimization: What s important?
Nonlinear Optimization: What s important? Julian Hall 10th May 2012 Convexity: convex problems A local minimizer is a global minimizer A solution of f (x) = 0 (stationary point) is a minimizer A global
More information1 Overview. 2 A Characterization of Convex Functions. 2.1 First-order Taylor approximation. AM 221: Advanced Optimization Spring 2016
AM 221: Advanced Optimization Spring 2016 Prof. Yaron Singer Lecture 8 February 22nd 1 Overview In the previous lecture we saw characterizations of optimality in linear optimization, and we reviewed the
More informationminimize x subject to (x 2)(x 4) u,
Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationAdditional Homework Problems
Additional Homework Problems Robert M. Freund April, 2004 2004 Massachusetts Institute of Technology. 1 2 1 Exercises 1. Let IR n + denote the nonnegative orthant, namely IR + n = {x IR n x j ( ) 0,j =1,...,n}.
More informationOptimization Theory. Lectures 4-6
Optimization Theory Lectures 4-6 Unconstrained Maximization Problem: Maximize a function f:ú n 6 ú within a set A f ú n. Typically, A is ú n, or the non-negative orthant {x0ú n x$0} Existence of a maximum:
More informationOptimization for Communications and Networks. Poompat Saengudomlert. Session 4 Duality and Lagrange Multipliers
Optimization for Communications and Networks Poompat Saengudomlert Session 4 Duality and Lagrange Multipliers P Saengudomlert (2015) Optimization Session 4 1 / 14 24 Dual Problems Consider a primal convex
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More informationSF2822 Applied nonlinear optimization, final exam Wednesday June
SF2822 Applied nonlinear optimization, final exam Wednesday June 3 205 4.00 9.00 Examiner: Anders Forsgren, tel. 08-790 7 27. Allowed tools: Pen/pencil, ruler and eraser. Note! Calculator is not allowed.
More informationOptimization. Charles J. Geyer School of Statistics University of Minnesota. Stat 8054 Lecture Notes
Optimization Charles J. Geyer School of Statistics University of Minnesota Stat 8054 Lecture Notes 1 One-Dimensional Optimization Look at a graph. Grid search. 2 One-Dimensional Zero Finding Zero finding
More informationApplied Lagrange Duality for Constrained Optimization
Applied Lagrange Duality for Constrained Optimization February 12, 2002 Overview The Practical Importance of Duality ffl Review of Convexity ffl A Separating Hyperplane Theorem ffl Definition of the Dual
More informationMiscellaneous Nonlinear Programming Exercises
Miscellaneous Nonlinear Programming Exercises Henry Wolkowicz 2 08 21 University of Waterloo Department of Combinatorics & Optimization Waterloo, Ontario N2L 3G1, Canada Contents 1 Numerical Analysis Background
More informationLinear and Combinatorial Optimization
Linear and Combinatorial Optimization The dual of an LP-problem. Connections between primal and dual. Duality theorems and complementary slack. Philipp Birken (Ctr. for the Math. Sc.) Lecture 3: Duality
More informationPrimal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization
Primal-dual relationship between Levenberg-Marquardt and central trajectories for linearly constrained convex optimization Roger Behling a, Clovis Gonzaga b and Gabriel Haeser c March 21, 2013 a Department
More informationGeneralization to inequality constrained problem. Maximize
Lecture 11. 26 September 2006 Review of Lecture #10: Second order optimality conditions necessary condition, sufficient condition. If the necessary condition is violated the point cannot be a local minimum
More informationPacific Journal of Optimization (Vol. 2, No. 3, September 2006) ABSTRACT
Pacific Journal of Optimization Vol., No. 3, September 006) PRIMAL ERROR BOUNDS BASED ON THE AUGMENTED LAGRANGIAN AND LAGRANGIAN RELAXATION ALGORITHMS A. F. Izmailov and M. V. Solodov ABSTRACT For a given
More informationLecture: Algorithms for LP, SOCP and SDP
1/53 Lecture: Algorithms for LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More informationLECTURE 10 LECTURE OUTLINE
LECTURE 10 LECTURE OUTLINE Min Common/Max Crossing Th. III Nonlinear Farkas Lemma/Linear Constraints Linear Programming Duality Convex Programming Duality Optimality Conditions Reading: Sections 4.5, 5.1,5.2,
More informationConvergence Analysis of Inexact Infeasible Interior Point Method. for Linear Optimization
Convergence Analysis of Inexact Infeasible Interior Point Method for Linear Optimization Ghussoun Al-Jeiroudi Jacek Gondzio School of Mathematics The University of Edinburgh Mayfield Road, Edinburgh EH9
More informationSymmetric Matrices and Eigendecomposition
Symmetric Matrices and Eigendecomposition Robert M. Freund January, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 Symmetric Matrices and Convexity of Quadratic Functions
More informationIntroduction to Mathematical Programming IE406. Lecture 10. Dr. Ted Ralphs
Introduction to Mathematical Programming IE406 Lecture 10 Dr. Ted Ralphs IE406 Lecture 10 1 Reading for This Lecture Bertsimas 4.1-4.3 IE406 Lecture 10 2 Duality Theory: Motivation Consider the following
More informationLMI Methods in Optimal and Robust Control
LMI Methods in Optimal and Robust Control Matthew M. Peet Arizona State University Lecture 02: Optimization (Convex and Otherwise) What is Optimization? An Optimization Problem has 3 parts. x F f(x) :
More informationOptimization Problems with Constraints - introduction to theory, numerical Methods and applications
Optimization Problems with Constraints - introduction to theory, numerical Methods and applications Dr. Abebe Geletu Ilmenau University of Technology Department of Simulation and Optimal Processes (SOP)
More informationBASICS OF CONVEX ANALYSIS
BASICS OF CONVEX ANALYSIS MARKUS GRASMAIR 1. Main Definitions We start with providing the central definitions of convex functions and convex sets. Definition 1. A function f : R n R + } is called convex,
More informationWe describe the generalization of Hazan s algorithm for symmetric programming
ON HAZAN S ALGORITHM FOR SYMMETRIC PROGRAMMING PROBLEMS L. FAYBUSOVICH Abstract. problems We describe the generalization of Hazan s algorithm for symmetric programming Key words. Symmetric programming,
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationOptimality Conditions for Constrained Optimization
72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationInterior-Point Methods
Interior-Point Methods Stephen Wright University of Wisconsin-Madison Simons, Berkeley, August, 2017 Wright (UW-Madison) Interior-Point Methods August 2017 1 / 48 Outline Introduction: Problems and Fundamentals
More informationLecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016
Lecture 1: Entropy, convexity, and matrix scaling CSE 599S: Entropy optimality, Winter 2016 Instructor: James R. Lee Last updated: January 24, 2016 1 Entropy Since this course is about entropy maximization,
More informationarxiv: v1 [math.oc] 1 Jul 2016
Convergence Rate of Frank-Wolfe for Non-Convex Objectives Simon Lacoste-Julien INRIA - SIERRA team ENS, Paris June 8, 016 Abstract arxiv:1607.00345v1 [math.oc] 1 Jul 016 We give a simple proof that the
More informationChapter 2 Convex Analysis
Chapter 2 Convex Analysis The theory of nonsmooth analysis is based on convex analysis. Thus, we start this chapter by giving basic concepts and results of convexity (for further readings see also [202,
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationYou should be able to...
Lecture Outline Gradient Projection Algorithm Constant Step Length, Varying Step Length, Diminishing Step Length Complexity Issues Gradient Projection With Exploration Projection Solving QPs: active set
More informationComputational Optimization. Mathematical Programming Fundamentals 1/25 (revised)
Computational Optimization Mathematical Programming Fundamentals 1/5 (revised) If you don t know where you are going, you probably won t get there. -from some book I read in eight grade If you do get there,
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationConvex Optimization and Modeling
Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual
More informationA Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions
A Unified Analysis of Nonconvex Optimization Duality and Penalty Methods with General Augmenting Functions Angelia Nedić and Asuman Ozdaglar April 16, 2006 Abstract In this paper, we study a unifying framework
More informationLecture 3 Interior Point Methods and Nonlinear Optimization
Lecture 3 Interior Point Methods and Nonlinear Optimization Robert J. Vanderbei April 16, 2012 Machine Learning Summer School La Palma http://www.princeton.edu/ rvdb Example: Basis Pursuit Denoising L
More informationMarch 5, 2012 MATH 408 FINAL EXAM SAMPLE
March 5, 202 MATH 408 FINAL EXAM SAMPLE Partial Solutions to Sample Questions (in progress) See the sample questions for the midterm exam, but also consider the following questions. Obviously, a final
More informationUsing interior point methods for optimization in training very large scale Support Vector Machines. Interior Point Methods for QP. Elements of the IPM
School of Mathematics T H E U N I V E R S I T Y O H Using interior point methods for optimization in training very large scale Support Vector Machines F E D I N U R Part : Interior Point Methods for QP
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee227c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee227c@berkeley.edu
More informationQuiz Discussion. IE417: Nonlinear Programming: Lecture 12. Motivation. Why do we care? Jeff Linderoth. 16th March 2006
Quiz Discussion IE417: Nonlinear Programming: Lecture 12 Jeff Linderoth Department of Industrial and Systems Engineering Lehigh University 16th March 2006 Motivation Why do we care? We are interested in
More informationConvex Optimization. Newton s method. ENSAE: Optimisation 1/44
Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)
More informationEnhanced Fritz John Optimality Conditions and Sensitivity Analysis
Enhanced Fritz John Optimality Conditions and Sensitivity Analysis Dimitri P. Bertsekas Laboratory for Information and Decision Systems Massachusetts Institute of Technology March 2016 1 / 27 Constrained
More informationMaximum likelihood estimation
Maximum likelihood estimation Guillaume Obozinski Ecole des Ponts - ParisTech Master MVA Maximum likelihood estimation 1/26 Outline 1 Statistical concepts 2 A short review of convex analysis and optimization
More informationCS-E4830 Kernel Methods in Machine Learning
CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationLagrange Relaxation and Duality
Lagrange Relaxation and Duality As we have already known, constrained optimization problems are harder to solve than unconstrained problems. By relaxation we can solve a more difficult problem by a simpler
More informationLagrangian Duality Theory
Lagrangian Duality Theory Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Chapter 14.1-4 1 Recall Primal and Dual
More informationIE 521 Convex Optimization Homework #1 Solution
IE 521 Convex Optimization Homework #1 Solution your NAME here your NetID here February 13, 2019 Instructions. Homework is due Wednesday, February 6, at 1:00pm; no late homework accepted. Please use the
More informationAlgorithms for Constrained Optimization
1 / 42 Algorithms for Constrained Optimization ME598/494 Lecture Max Yi Ren Department of Mechanical Engineering, Arizona State University April 19, 2015 2 / 42 Outline 1. Convergence 2. Sequential quadratic
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationMathematical Economics. Lecture Notes (in extracts)
Prof. Dr. Frank Werner Faculty of Mathematics Institute of Mathematical Optimization (IMO) http://math.uni-magdeburg.de/ werner/math-ec-new.html Mathematical Economics Lecture Notes (in extracts) Winter
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More information