Lecture Note 5: Semidefinite Programming for Stability Analysis


 Gervais George
 2 years ago
 Views:
Transcription
1 ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State University, Columbu, Ohio, USA Spring 2017 Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 1 / 39
2 Outline Motivation Some Linear Algebra A Tiny Bit of Optimization and Duality Linear Matrix Inequalities Semidefinite Programming Problems Some Examples Conclusion Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 2 / 39
3 Motivation Converse Lyapunov function theorems are not constructive Basic idea for Lyapunov function synthesis  Select Lyapunov function structure (e.g. quadratic, polynomial, piecewise quadratic,...)  Parameterize Lyapunov function candidates  Find values of parameters to satisfy Lyapunov conditions Many Lyapunov synthesis problems can be formulated as Semidefinite Programming (SDP) problems. SDP has broad applications in many other fields as well Motivation Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 3 / 39
4 Real Symmetric Matrices S n : set of real symmetric matrices All eigenvalues are real There exists a full set of orthogonal eigenvectors Spectral decomposition: If A S n, then A = QΛQ T, where Λ diagonal and Q is unitary. Some Linear Algebra Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 4 / 39
5 Positive Semidefinite Matrices I A S n is called positive semidefinite (p.s.d.), denoted by A 0, if x T Ax 0, x R n A S n is called positive definite (p.d.), denoted by A 0, if x T Ax > 0 for all nonzero x R n S n +: set of all p.s.d. (symmetric) matrices S n ++: set of all p.d. (symmetric) matrices p.s.d. [ or p.d. matrices ] can also be defined for nonsymmetric matrices. 1 1 e.g.: 1 1 We assume p.s.d. and p.d. are symmetric (unless otherwise noted) Notation: A B (resp. A B ) means A B S n + (resp. A B S n ++) Some Linear Algebra Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 5 / 39
6 Positive Semidefinite Matrices II Other equivalent definitions for symmetric p.s.d. matrices:  All 2 n 1 principal minors of A are nonnegative  All eigs of A are nonnegative  There exists a factorization A = B T B Other equivalent definitions for p.d. matrices:  All n leading principal minors of A are positive  All eigs of A are strictly positive  There exists a factorization A = B T B with B square and nonsingular. Some Linear Algebra Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 6 / 39
7 Positive Semidefinite Matrices III Useful facts:  If T nonsingular, A 0 T T AT 0; and A 0 T T AT 0  Inner product on R m n : < A, B > tr(a T B) A B. Some Linear Algebra Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 7 / 39
8 Positive Semidefinite Matrices IV  For A, B S n +, tr(ab) 0  For any symmetric A S n, λ min(a) µ A µi and λ max(a) β A βi Some Linear Algebra Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 8 / 39
9 Convex Set Recall from Lecture 3. Convex Set: A set S is convex if x 1, x 2 S αx 1 + (1 α)x 2 S, α [0, 1] Convex combination of x 1,..., x k : { α 1 x 1 + α 2 x α k x k : α i 0, and i α i = 1 } Convex hull: co {S} set of all convex combinations of points in S Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 9 / 39
10 Convex Cone A set S is called a cone if λ > 0, x S λx S. Conic combination of x 1 and x 2 : x = α 1 x 1 + α 2 x 2 with α 1, α 2 0 Convex cone: 1. a cone that is convex 2. equivalently, a set that contains all the conic combinations of points in the set Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 10 / 39
11 Positive Semidefinite Cone The set of positive semidefinite matrices (i.e. S n +) is a convex cone and is referred to as the positive semidefinite (PSD) cone Recall that if A, B S n +, then tr(ab) 0. This indicates that the cone S n + is acute. Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 11 / 39
12 Operations that preserve convexity I Intersection of possibly infinite number of convex sets:  e.g.: polyhedron:  e.g.: PSD cone: Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 12 / 39
13 Operations that preserve convexity II Affine mapping f : R n R m (i.e. f(x) = Ax + b)  f(x) = {f(x) : x X} is convex whenever X R n is convex e.g.: Ellipsoid: E 1 = {x R n : (x x c) T P (x x c) 1} or equivalently E 2 = {x c + Au : u 2 1}  f 1 (Y ) = {x R n : f(x) Y } is convex whenever Y R m is convex e.g.: {Ax b} = f 1 (R n +), where R n + is nonnegative orthant Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 13 / 39
14 Convex Function I Consider a finite dimensional vector space X. Let D X be convex. Definition 1 (Convex Function). A function f : D R is called convex if f(αx 1 + (1 α)x 2 ) αf(x 1 ) + (1 α)f(x 2 ), x 1, x 2 D, α [0, 1] f : D R is called strictly convex if f(αx 1 + (1 α)x 2 ) < αf(x 1 ) + (1 α)f(x 2 ), x 1 x 2 D, α [0, 1] f : D R is called concave if f is convex Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 14 / 39
15 Convex Function II How to check a function is convex? Directly use definition Firstorder condition: If f is differentiable over an open set that contains D, then f is convex over D iff f(z) f(x) + f(x) T (z x), x, z D Secondorder condition: Suppose f is twicely differentiable over an open set that contains D, then f is convex over D iff 2 f(x) 0, x D Many other conditions, tricks,... see [BV04]. Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 15 / 39
16 Convex Function III Examples of convex functions: In general, affine functions are both convex and concave  e.g.: f(x) = a T x + b, for x R n  e.g.: f(x) = tr(a T X) + c = m n i=1 j=1 AijXij + c, for X Rm n All norms are convex  e.g. in R n : f(x) = x p = ( n i=1 xi p) 1/p ; f(x) = x = max k x k  e.g. in R m n : f(x) = X 2 = σ max(x) Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 16 / 39
17 Convex Function IV Lemma 1 (Jensen s Inequality). If f : D R is a convex function, then for any possible choices of x 1,..., x m D and α 1,..., α m 0 with m l=1 α l = 1 we have f(α 1x α mx m) α 1f(x 1) + + α mf(x m) α k s can be viewed as the probability that x k occurs Probabilistic version of Jensen s inequality reads: f(e(x)) E(f(X)) for any random variable X taking value in D Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 17 / 39
18 Nonlinear Optimization and Duality I Nonlinear Optimization: minimize: subject to: f 0 (x) f i (x) 0, i = 1,... m h i (x) = 0, i = 1,..., q decision variable x R n, domain D, referred to as primal problem optimal value p is called a convex optimization problem if f 0,..., f m are convex and h 1,..., h q are affine typically convex optimization can be solved efficiently Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 18 / 39
19 Nonlinear Optimization and Duality II Associated Lagrangian: L : D R m R q R m q L(x, λ, ν) = f 0 (x) + λ i f i (x) + ν i h i (x) i=1 i=1 weighted sum of objective and constraints functions λ i : Lagrangian multiplier associated with f i (x) 0 ν i : Lagrangian multiplier associated with h i (x) = 0 Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 19 / 39
20 Nonlinear Optimization and Duality III Lagrange dual function: g : R m R q : R g(λ, ν) = inf L(x, λ, ν) x D { m = inf f 0 (x) + λ i f i (x) + x D i=1 g is concave, can be for some λ, ν } q ν i h i (x) Lower bound property: If λ 0 (elementwise), then g(λ, ν) p i=1 Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 20 / 39
21 Nonlinear Optimization and Duality IV Lagrange Dual Problem: { maximize : g(λ, ν) subject to: λ 0 Find the best lower bound on p using the Lagrange dual function a convex optimization problem even when the primal is nonconvex optimal value denoted d (λ, ν) is called dual feasible if λ 0 and (λ, ν) dom(g) Often simplifed by making the implicit constraint (λ, ν) dom(g) explicit Example: Linear Program and its dual minimize: c T x subject to: Ax = b x 0 { maximize: b T ν subject to: A T ν + c 0 Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 21 / 39
22 Nonlinear Optimization and Duality V Duality Theorems Weak Duality: d p  always hold (for convex and nonconvex problems)  can be used to find nontrivial lower bounds for difficult problems Strong Duality: d = p  not true in general, but typically holds for convex problems  conditions that guarantee strong duality in convex problems are called constraint qualifications  Slater s constraint qualification: Primal is strictly feasible Basic Optimization Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 22 / 39
23 Linear Matrix Inequalities I Standard form: Given symmetric matrices F 0,..., F m S n, F (x) = F 0 + x 1 F x m F m 0 is called a Linear Matrix Inequality in x = (x 1,..., x m ) T R m The function F (x) is affine in x The constraint set {x R n : F (x) 0} is nonlinear but convex Linear Matrix Inequalities Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 23 / 39
24 Linear Matrix Inequalities II Example 1 (LMI in Standard Form). Characterize the constraint set: F (x) = [ ] x1 + x 2 x x x 3 Linear Matrix Inequalities Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 24 / 39
25 Linear Matrix Inequalities III General Linear Matrix Inequalities (LMI)  Let X be a finitedimensional real vector space.  F : X S n is an affine mapping from X to n n symmetric matrices  Then F (X) 0 is called also an LMI in variable X X  Translation to standard form: Choose a basis X 1,..., X m of X and represent X = x 1X x mx m for any X X. For a given affine mapping F : X S n, we can define ˆF : R m S n as ˆF (x) F (X) = F (0) + m x i[f (X i) F (0)] i=1 where x is the coordinate of X w.r.t. the basis X 1,..., X m. Linear Matrix Inequalities Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 25 / 39
26 Linear Matrix Inequalities IV Example 2. Find conditions on matrix P to ensure that V (x) = x T P x is a Lyapunov function for a linear system ẋ = Ax Linear Matrix Inequalities Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 26 / 39
27 Schur Complement Lemma I Lemma 2 (Schur Complement Lemma). [ ] A B Define M = B T. The following three sets of inequalities are equivalent. C M 0 { A 0 C B T A 1 B 0 { C 0 A BC 1 B T 0 Proof: The lemma follows immediately from the following identities: [ ] [ ] [ ] [ ] I 0 A B I A 1 B A 0 B T A 1 I B T = C 0 I 0 C B T A 1 B [ I BC 1 ] [ A B ] [ 0 I B T C I 0 C 1 B T I ] = [ A BC 1 B T 0 0 C ] Linear Matrix Inequalities Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 27 / 39
28 Schur Complement Lemma II The proof of Schur complement lemma also reveals more general relations between the numbers of negative, zero, positive eigenvalues of  M vs. A and C B T A 1 B  M vs. C and A BC 1 B T Schur complement lemma is a very useful result to transform nonlinear (quadratic or bilinear) matrix inequalities to linear ones. Linear Matrix Inequalities Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 28 / 39
29 Semidefinite Programming I Semidefinite Programming (SDP) Problem: Optimization problem with linear objective, and Linear Matrix Inequality and linear equality constraints: minimize: c T x subject to: F 0 + x 1 F x m F m 0 (1) Ax = b Linear equality constraint in (1) can be eliminated. So essentially SDP can be viewed as optimizing linear function subject to only LMI constraints. SDP is a particular class of convex optimization problem. Global optimal solution can be found efficiently. Optimizing nonlinear but convex cost function subject to LMI constraints is also a convex optimization that can often be solved efficiently. Semidefinite Programming Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 29 / 39
30 Semidefinite Programming II Standard forms of SDP in matrix variable: SDP Standard Prime Form: min subject to: X S n : f p(x) = C X A i X = b i, i = 1,..., m X 0 (2) SDP Dual form: max : y R m subject to: f d(y) = b T y m i=1 y ia i C (3) One can derive the dual from the prime using either standard Lagrange duality method or more specialized Fenchel duality results The dual form (3) is equivalent to (1) (after eliminating the equality constraint Ax = b in (1)) Semidefinite Programming Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 30 / 39
31 Semidefinite Programming III SDP Weak Duality: f p (X) f d (y) for any primal and dual feasible X and y SDP Strong Duality: f p (X ) = f d (y ) holds under Slater s condition: Many control and optimization problem can be formulated or translated into SDP problems Various computationally difficult optimization problems can be effectively approximated by SDP problems (SDP relaxation...) We will see some examples after introducing an important technique: Sprocedure Semidefinite Programming Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 31 / 39
32 SProcedure I Many stability/engineering problems require to certify that a given function is signdefinite over certain subset of the space Mathematically, this condition can be stated as follows: g 0 (x) 0 on {x R n g 1 (x) 0,..., g m (x) 0} (4) Given functions g 0,..., g m, we want to know whether the condition holds. Sometimes we may also want to find a g 0 satisfying this condition for given g 1,..., g m. Conservative but useful condition: PSD functions s i (x) s.t. g 0 (x) i s i (x)g i (x) 0, x R n This is the socalled Generalized SProcedure Semidefinite Programming Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 32 / 39
33 SProcedure II Now consider an important special case: g i (x) = x T G i x, i = 0, 1,... are quadratic functions Requirement (4) becomes: x R n, x T G 1 x 0,..., x T G k x 0 x T G 0 x 0 Sufficient condition (Sprocedure): α 1,..., α m 0 with G 0 α 1 G α m G m SProcedure is lossless if m = 1 and ˆx s.t. ˆx T G 1ˆx > 0 (constraint qualification) Semidefinite Programming Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 33 / 39
34 Some Examples I Example 3 (Eigenvalue Optimization). Given symmetric matrices A 0, A 1,..., A m. Let S(w) = A 0 + i w ia i. Find weights {w i } m i=1 to minimize λ max(s(w)) SDP Examples Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 34 / 39
35 Some Examples II Example 4 (Ellipsoid inequality). Given R S n ++, the set E = {x R n : (x x c ) T R(x x c ) < 1} is an ellipsoid with center x c. Find the point in E that is the closet to the origin. SDP Examples Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 35 / 39
36 Some Examples III Example 5 (Linear Feedback Control Gain Design). Given a linear control system ẋ = Ax + Bu with linear state feedback u = Kx. Find K to stabilize the system SDP Examples Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 36 / 39
37 Some Examples IV Example 6 (Robust Stabiltiy). Given system ẋ = Ax + u with uncertain feedback u = g(x). Suppose all we know is that the feedback law satisfies: g(x) 2 β x 2. Find Lyapunov function V (x) = x T P x to ensure exponential stability. SDP Examples Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 37 / 39
38 Concluding Remarks Basic knowledge on optimization and Lagrange duality Linear matrix inequalities impose convex constraints Semidefinite programming problem: optimize linear cost subject to LMI constraints SDP has broad applications in various engineering fields: signal processing, networking, communication, control, machine learning, big data... Further reading [BV04; Boy+94; VB00] Next Lecture: Stability analysis for hybrid systems Concluding Remarks Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 38 / 39
39 References [Boy+94] [VB00] [BV04] Stephen Boyd et al. Linear matrix inequalities in system and control theory. SIAM, Jeremy G VanAntwerp and Richard D Braatz. A tutorial on linear and bilinear matrix inequalities. In: Journal of process control 10.4 (2000). Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, Concluding Remarks Lecture 5 (ECE7850 Sp17) Wei Zhang(OSU) 39 / 39
I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec  Spring 2010
I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec  Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0
More informationIntroduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research
Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,
More informationLagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470  Convex Optimization Fall 201718, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual
More informationLecture 6: Conic Optimization September 8
IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions
More information14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.
CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity
More informationEE 227A: Convex Optimization and Applications October 14, 2008
EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider
More informationLecture Note 1: Background
ECE5463: Introduction to Robotics Lecture Note 1: Background Prof. Wei Zhang Department of Electrical and Computer Engineering Ohio State University Columbus, Ohio, USA Spring 2018 Lecture 1 (ECE5463 Sp18)
More informationSemidefinite Programming Basics and Applications
Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent
More informationTutorial on Convex Optimization for Engineers Part II
Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D98684 Ilmenau, Germany jens.steinwandt@tuilmenau.de
More information4. Algebra and Duality
41 Algebra and Duality P. Parrilo and S. Lall, CDC 2003 2003.12.07.01 4. Algebra and Duality Example: nonconvex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone
More informationConvex Optimization M2
Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More information5. Duality. Lagrangian
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationInterior Point Methods: SecondOrder Cone Programming and Semidefinite Programming
School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: SecondOrder Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very wellwritten and a pleasure to read. The
More information61 The Positivstellensatz P. Parrilo and S. Lall, ECC
61 The Positivstellensatz P. Parrilo and S. Lall, ECC 2003 2003.09.02.10 6. The Positivstellensatz Basic semialgebraic sets Semialgebraic sets TarskiSeidenberg and quantifier elimination Feasibility
More informationModule 04 Optimization Problems KKT Conditions & Solvers
Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to CyberPhysical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September
More informationLecture: Duality.
Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt2016fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong
More informationA Brief Review on Convex Optimization
A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationSEMIDEFINITE PROGRAM BASICS. Contents
SEMIDEFINITE PROGRAM BASICS BRIAN AXELROD Abstract. A introduction to the basics of Semidefinite programs. Contents 1. Definitions and Preliminaries 1 1.1. Linear Algebra 1 1.2. Convex Analysis (on R n
More informationELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications
ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March
More informationSemidefinite Programming Duality and Linear Timeinvariant Systems
Semidefinite Programming Duality and Linear Timeinvariant Systems Venkataramanan (Ragu) Balakrishnan School of ECE, Purdue University 2 July 2004 Workshop on Linear Matrix Inequalities in Control LAASCNRS,
More informationConvex Optimization Boyd & Vandenberghe. 5. Duality
5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized
More informationExample: feasibility. Interpretation as formal proof. Example: linear inequalities and Farkas lemma
41 Algebra and Duality P. Parrilo and S. Lall 2006.06.07.01 4. Algebra and Duality Example: nonconvex polynomial optimization Weak duality and duality gap The dual is not intrinsic The cone of valid
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms  A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationCSCI : Optimization and Control of Networks. Review on Convex Optimization
CSCI7000016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 28. Suvrit Sra. (Algebra + Optimization) 02 May, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 28 (Algebra + Optimization) 02 May, 2013 Suvrit Sra Admin Poster presentation on 10th May mandatory HW, Midterm, Quiz to be reweighted Project final report
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions HyangWon Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More informationDuality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities
Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form
More informationLinear and nonlinear programming
Linear and nonlinear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationLecture: Convex Optimization Problems
1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt2015fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization
More informationSemidefinite Programming
Semidefinite Programming Notes by Bernd Sturmfels for the lecture on June 26, 208, in the IMPRS Ringvorlesung Introduction to Nonlinear Algebra The transition from linear algebra to nonlinear algebra has
More information4. Convex optimization problems
Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization
More informationLecture: Examples of LP, SOCP and SDP
1/34 Lecture: Examples of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:
More informationFrom Convex Optimization to Linear Matrix Inequalities
Dep. of Information Engineering University of Pisa (Italy) From Convex Optimization to Linear Matrix Inequalities eng. Sergio Grammatico grammatico.sergio@gmail.com Class of Identification of Uncertain
More informationLMI MODELLING 4. CONVEX LMI MODELLING. Didier HENRION. LAASCNRS Toulouse, FR Czech Tech Univ Prague, CZ. Universidad de Valladolid, SP March 2009
LMI MODELLING 4. CONVEX LMI MODELLING Didier HENRION LAASCNRS Toulouse, FR Czech Tech Univ Prague, CZ Universidad de Valladolid, SP March 2009 Minors A minor of a matrix F is the determinant of a submatrix
More informationN. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:
0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything
More informationReal Symmetric Matrices and Semidefinite Programming
Real Symmetric Matrices and Semidefinite Programming Tatsiana Maskalevich Abstract Symmetric real matrices attain an important property stating that all their eigenvalues are real. This gives rise to many
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a realvalued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17
EE/ACM 150  Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko
More informationICSE4030 Kernel Methods in Machine Learning
ICSE4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This
More informationRelaxations and Randomized Methods for Nonconvex QCQPs
Relaxations and Randomized Methods for Nonconvex QCQPs Alexandre d Aspremont, Stephen Boyd EE392o, Stanford University Autumn, 2003 Introduction While some special classes of nonconvex problems can be
More informationRobust and Optimal Control, Spring 2015
Robust and Optimal Control, Spring 2015 Instructor: Prof. Masayuki Fujita (S5303B) G. Sum of Squares (SOS) G.1 SOS Program: SOS/PSD and SDP G.2 Duality, valid ineqalities and Cone G.3 Feasibility/Optimization
More informationE5295/5B5749 Convex optimization with engineering applications. Lecture 5. Convex programming and semidefinite programming
E5295/5B5749 Convex optimization with engineering applications Lecture 5 Convex programming and semidefinite programming A. Forsgren, KTH 1 Lecture 5 Convex optimization 2006/2007 Convex quadratic program
More informationThe Lagrangian L : R d R m R r R is an (easier to optimize) lower bound on the original problem:
HT05: SC4 Statistical Data Mining and Machine Learning Dino Sejdinovic Department of Statistics Oxford Convex Optimization and slides based on Arthur Gretton s Advanced Topics in Machine Learning course
More informationConvex Optimization & Lagrange Duality
Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT
More informationLagrange duality. The Lagrangian. We consider an optimization program of the form
Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization
More informationMATH 829: Introduction to Data Mining and Analysis Support vector machines and kernels
1/12 MATH 829: Introduction to Data Mining and Analysis Support vector machines and kernels Dominique Guillot Departments of Mathematical Sciences University of Delaware March 14, 2016 Separating sets:
More informationMIT Algebraic techniques and semidefinite optimization February 14, Lecture 3
MI 6.97 Algebraic techniques and semidefinite optimization February 4, 6 Lecture 3 Lecturer: Pablo A. Parrilo Scribe: Pablo A. Parrilo In this lecture, we will discuss one of the most important applications
More informationSupport Vector Machines
Support Vector Machines Support vector machines (SVMs) are one of the central concepts in all of machine learning. They are simply a combination of two ideas: linear classification via maximum (or optimal
More informationIntroduction and Math Preliminaries
Introduction and Math Preliminaries Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Appendices A, B, and C, Chapter
More informationSparse Optimization Lecture: Dual Certificate in l 1 Minimization
Sparse Optimization Lecture: Dual Certificate in l 1 Minimization Instructor: Wotao Yin July 2013 Note scriber: Zheng Sun Those who complete this lecture will know what is a dual certificate for l 1 minimization
More informationCSC Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming
CSC2411  Linear Programming and Combinatorial Optimization Lecture 10: Semidefinite Programming Notes taken by Mike Jamieson March 28, 2005 Summary: In this lecture, we introduce semidefinite programming
More informationOptimality, Duality, Complementarity for Constrained Optimization
Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of WisconsinMadison May 2014 Wright (UWMadison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear
More informationLagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DSGA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
More informationCSE4830 Kernel Methods in Machine Learning
CSE4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This
More informationc 2000 Society for Industrial and Applied Mathematics
SIAM J. OPIM. Vol. 10, No. 3, pp. 750 778 c 2000 Society for Industrial and Applied Mathematics CONES OF MARICES AND SUCCESSIVE CONVEX RELAXAIONS OF NONCONVEX SES MASAKAZU KOJIMA AND LEVEN UNÇEL Abstract.
More informationGeometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as
Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.
More informationLecture 14: Optimality Conditions for Conic Problems
EE 227A: Conve Optimization and Applications March 6, 2012 Lecture 14: Optimality Conditions for Conic Problems Lecturer: Laurent El Ghaoui Reading assignment: 5.5 of BV. 14.1 Optimality for Conic Problems
More informationLecture 7: Convex Optimizations
Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 6. Suvrit Sra. (Conic optimization) 07 Feb, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 6 (Conic optimization) 07 Feb, 2013 Suvrit Sra Organizational Info Quiz coming up on 19th Feb. Project teams by 19th Feb Good if you can mix your research
More informationConvex optimization problems. Optimization problem in standard form
Convex optimization problems optimization problem in standard form convex optimization problems linear optimization quadratic optimization geometric programming quasiconvex optimization generalized inequality
More informationHomework Set #6  Solutions
EE 15  Applications of Convex Optimization in Signal Processing and Communications Dr Andre Tkacenko JPL Third Term 111 Homework Set #6  Solutions 1 a The feasible set is the interval [ 4] The unique
More informationSemidefinite and Second Order Cone Programming Seminar Fall 2012 Project: Robust Optimization and its Application of Robust Portfolio Optimization
Semidefinite and Second Order Cone Programming Seminar Fall 2012 Project: Robust Optimization and its Application of Robust Portfolio Optimization Instructor: Farid Alizadeh Author: Ai Kagawa 12/12/2012
More informationA Note on Nonconvex Minimax Theorem with Separable Homogeneous Polynomials
A Note on Nonconvex Minimax Theorem with Separable Homogeneous Polynomials G. Y. Li Communicated by Harold P. Benson Abstract The minimax theorem for a convexconcave bifunction is a fundamental theorem
More informationA ten page introduction to conic optimization
CHAPTER 1 A ten page introduction to conic optimization This background chapter gives an introduction to conic optimization. We do not give proofs, but focus on important (for this thesis) tools and concepts.
More informationCOURSE ON LMI PART I.2 GEOMETRY OF LMI SETS. Didier HENRION henrion
COURSE ON LMI PART I.2 GEOMETRY OF LMI SETS Didier HENRION www.laas.fr/ henrion October 2006 Geometry of LMI sets Given symmetric matrices F i we want to characterize the shape in R n of the LMI set F
More informationIntroduction to Semidefinite Programming I: Basic properties a
Introduction to Semidefinite Programming I: Basic properties and variations on the GoemansWilliamson approximation algorithm for maxcut MFO seminar on Semidefinite Programming May 30, 2010 Semidefinite
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework  MC/MC Constrained optimization
More informationLecture 1. 1 Conic programming. MA 796S: Convex Optimization and Interior Point Methods October 8, Consider the conic program. min.
MA 796S: Convex Optimization and Interior Point Methods October 8, 2007 Lecture 1 Lecturer: Kartik Sivaramakrishnan Scribe: Kartik Sivaramakrishnan 1 Conic programming Consider the conic program min s.t.
More informationA sensitivity result for quadratic semidefinite programs with an application to a sequential quadratic semidefinite programming algorithm
Volume 31, N. 1, pp. 205 218, 2012 Copyright 2012 SBMAC ISSN 01018205 / ISSN 18070302 (Online) www.scielo.br/cam A sensitivity result for quadratic semidefinite programs with an application to a sequential
More informationSummer School: Semidefinite Optimization
Summer School: Semidefinite Optimization Christine Bachoc Université Bordeaux I, IMB Research Training Group Experimental and Constructive Algebra Haus Karrenberg, Sept. 3  Sept. 7, 2012 Duality Theory
More informationsubject to (x 2)(x 4) u,
Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the
More informationChapter 2: Preliminaries and elements of convex analysis
Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt1415.shtml Academic year 201415
More informationDuality. Geoff Gordon & Ryan Tibshirani Optimization /
Duality Geoff Gordon & Ryan Tibshirani Optimization 10725 / 36725 1 Duality in linear programs Suppose we want to find lower bound on the optimal value in our convex problem, B min x C f(x) E.g., consider
More informationarzelier
COURSE ON LMI OPTIMIZATION WITH APPLICATIONS IN CONTROL PART II.1 LMIs IN SYSTEMS CONTROL STATESPACE METHODS STABILITY ANALYSIS Didier HENRION www.laas.fr/ henrion henrion@laas.fr Denis ARZELIER www.laas.fr/
More informationORF 523 Lecture 9 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Thursday, March 10, 2016
ORF 523 Lecture 9 Spring 2016, Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Thursday, March 10, 2016 When in doubt on the accuracy of these notes, please cross check with the instructor
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions logconcave and logconvex functions
More informationConvex Optimization in Communications and Signal Processing
Convex Optimization in Communications and Signal Processing Prof. Dr.Ing. Wolfgang Gerstacker 1 University of ErlangenNürnberg Institute for Digital Communications National Technical University of Ukraine,
More informationCharacterizing Robust Solution Sets of Convex Programs under Data Uncertainty
Characterizing Robust Solution Sets of Convex Programs under Data Uncertainty V. Jeyakumar, G. M. Lee and G. Li Communicated by Sándor Zoltán Németh Abstract This paper deals with convex optimization problems
More informationSemidefinite Programming
Semidefinite Programming Basics and SOS Fernando Mário de Oliveira Filho Campos do Jordão, 2 November 23 Available at: www.ime.usp.br/~fmario under talks Conic programming V is a real vector space h, i
More informationAppendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS
Appendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS Here we consider systems of linear constraints, consisting of equations or inequalities or both. A feasible solution
More informationTwo hours. To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER. xx xxxx 2017 xx:xx xx.
Two hours To be provided by Examinations Office: Mathematical Formula Tables. THE UNIVERSITY OF MANCHESTER CONVEX OPTIMIZATION  SOLUTIONS xx xxxx 27 xx:xx xx.xx Answer THREE of the FOUR questions. If
More informationTrust Region Problems with Linear Inequality Constraints: Exact SDP Relaxation, Global Optimality and Robust Optimization
Trust Region Problems with Linear Inequality Constraints: Exact SDP Relaxation, Global Optimality and Robust Optimization V. Jeyakumar and G. Y. Li Revised Version: September 11, 2013 Abstract The trustregion
More informationMotivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:
CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through
More informationConvex Optimization M2
Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial
More informationSolving large Semidefinite Programs  Part 1 and 2
Solving large Semidefinite Programs  Part 1 and 2 Franz Rendl http://www.math.uniklu.ac.at AlpenAdriaUniversität Klagenfurt Austria F. Rendl, Singapore workshop 2006 p.1/34 Overview Limits of Interior
More informationConvex Optimization. Lecture 12  Equality Constrained Optimization. Instructor: Yuanzhang Xiao. Fall University of Hawaii at Manoa
Convex Optimization Lecture 12  Equality Constrained Optimization Instructor: Yuanzhang Xiao University of Hawaii at Manoa Fall 2017 1 / 19 Today s Lecture 1 Basic Concepts 2 for Equality Constrained
More informationLecture Notes on Support Vector Machine
Lecture Notes on Support Vector Machine Feng Li fli@sdu.edu.cn Shandong University, China 1 Hyperplane and Margin In a ndimensional space, a hyper plane is defined by ω T x + b = 0 (1) where ω R n is
More informationIntroduction to Nonlinear Stochastic Programming
School of Mathematics T H E U N I V E R S I T Y O H F R G E D I N B U Introduction to Nonlinear Stochastic Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio SPS
More informationLargest dual ellipsoids inscribed in dual cones
Largest dual ellipsoids inscribed in dual cones M. J. Todd June 23, 2005 Abstract Suppose x and s lie in the interiors of a cone K and its dual K respectively. We seek dual ellipsoidal norms such that
More information4TE3/6TE3. Algorithms for. Continuous Optimization
4TE3/6TE3 Algorithms for Continuous Optimization (Duality in Nonlinear Optimization ) Tamás TERLAKY Computing and Software McMaster University Hamilton, January 2004 terlaky@mcmaster.ca Tel: 27780 Optimality
More informationFRTN10 Multivariable Control, Lecture 13. Course outline. The Qparametrization (Youla) Example: Springmass System
FRTN Multivariable Control, Lecture 3 Anders Robertsson Automatic Control LTH, Lund University Course outline The Qparametrization (Youla) LL5 Purpose, models and loopshaping by hand L6L8 Limitations
More informationCourse Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979)
Course Outline FRTN Multivariable Control, Lecture Automatic Control LTH, 6 LL Specifications, models and loopshaping by hand L6L8 Limitations on achievable performance L9L Controller optimization:
More informationDistributionally robust optimization techniques in batch bayesian optimisation
Distributionally robust optimization techniques in batch bayesian optimisation Nikitas Rontsis June 13, 2016 1 Introduction This report is concerned with performing batch bayesian optimization of an unknown
More informationUC Berkeley Department of Electrical Engineering and Computer Science. EECS 227A Nonlinear and Convex Optimization. Solutions 5 Fall 2009
UC Berkeley Department of Electrical Engineering and Computer Science EECS 227A Nonlinear and Convex Optimization Solutions 5 Fall 2009 Reading: Boyd and Vandenberghe, Chapter 5 Solution 5.1 Note that
More informationThe Qparametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Springmass System
The Qparametrization (Youla) Lecture 3: Synthesis by Convex Optimization controlled variables z Plant distubances w Example: Springmass system measurements y Controller control inputs u Idea for lecture
More informationAlgebra II. Paulius Drungilas and Jonas Jankauskas
Algebra II Paulius Drungilas and Jonas Jankauskas Contents 1. Quadratic forms 3 What is quadratic form? 3 Change of variables. 3 Equivalence of quadratic forms. 4 Canonical form. 4 Normal form. 7 Positive
More information