Convex Optimization. Lieven Vandenberghe Electrical Engineering Department, UCLA. Joint work with Stephen Boyd, Stanford University

Size: px
Start display at page:

Download "Convex Optimization. Lieven Vandenberghe Electrical Engineering Department, UCLA. Joint work with Stephen Boyd, Stanford University"

Transcription

1 Convex Optimization Lieven Vandenberghe Electrical Engineering Department, UCLA Joint work with Stephen Boyd, Stanford University Ph.D. School in Optimization in Computer Vision DTU, May 19, 2008

2 Introduction

3 Mathematical optimization minimize f 0 (x) subject to f i (x) 0, i = 1,...,m x = (x 1,...,x n ): optimization variables f 0 : R n R: objective function f i : R n R, i = 1,...,m: constraint functions 1

4 Solving optimization problems General optimization problem can be extremely difficult methods involve compromise: long computation time or local optimality Exceptions: certain problem classes can be solved efficiently and reliably linear least-squares problems linear programming problems convex optimization problems 2

5 Least-squares minimize Ax b 2 2 analytical solution: x = (A T A) 1 A T b reliable and efficient algorithms and software computation time proportional to n 2 p (for A R p n ); less if structured a widely used technology Using least-squares least-squares problems are easy to recognize standard techniques increase flexibility (weights, regularization,... ) 3

6 Linear programming minimize c T x subject to a T i x b i, i = 1,...,m no analytical formula for solution; extensive theory reliable and efficient algorithms and software computation time proportional to n 2 m if m n; less with structure a widely used technology Using linear programming not as easy to recognize as least-squares problems a few standard tricks used to convert problems into linear programs (e.g., problems involving l 1 - or l -norms, piecewise-linear functions) 4

7 Convex optimization problem minimize f 0 (x) subject to f i (x) 0, i = 1,...,m objective and constraint functions are convex: f i (θx + (1 θ)y) θf i (x) + (1 θ)f i (y) for all x, y, 0 θ 1 includes least-squares problems and linear programs as special cases 5

8 Solving convex optimization problems no analytical solution reliable and efficient algorithms computation time (roughly) proportional to max{n 3, n 2 m, F }, where F is cost of evaluating f i s and their first and second derivatives almost a technology Using convex optimization often difficult to recognize many tricks for transforming problems into convex form surprisingly many problems can be solved via convex optimization 6

9 History 1940s: linear programming minimize c T x subject to a T i x b i, i = 1,...,m 1950s: quadratic programming 1960s: geometric programming 1990s: semidefinite programming, second-order cone programming, quadratically constrained quadratic programming, robust optimization, sum-of-squares programming,... 7

10 New applications since 1990 linear matrix inequality techniques in control circuit design via geometric programming support vector machine learning via quadratic programming semidefinite pogramming relaxations in combinatorial optimization applications in structural optimization, statistics, signal processing, communications, image processing, quantum information theory, finance,... 8

11 Interior-point methods Linear programming 1984 (Karmarkar): first practical polynomial-time algorithm : efficient implementations for large-scale LPs Nonlinear convex optimization around 1990 (Nesterov & Nemirovski): polynomial-time interior-point methods for nonlinear convex programming since 1990: extensions and high-quality software packages 9

12 Traditional and new view of convex optimization Traditional: special case of nonlinear programming with interesting theory New: extension of LP, as tractable but substantially more general reflected in notation: cone programming minimize c T x subject to Ax b is inequality with respect to non-polyhedral convex cone 10

13 Outline Convex sets and functions Modeling systems Cone programming Robust optimization Semidefinite relaxations l 1 -norm sparsity heuristics Interior-point algorithms 11

14 Convex Sets and Functions

15 Convex sets Contains line segment between any two points in the set x 1,x 2 C, 0 θ 1 = θx 1 + (1 θ)x 2 C example: one convex, two nonconvex sets: 12

16 Examples and properties solution set of linear equations solution set of linear inequalities norm balls {x x R} and norm cones {(x, t) x t} set of positive semidefinite matrices image of a convex set under a linear transformation is convex inverse image of a convex set under a linear transformation is convex intersection of convex sets is convex 13

17 Convex functions domain dom f is a convex set and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all x,y domf, 0 θ 1 (x, f(x)) (y, f(y)) f is concave if f is convex 14

18 Examples expx, log x, xlog x are convex x α is convex for x > 0 and α 1 or α 0; x α is convex for α 1 quadratic-over-linear function x T x/t is convex in x, t for t > 0 geometric mean (x 1 x 2 x n ) 1/n is concave for x 0 log detx is concave on set of positive definite matrices log(e x 1 + e x n ) is convex linear and affine functions are convex and concave norms are convex 15

19 Pointwise maximum Operations that preserve convexity if f(x, y) is convex in x for fixed y, then is convex in x g(x) = sup f(x, y) y A Composition rules if h is convex and increasing and g is convex, then h(g(x)) is convex Perspective if f(x) is convex then tf(x/t) is convex in x, t for t > 0 16

20 Example m lamps illuminating n (small, flat) patches lamp power p j θ kj r kj illumination I k intensity I k at patch k depends linearly on lamp powers p j : I k = a T k p Problem: achieve desired illumination I k 1 with bounded lamp powers minimize max k=1,...,n log(a T k p) subject to 0 p j p max, j = 1,...,m 17

21 Convex formulation: problem is equivalent to minimize max k=1,...,n max{a T k p,1/at k p} subject to 0 p j p max, j = 1,...,m 5 max{u, 1/u} cost function is convex because maximum of convex functions is convex u 18

22 Quasiconvex functions domain dom f is convex and the sublevel sets are convex for all α S α = {x dom f f(x) α} β α a b c f is quasiconcave if f is quasiconvex 19

23 x is quasiconvex on R Examples ceil(x) = inf{z Z z x} is quasiconvex and quasiconcave log x is quasiconvex and quasiconcave on R ++ f(x 1, x 2 ) = x 1 x 2 is quasiconcave on R 2 ++ linear-fractional function f(x) = at x + b c T x + d, domf = {x ct x + d > 0} is quasiconvex and quasiconcave distance ratio f(x) = x a 2 x b 2, dom f = {x x a 2 x b 2 } is quasiconvex 20

24 Quasiconvex optimization Example minimize p(x)/q(x) subject to Ax b p convex, q concave, and p(x) 0, q(x) > 0 Equivalent formulation (variables x, t) minimize t subjec to p(x) tq(x) 0 Ax b for fixed t, constraint is a convex feasibility problem can determine optimal t via bisection 21

25 Modeling Systems

26 Convex optimization modeling systems allow simple specification of convex problems in natural form declare optimization variables form affine, convex, concave expressions specify objective and constraints automatically transform problem to canonical form, call solver, transform back built using object-oriented methods and/or compiler-compilers 22

27 Example minimize m w i log(b i a T i x) i=1 variable x R n ; parameters a i, b i, w i > 0 are given Specification in CVX (Grant, Boyd & Ye) cvx begin variable x(n) minimize ( -w * log(b-a*x) ) cvx end 23

28 Example minimize Ax b 2 + λ x 1 subject to Fx g + ( i=1 x i)h variable x R n ; parameters A, b, F, g, h given CVX specification cvx begin variable x(n) minimize ( norm(a*x-b,2) + lambda*norm(x,1) ) subject to F*x <= g + sum(x)*h cvx end 24

29 Illumination problem minimize max k=1,...,n max{a T k x, 1/aT k x} subject to 0 x 1 variable x R m ; parameters a k given (and nonnegative) CVX specification cvx begin variable x(m) minimize ( max( [ A*x; inv_pos(a*x) ] ) subject to x >= 0 x <= 1 cvx end 25

30 History general purpose optimization modeling systems AMPL, GAMS (1970s) systems for SDPs/LMIs (1990s): SDPSOL (Wu, Boyd), LMILAB (Gahinet, Nemirovski), LMITOOL (El Ghaoui) YALMIP (Löfberg 2000) automated convexity checking (Crusius PhD thesis 2002) disciplined convex programming (DCP) (Grant, Boyd, Ye 2004) CVX (Grant, Boyd, Ye 2005) CVXOPT (Dahl, Vandenberghe 2005) GGPLAB (Mutapcic, Koh, et al 2006) CVXMOD (Mattingley 2007) 26

31 Cone Programming

32 Linear programming minimize c T x subject to Ax b is elementwise inequality between vectors c Ax b x 27

33 Linear discrimination separate two sets of points {x 1,...,x N }, {y 1,...,y M } by a hyperplane a T x i + b > 0, i = 1,...,N a T y i + b < 0 i = 1,...,M homogeneous in a, b, hence equivalent to the linear inequalities (in a, b) a T x i + b 1, i = 1,...,N, a T y i + b 1, i = 1,...,M 28

34 Approximate linear separation of non-separable sets minimize N max{0,1 a T x i b} + i=1 M max{0, 1 + a T y i + b} i=1 can be interpreted as a heuristic for minimizing #misclassified points 29

35 Linear programming formulation minimize N max{0,1 a T x i b} + i=1 M max{0, 1 + a T y i + b} i=1 Equivalent LP N minimize i=1 u i + M i=1 v i minimize u i 1 a T x i b, v i 1 + a T y i + b, u 0, v 0 i = 1,...,N i = 1,...,M variables a, b, u R N, v R M 30

36 Cone programming minimize c T x subject to Ax K b y K z means z y K, where K is a proper convex cone extends linear programming (K = R m +) to nonpolyhedral cones (duality) theory and algorithms very similar to linear programming 31

37 Second-order cone programming 1 Second-order cone t 0.5 C m+1 = {(x, t) R m R x t} Second-order cone program x x minimize f T x subject to A i x + b i 2 c T i x + d i, Fx = g i = 1,...,m inequality constraints require (A i x + b i, c T i x + d i) C mi +1 32

38 Linear program with chance constraints minimize c T x subject to prob(a T i x b i) η, i = 1,...,m a i is Gaussian with mean ā i, covariance Σ i, and η 1/2 Equivalent SOCP minimize c T x subject to ā T i x + Φ 1 (η) Σ 1/2 i x 2 b i, i = 1,...,m Φ(x) is zero-mean unit-variance Gaussian CDF 33

39 Semidefinite programming 1 Positive semidefinite cone X S m + = {X S m X 0} X X 11 1 Semidefinite programming minimize c T x subject to x 1 A x n A n B constraint requires B x 1 A 1 x n A n S m + 34

40 Eigenvalue minimization minimize λ max (A(x)) where A(x) = A 0 + x 1 A x n A n (with given A i S k ) equivalent SDP minimize t subject to A(x) ti variables x R n, t R follows from λ max (A) t A ti 35

41 Matrix norm minimization minimize A(x) 2 = ( λ max (A(x) T A(x)) ) 1/2 where A(x) = A 0 + x 1 A x n A n (with given A i R p q ) equivalent SDP minimize subject to t [ ti A(x) T A(x) ti ] 0 variables x R n, t R constraint follows from A 2 t A T A t 2 I, t 0 [ ti A A T ti ] 0 36

42 Chebyshev inequalities Classical inequality: if X is a r.v. with E X = 0, E X 2 = σ 2, then prob( X 1) σ 2 Generalized inequality: sharp lower bounds on prob(x C) X R n is a random variable with known moments E X = a, E XX T = S C R n is defined by quadratic inequalities C = {x x T A i x + 2b T i x + c i < 0, i = 1,...,m} 37

43 Equivalent SDP maximize subject to 1 tr(sp) 2a T q r [ ] [ ] P q Ai b q T τ i r 1 i b T, i = 1,...,m i c i τ i 0, i = 1,...,m [ ] P q q T 0 r an SDP with variables P S n, q R n, scalars r, τ i optimal value is tight lower bound on prob(x C) solution provides distribution that achieves lower bound 38

44 Example C a a = E X; dashed line shows {x (x a) T (S aa T ) 1 (x a) = 1} lower bound on prob(x C) is achieved by distribution in red 39

45 Detection example x = s + v x R n : received signal s: transmitted signal s {s 1,s 2,...,s N } (one of N possible symbols) v: noise with E v = 0, E vv T = σ 2 I Detection problem: given observed value of x, estimate s 40

46 Example (N = 7): bound on probability of correct detection of s 1 is s 3 s 2 s 4 s 1 s 7 s 5 s 6 dots: distribution with probability of correct detection

47 Duality Cone program minimize c T x subject to Ax K b Dual cone program maximize b T z subject to A T z + c = 0 z K 0 K is the dual cone: K = {z z T x 0 for all x K} nonnegative orthant, 2nd order cone, PSD cone are self-dual: K = K Properties: optimal values are equal (if primal or dual is strictly feasible) 42

48 Robust Optimization

49 Robust optimization (worst-case) robust convex optimization problem minimize sup θ A f 0 (x, θ) subject to sup θ A f i (x, θ) 0, i = 1,...,m x is optimization variable; θ is an unknown parameter f i convex in x for fixed θ tractability depends on A (Ben-Tal, Nemirovski, El Ghaoui, Bertsimas,... ) 43

50 Robust linear programming minimize c T x subject to a T i x b i a i A i, i = 1,...,m coefficients unknown but contained in ellipsoids A i : A i = {ā i + P i u u 2 1} (ā i R n, P i R n n ) center is ā i, semi-axes determined by singular values/vectors of P i Equivalent SOCP minimize c T x subject to ā T i x + P i Tx 2 b i, i = 1,...,m 44

51 Robust least-squares minimize sup u 2 1 (A 0 + u 1 A u p A p )x b 2 coefficient matrix lies in ellipsoid; choose x to minimize worst-case residual norm Equivalent SDP minimize t 1 + t 2 I P(x) A 0 x b subject to P(x) T t 1 I 0 (A 0 x b) T 0 t 2 0 where P(x) = [ A 1 x A 2 x A p x ] 45

52 Example (p = 2, u uniformly distributed in unit disk) x rls frequency x tik 0.05 x ls r(u) = A(u)x b 2 x tik minimizes A 0 x b x

53 Semidefinite Relaxations

54 Relaxation and randomization convex optimization is increasingly used to find good bounds for hard (i.e., nonconvex) problems, via relaxation as a heuristic for finding suboptimal points, often via randomization 47

55 Boolean least-squares Semidefinite relaxations minimize Ax b 2 2 subject to x 2 i = 1, i = 1,...,n. a basic problem in digital communciations non-convex, very hard to solve exactly Equivalent formulation minimize tr(a T AZ) 2b T Az + b T b subject to Z ii = 1, i = 1,...,n Z = zz T follows from Az b 2 2 = tr(a T AZ) 2b T Az + b T b if Z = zz T 48

56 Semidefinite relaxation replace constraint Z = zz T with Z zz T an SDP with variables Z, z minimize tr(a T AZ) 2b T Az + b T b subject to Z ii = 1, i = 1,...,n [ ] Z z z T 0 1 optimal value is a lower bound for Boolean LS optimal value rounding Z, z gives suboptimal solution for Boolean LS Randomized rounding generate vector from N(z, Z zz T ) round components to ±1 49

57 Example (randomly chosen) parameters A R , b R 150 x R 100, so feasible set has points SDP bound rounded LS solution frequency Ax b 2 /(SDP bound) distribution of randomized solutions based on SDP solution 50

58 Sums of squares and semidefinite programming Sum of squares: a function of the form f(t) = s ( y T k q(t) ) 2 k=1 q(t): vector of basis functions (polynomial, trigonometric,... ) SDP parametrization: f(t) = q(t) T Xq(t), X 0 a sufficient condition for nonnegativity of f, useful in nonconvex polynomial optimization (Parrilo, Lasserre, Henrion, De Klerk... ) in some important special cases, necessary and sufficient 51

59 Example: Cosine polynomials f(ω) = x 0 + x 1 cosω + + x 2n cos 2nω 0 Sum of squares theorem: f(ω) 0 for α ω β if and only if f(ω) = g 1 (ω) 2 + s(ω)g 2 (ω) 2 g 1, g 2 : cosine polynomials of degree n and n 1 s(ω) = (cosω cosβ)(cos α cos ω) is a given weight function Equivalent SDP formulation: f(ω) 0 for α ω β if and only if x T p(ω) = q 1 (ω) T X 1 q 1 (ω) + s(ω)q 2 (ω) T X 2 q 2 (ω), X 1 0, X 2 0 p, q 1, q 2 : basis vectors (1, cosω, cos(2ω),...) up to order 2n, n, n 1 52

60 Example: Linear-phase Nyquist filter minimize sup ω ωs h 0 + h 1 cosω + + h n cos nω with h 0 = 1/M, h km = 0 for positive integer k 10 0 H(ω) ω (Example with n = 50, M = 5, ω s = 0.69) 53

61 SDP formulation minimize t subject to t H(ω) t, ω s ω π Equivalent SDP minimize t subject to t H(ω) = q 1 (ω) T X 1 q 1 (ω) + s(ω)q 2 (ω) T X 2 q 2 (ω) t + H(ω) = q 1 (ω) T X 3 q 1 (ω) + s(ω)q 2 (ω) T X 3 q 2 (ω) X 1 0, X 2 0, X 3 0, X 4 0 Variables t, h i (i km), 4 matrices X i of size roughly n 54

62 Multivariate trigonometric sums of squares h(ω) = n k= nx k e jkt ω = i g i (ω) 2, (x k = x k, ω R d ) g i is a polynomial in e jktω ; can have degree higher than n necessary for positivity of R restricting the degrees of g i gives a sufficient condition for nonnegativity Spectral mask constraints defined by trigonometric polynomials d i h(ω) = s 0 (ω) + i d i (ω)s i (ω), s i is s.o.s. guarantees h(ω) 0 on {ω d i (ω 0} (B. Dumitrescu) 55

63 Two-dimensional FIR filter design minimize δ s subject to 1 H(ω) δ p, ω D p H(ω) δ s, ω D s, where H(ω) = n i=0 n k=0 h ik cosiω 1 coskω 2 3 ω D s D p ω 1 H(ω) (db) ω2 2 2 ω

64 1-Norm Sparsity Heuristics

65 1-Norm heuristics use l 1 -norm x 1 as convex approximation of the l 0 - norm card(x) sparse regressor selection (Tibshirani, Hastie,... ) minimize Ax b 2 + ρ x 1 sparse signal representation (basis pursuit, sparse compression) (Donoho, Candes, Tao, Romberg,... ) minimize x 1 subject to Ax = b minimize x 1 subject to Ax b 2 ǫ 57

66 Norm approximation minimize Ax b 2 minimize Ax b 1 example (A is ): histograms of residuals 2-norm norm note large number of zero residuals in 1-norm solution 58

67 Robust regression f(t) t 42 points t i, y i (circles), including two outliers function f(t) = α + βt fitted using 2-norm (dashed) and 1-norm 59

68 Sparse reconstruction signal ˆx R n with n = 1000, 10 nonzero components m = 100 random noisy measurements b = Aˆx + v A ij N(0, 1) i.i.d. and v N(0, σ 2 I), σ =

69 l 2 -Norm reconstruction minimize Ax b x left: exact signal ˆx; right: l 2 reconstruction 61

70 l 1 -Norm reconstruction minimize Ax b 2 + x left: exact signal ˆx; right: l 1 reconstruction 62

71 Interior-Point Algorithms

72 Interior-point algorithms handle linear and nonlinear convex problems follow central path as guide to the solution (using Newton s method) worst-case complexity theory: # Newton iterations problem size in practice: # Newton steps between 10 and 50 performance is similar across wide range of problem dimensions, problem data, problem classes controlled by a small number of easily tuned algorithm parameters 63

73 Primal and dual cone program Cone program minimize c T x subject to Ax + s = b s K 0 maximize b T y subject to A T z + c = 0 z K 0 s K 0 means s K (convex cone) z K 0 means z K (dual cone K = {z s T z 0 s K}) Examples (of self-dual cones: K = K ) linear program: K is nonnegative orthant second order cone program: K is second order cone {(t,x) x 2 t} semidefinite program: K is cone of positive semidefinite matrices 64

74 Central path solution {(x(t), s(t)) t > 0} of minimize tc T x + φ(s) subject to Ax + s = b φ is a logarithmic barrier for primal cone K nonnegative orthant: φ(u) = k log u k second order cone: φ(u,v) = log(u 2 v T v) positive semidefinite cone: φ(v ) = log detv 65

75 Example: central path for linear program minimize c T x subject to Ax b c x x(t) 66

76 Central path optimality conditions Newton equation Ax + s = b, A T z + c = 0, z + 1 t φ(s) = 0 Newton equation: linearize optimality conditions [ 0 s ] + [ 0 A T A 0 ] [ x z ] = [ c A T z b Ax s ] z + 1 t 2 φ(s) s = z 1 t φ(s) gives search directions x, s, z many variations (e.g., primal-dual symmetric linearizations) 67

77 Computational effort per Newton step Newton step effort dominated by solving linear equations to find search direction equations inherit structure from underlying problem equations same as for weighted LS problem of similar size and structure Conclusion we can solve a convex problem with about the same effort as solving 30 least-squares problems 68

78 Direct methods for exploiting sparsity well developed, since late 1970s based on (heuristic) variable orderings, sparse factorizations standard in general purpose LP, QP, GP, SOCP implementations can solve problems with up to 10 5 variables, constraints (depending on sparsity pattern) 69

79 Some convex optimization solvers primal-dual, interior-point, exploit sparsity many for LP, QP (GLPK, CPLEX,... ) SeDuMi, SDPT3 (open source; Matlab; LP, SOCP, SDP) DSDP, CSDP, SDPA (open source; C; SDP) MOSEK (commercial; C with Matlab interface; LP, SOCP, GP,... ) solver.com (commercial; excel interface; LP, SOCP) GPCVX (open source; Matlab; GP) CVXOPT (open source; Python/C; LP, SOCP, SDP, GP,... )... and many others 70

80 Problem structure beyond sparsity state structure Toeplitz, circulant, Hankel; displacement rank fast transform (DFT, wavelet,... ) Kronecker, Lyapunov structure symmetry can exploit for efficiency, but not in most generic solvers 71

81 Example: 1-norm approximation minimize Ax b 1 Equivalent LP minimize k y k subject to y Ax b y Newton equation (D 1, D 2 positive diagonal) 0 0 A T A T 0 0 I I A I D 1 0 A I 0 D 2 x y z 1 z 2 = r 1 r 2 r 3 r 4 reduces to equation of the form A T DA x = r cost = cost of (weighted) least squares problem 72

82 Iterative methods conjugate-gradient (and variants like LSQR) exploit general structure rely on fast methods to evaluate Ax and A T y, where A is huge can terminate early, to get truncated-newton interior-point method can solve huge problems (10 7 variables, constraints), with good preconditioner proper tuning some luck 73

83 Solving specific problems in developing custom solver for specific application, we can exploit structure very efficiently determine ordering, memory allocation beforehand cut corners in algorithm, e.g., terminate early use warm start to get very fast solver opens up possibility of real-time embedded convex optimization 74

84 Conclusions

85 Convex optimization Fundamental theory recent advances include new problem classes, robust optimization, semidefinite relaxations of nonconvex problems, l 1 -norm heuristics... Applications Recent applications in wide range of areas; many more to be discovered Algorithms and software High-quality general-purpose implementations of interior-point methods Customized implementations can be orders of magnitude faster Good modeling systems With the right software, suitable for embedded applications 75

4. Convex optimization problems

4. Convex optimization problems Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

More information

Disciplined Convex Programming

Disciplined Convex Programming Disciplined Convex Programming Stephen Boyd Michael Grant Electrical Engineering Department, Stanford University University of Pennsylvania, 3/30/07 Outline convex optimization checking convexity via convex

More information

Lecture: Convex Optimization Problems

Lecture: Convex Optimization Problems 1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization

More information

4. Convex optimization problems

4. Convex optimization problems Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

More information

Convex optimization problems. Optimization problem in standard form

Convex optimization problems. Optimization problem in standard form Convex optimization problems optimization problem in standard form convex optimization problems linear optimization quadratic optimization geometric programming quasiconvex optimization generalized inequality

More information

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions 3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions

More information

Course Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979)

Course Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979) Course Outline FRTN Multivariable Control, Lecture Automatic Control LTH, 6 L-L Specifications, models and loop-shaping by hand L6-L8 Limitations on achievable performance L9-L Controller optimization:

More information

Advances in Convex Optimization: Theory, Algorithms, and Applications

Advances in Convex Optimization: Theory, Algorithms, and Applications Advances in Convex Optimization: Theory, Algorithms, and Applications Stephen Boyd Electrical Engineering Department Stanford University (joint work with Lieven Vandenberghe, UCLA) ISIT 02 ISIT 02 Lausanne

More information

Convex Optimization M2

Convex Optimization M2 Convex Optimization M2 Lecture 8 A. d Aspremont. Convex Optimization M2. 1/57 Applications A. d Aspremont. Convex Optimization M2. 2/57 Outline Geometrical problems Approximation problems Combinatorial

More information

Convex Optimization and l 1 -minimization

Convex Optimization and l 1 -minimization Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l

More information

ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications

ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

A Brief Review on Convex Optimization

A Brief Review on Convex Optimization A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

More information

15. Conic optimization

15. Conic optimization L. Vandenberghe EE236C (Spring 216) 15. Conic optimization conic linear program examples modeling duality 15-1 Generalized (conic) inequalities Conic inequality: a constraint x K where K is a convex cone

More information

Lecture: Examples of LP, SOCP and SDP

Lecture: Examples of LP, SOCP and SDP 1/34 Lecture: Examples of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2018.html wenzw@pku.edu.cn Acknowledgement:

More information

Lecture 2: Convex functions

Lecture 2: Convex functions Lecture 2: Convex functions f : R n R is convex if dom f is convex and for all x, y dom f, θ [0, 1] f is concave if f is convex f(θx + (1 θ)y) θf(x) + (1 θ)f(y) x x convex concave neither x examples (on

More information

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions 3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions

More information

6. Approximation and fitting

6. Approximation and fitting 6. Approximation and fitting Convex Optimization Boyd & Vandenberghe norm approximation least-norm problems regularized approximation robust approximation 6 Norm approximation minimize Ax b (A R m n with

More information

CSCI : Optimization and Control of Networks. Review on Convex Optimization

CSCI : Optimization and Control of Networks. Review on Convex Optimization CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

More information

FRTN10 Multivariable Control, Lecture 13. Course outline. The Q-parametrization (Youla) Example: Spring-mass System

FRTN10 Multivariable Control, Lecture 13. Course outline. The Q-parametrization (Youla) Example: Spring-mass System FRTN Multivariable Control, Lecture 3 Anders Robertsson Automatic Control LTH, Lund University Course outline The Q-parametrization (Youla) L-L5 Purpose, models and loop-shaping by hand L6-L8 Limitations

More information

The Q-parametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Spring-mass System

The Q-parametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Spring-mass System The Q-parametrization (Youla) Lecture 3: Synthesis by Convex Optimization controlled variables z Plant distubances w Example: Spring-mass system measurements y Controller control inputs u Idea for lecture

More information

12. Interior-point methods

12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

More information

Introduction to Convex Optimization

Introduction to Convex Optimization Introduction to Convex Optimization Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Optimization

More information

CS295: Convex Optimization. Xiaohui Xie Department of Computer Science University of California, Irvine

CS295: Convex Optimization. Xiaohui Xie Department of Computer Science University of California, Irvine CS295: Convex Optimization Xiaohui Xie Department of Computer Science University of California, Irvine Course information Prerequisites: multivariate calculus and linear algebra Textbook: Convex Optimization

More information

Problem structure in semidefinite programs arising in control and signal processing

Problem structure in semidefinite programs arising in control and signal processing Problem structure in semidefinite programs arising in control and signal processing Lieven Vandenberghe Electrical Engineering Department, UCLA Joint work with: Mehrdad Nouralishahi, Tae Roh Semidefinite

More information

Optimisation convexe: performance, complexité et applications.

Optimisation convexe: performance, complexité et applications. Optimisation convexe: performance, complexité et applications. Introduction, convexité, dualité. A. d Aspremont. M2 OJME: Optimisation convexe. 1/128 Today Convex optimization: introduction Course organization

More information

Nonlinear Programming Models

Nonlinear Programming Models Nonlinear Programming Models Fabio Schoen 2008 http://gol.dsi.unifi.it/users/schoen Nonlinear Programming Models p. Introduction Nonlinear Programming Models p. NLP problems minf(x) x S R n Standard form:

More information

Convex Optimization in Classification Problems

Convex Optimization in Classification Problems New Trends in Optimization and Computational Algorithms December 9 13, 2001 Convex Optimization in Classification Problems Laurent El Ghaoui Department of EECS, UC Berkeley elghaoui@eecs.berkeley.edu 1

More information

Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

More information

The moment-lp and moment-sos approaches

The moment-lp and moment-sos approaches The moment-lp and moment-sos approaches LAAS-CNRS and Institute of Mathematics, Toulouse, France CIRM, November 2013 Semidefinite Programming Why polynomial optimization? LP- and SDP- CERTIFICATES of POSITIVITY

More information

Convex Optimization: Applications

Convex Optimization: Applications Convex Optimization: Applications Lecturer: Pradeep Ravikumar Co-instructor: Aarti Singh Convex Optimization 1-75/36-75 Based on material from Boyd, Vandenberghe Norm Approximation minimize Ax b (A R m

More information

Geometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as

Geometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.

More information

minimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1,

minimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1, 4 Duality 4.1 Numerical perturbation analysis example. Consider the quadratic program with variables x 1, x 2, and parameters u 1, u 2. minimize x 2 1 +2x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x

More information

Sparse Optimization Lecture: Basic Sparse Optimization Models

Sparse Optimization Lecture: Basic Sparse Optimization Models Sparse Optimization Lecture: Basic Sparse Optimization Models Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know basic l 1, l 2,1, and nuclear-norm

More information

LMI MODELLING 4. CONVEX LMI MODELLING. Didier HENRION. LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ. Universidad de Valladolid, SP March 2009

LMI MODELLING 4. CONVEX LMI MODELLING. Didier HENRION. LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ. Universidad de Valladolid, SP March 2009 LMI MODELLING 4. CONVEX LMI MODELLING Didier HENRION LAAS-CNRS Toulouse, FR Czech Tech Univ Prague, CZ Universidad de Valladolid, SP March 2009 Minors A minor of a matrix F is the determinant of a submatrix

More information

Convex Optimization of Graph Laplacian Eigenvalues

Convex Optimization of Graph Laplacian Eigenvalues Convex Optimization of Graph Laplacian Eigenvalues Stephen Boyd Stanford University (Joint work with Persi Diaconis, Arpita Ghosh, Seung-Jean Kim, Sanjay Lall, Pablo Parrilo, Amin Saberi, Jun Sun, Lin

More information

Convex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)

Convex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples

More information

Tutorial on Convex Optimization: Part II

Tutorial on Convex Optimization: Part II Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

More information

Linear and non-linear programming

Linear and non-linear programming Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction ESE 605 Modern Convex Optimization mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

8. Geometric problems

8. Geometric problems 8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 1 Minimum volume ellipsoid around a set Löwner-John ellipsoid

More information

8. Geometric problems

8. Geometric problems 8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 Minimum volume ellipsoid around a set Löwner-John ellipsoid

More information

Fast Algorithms for SDPs derived from the Kalman-Yakubovich-Popov Lemma

Fast Algorithms for SDPs derived from the Kalman-Yakubovich-Popov Lemma Fast Algorithms for SDPs derived from the Kalman-Yakubovich-Popov Lemma Venkataramanan (Ragu) Balakrishnan School of ECE, Purdue University 8 September 2003 European Union RTN Summer School on Multi-Agent

More information

A direct formulation for sparse PCA using semidefinite programming

A direct formulation for sparse PCA using semidefinite programming A direct formulation for sparse PCA using semidefinite programming A. d Aspremont, L. El Ghaoui, M. Jordan, G. Lanckriet ORFE, Princeton University & EECS, U.C. Berkeley Available online at www.princeton.edu/~aspremon

More information

Convex Optimization. (EE227A: UC Berkeley) Lecture 6. Suvrit Sra. (Conic optimization) 07 Feb, 2013

Convex Optimization. (EE227A: UC Berkeley) Lecture 6. Suvrit Sra. (Conic optimization) 07 Feb, 2013 Convex Optimization (EE227A: UC Berkeley) Lecture 6 (Conic optimization) 07 Feb, 2013 Suvrit Sra Organizational Info Quiz coming up on 19th Feb. Project teams by 19th Feb Good if you can mix your research

More information

A direct formulation for sparse PCA using semidefinite programming

A direct formulation for sparse PCA using semidefinite programming A direct formulation for sparse PCA using semidefinite programming A. d Aspremont, L. El Ghaoui, M. Jordan, G. Lanckriet ORFE, Princeton University & EECS, U.C. Berkeley A. d Aspremont, INFORMS, Denver,

More information

III. Applications in convex optimization

III. Applications in convex optimization III. Applications in convex optimization nonsymmetric interior-point methods partial separability and decomposition partial separability first order methods interior-point methods Conic linear optimization

More information

Lecture Note 5: Semidefinite Programming for Stability Analysis

Lecture Note 5: Semidefinite Programming for Stability Analysis ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State

More information

Sparse Covariance Selection using Semidefinite Programming

Sparse Covariance Selection using Semidefinite Programming Sparse Covariance Selection using Semidefinite Programming A. d Aspremont ORFE, Princeton University Joint work with O. Banerjee, L. El Ghaoui & G. Natsoulis, U.C. Berkeley & Iconix Pharmaceuticals Support

More information

Fast ADMM for Sum of Squares Programs Using Partial Orthogonality

Fast ADMM for Sum of Squares Programs Using Partial Orthogonality Fast ADMM for Sum of Squares Programs Using Partial Orthogonality Antonis Papachristodoulou Department of Engineering Science University of Oxford www.eng.ox.ac.uk/control/sysos antonis@eng.ox.ac.uk with

More information

COURSE ON LMI PART I.2 GEOMETRY OF LMI SETS. Didier HENRION henrion

COURSE ON LMI PART I.2 GEOMETRY OF LMI SETS. Didier HENRION   henrion COURSE ON LMI PART I.2 GEOMETRY OF LMI SETS Didier HENRION www.laas.fr/ henrion October 2006 Geometry of LMI sets Given symmetric matrices F i we want to characterize the shape in R n of the LMI set F

More information

12. Interior-point methods

12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

More information

POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS

POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS POLYNOMIAL OPTIMIZATION WITH SUMS-OF-SQUARES INTERPOLANTS Sercan Yıldız syildiz@samsi.info in collaboration with Dávid Papp (NCSU) OPT Transition Workshop May 02, 2017 OUTLINE Polynomial optimization and

More information

Agenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms

Agenda. Interior Point Methods. 1 Barrier functions. 2 Analytic center. 3 Central path. 4 Barrier method. 5 Primal-dual path following algorithms Agenda Interior Point Methods 1 Barrier functions 2 Analytic center 3 Central path 4 Barrier method 5 Primal-dual path following algorithms 6 Nesterov Todd scaling 7 Complexity analysis Interior point

More information

IOE 611/Math 663: Nonlinear Programming

IOE 611/Math 663: Nonlinear Programming 1. Introduction I course logistics I mathematical optimization I least-squares and linear programming I convex optimization I example I course goals and topics I nonlinear optimization I brief history

More information

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44 Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

More information

Semidefinite Programming Basics and Applications

Semidefinite Programming Basics and Applications Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent

More information

4. Convex optimization problems (part 1: general)

4. Convex optimization problems (part 1: general) EE/AA 578, Univ of Washington, Fall 2016 4. Convex optimization problems (part 1: general) optimization problem in standard form convex optimization problems quasiconvex optimization 4 1 Optimization problem

More information

Relaxations and Randomized Methods for Nonconvex QCQPs

Relaxations and Randomized Methods for Nonconvex QCQPs Relaxations and Randomized Methods for Nonconvex QCQPs Alexandre d Aspremont, Stephen Boyd EE392o, Stanford University Autumn, 2003 Introduction While some special classes of nonconvex problems can be

More information

Module 04 Optimization Problems KKT Conditions & Solvers

Module 04 Optimization Problems KKT Conditions & Solvers Module 04 Optimization Problems KKT Conditions & Solvers Ahmad F. Taha EE 5243: Introduction to Cyber-Physical Systems Email: ahmad.taha@utsa.edu Webpage: http://engineering.utsa.edu/ taha/index.html September

More information

Semidefinite and Second Order Cone Programming Seminar Fall 2012 Project: Robust Optimization and its Application of Robust Portfolio Optimization

Semidefinite and Second Order Cone Programming Seminar Fall 2012 Project: Robust Optimization and its Application of Robust Portfolio Optimization Semidefinite and Second Order Cone Programming Seminar Fall 2012 Project: Robust Optimization and its Application of Robust Portfolio Optimization Instructor: Farid Alizadeh Author: Ai Kagawa 12/12/2012

More information

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

More information

EE Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko, JPL Third Term

EE Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko, JPL Third Term EE 150 - Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko JPL Third Term 2011-2012 Due on Thursday May 3 in class. Homework Set #4 1. (10 points) (Adapted

More information

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 18

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 18 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 18 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 31, 2012 Andre Tkacenko

More information

Gauge optimization and duality

Gauge optimization and duality 1 / 54 Gauge optimization and duality Junfeng Yang Department of Mathematics Nanjing University Joint with Shiqian Ma, CUHK September, 2015 2 / 54 Outline Introduction Duality Lagrange duality Fenchel

More information

January 29, Introduction to optimization and complexity. Outline. Introduction. Problem formulation. Convexity reminder. Optimality Conditions

January 29, Introduction to optimization and complexity. Outline. Introduction. Problem formulation. Convexity reminder. Optimality Conditions Olga Galinina olga.galinina@tut.fi ELT-53656 Network Analysis Dimensioning II Department of Electronics Communications Engineering Tampere University of Technology, Tampere, Finl January 29, 2014 1 2 3

More information

EE364 Review Session 4

EE364 Review Session 4 EE364 Review Session 4 EE364 Review Outline: convex optimization examples solving quasiconvex problems by bisection exercise 4.47 1 Convex optimization problems we have seen linear programming (LP) quadratic

More information

Convex Functions. Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK)

Convex Functions. Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK) Convex Functions Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK) Course on Convex Optimization for Wireless Comm. and Signal Proc. Jointly taught by Daniel P. Palomar and Wing-Kin (Ken) Ma

More information

Sparse PCA with applications in finance

Sparse PCA with applications in finance Sparse PCA with applications in finance A. d Aspremont, L. El Ghaoui, M. Jordan, G. Lanckriet ORFE, Princeton University & EECS, U.C. Berkeley Available online at www.princeton.edu/~aspremon 1 Introduction

More information

Matrix Rank Minimization with Applications

Matrix Rank Minimization with Applications Matrix Rank Minimization with Applications Maryam Fazel Haitham Hindi Stephen Boyd Information Systems Lab Electrical Engineering Department Stanford University 8/2001 ACC 01 Outline Rank Minimization

More information

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko

More information

Research overview. Seminar September 4, Lehigh University Department of Industrial & Systems Engineering. Research overview.

Research overview. Seminar September 4, Lehigh University Department of Industrial & Systems Engineering. Research overview. Research overview Lehigh University Department of Industrial & Systems Engineering COR@L Seminar September 4, 2008 1 Duality without regularity condition Duality in non-exact arithmetic 2 interior point

More information

Agenda. 1 Cone programming. 2 Convex cones. 3 Generalized inequalities. 4 Linear programming (LP) 5 Second-order cone programming (SOCP)

Agenda. 1 Cone programming. 2 Convex cones. 3 Generalized inequalities. 4 Linear programming (LP) 5 Second-order cone programming (SOCP) Agenda 1 Cone programming 2 Convex cones 3 Generalized inequalities 4 Linear programming (LP) 5 Second-order cone programming (SOCP) 6 Semidefinite programming (SDP) 7 Examples Optimization problem in

More information

Primal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization

Primal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization Primal-Dual Geometry of Level Sets and their Explanatory Value of the Practical Performance of Interior-Point Methods for Conic Optimization Robert M. Freund M.I.T. June, 2010 from papers in SIOPT, Mathematics

More information

Introduction and Math Preliminaries

Introduction and Math Preliminaries Introduction and Math Preliminaries Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/ yyye Appendices A, B, and C, Chapter

More information

Agenda. Applications of semidefinite programming. 1 Control and system theory. 2 Combinatorial and nonconvex optimization

Agenda. Applications of semidefinite programming. 1 Control and system theory. 2 Combinatorial and nonconvex optimization Agenda Applications of semidefinite programming 1 Control and system theory 2 Combinatorial and nonconvex optimization 3 Spectral estimation & super-resolution Control and system theory SDP in wide use

More information

Convex Optimization in Communications and Signal Processing

Convex Optimization in Communications and Signal Processing Convex Optimization in Communications and Signal Processing Prof. Dr.-Ing. Wolfgang Gerstacker 1 University of Erlangen-Nürnberg Institute for Digital Communications National Technical University of Ukraine,

More information

A Quick Tour of Linear Algebra and Optimization for Machine Learning

A Quick Tour of Linear Algebra and Optimization for Machine Learning A Quick Tour of Linear Algebra and Optimization for Machine Learning Masoud Farivar January 8, 2015 1 / 28 Outline of Part I: Review of Basic Linear Algebra Matrices and Vectors Matrix Multiplication Operators

More information

Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming

Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming School of Mathematics T H E U N I V E R S I T Y O H F E D I N B U R G Interior Point Methods: Second-Order Cone Programming and Semidefinite Programming Jacek Gondzio Email: J.Gondzio@ed.ac.uk URL: http://www.maths.ed.ac.uk/~gondzio

More information

Convex functions. Definition. f : R n! R is convex if dom f is a convex set and. f ( x +(1 )y) < f (x)+(1 )f (y) f ( x +(1 )y) apple f (x)+(1 )f (y)

Convex functions. Definition. f : R n! R is convex if dom f is a convex set and. f ( x +(1 )y) < f (x)+(1 )f (y) f ( x +(1 )y) apple f (x)+(1 )f (y) Convex functions I basic properties and I operations that preserve convexity I quasiconvex functions I log-concave and log-convex functions IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions

More information

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February

More information

Stochastic Subgradient Methods

Stochastic Subgradient Methods Stochastic Subgradient Methods Stephen Boyd and Almir Mutapcic Notes for EE364b, Stanford University, Winter 26-7 April 13, 28 1 Noisy unbiased subgradient Suppose f : R n R is a convex function. We say

More information

From Convex Optimization to Linear Matrix Inequalities

From Convex Optimization to Linear Matrix Inequalities Dep. of Information Engineering University of Pisa (Italy) From Convex Optimization to Linear Matrix Inequalities eng. Sergio Grammatico grammatico.sergio@gmail.com Class of Identification of Uncertain

More information

Determinant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University

Determinant maximization with linear. S. Boyd, L. Vandenberghe, S.-P. Wu. Information Systems Laboratory. Stanford University Determinant maximization with linear matrix inequality constraints S. Boyd, L. Vandenberghe, S.-P. Wu Information Systems Laboratory Stanford University SCCM Seminar 5 February 1996 1 MAXDET problem denition

More information

9. Geometric problems

9. Geometric problems 9. Geometric problems EE/AA 578, Univ of Washington, Fall 2016 projection on a set extremal volume ellipsoids centering classification 9 1 Projection on convex set projection of point x on set C defined

More information

Constrained Optimization and Lagrangian Duality

Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

More information

Conic Linear Programming. Yinyu Ye

Conic Linear Programming. Yinyu Ye Conic Linear Programming Yinyu Ye December 2004, revised October 2017 i ii Preface This monograph is developed for MS&E 314, Conic Linear Programming, which I am teaching at Stanford. Information, lecture

More information

CS675: Convex and Combinatorial Optimization Fall 2016 Convex Optimization Problems. Instructor: Shaddin Dughmi

CS675: Convex and Combinatorial Optimization Fall 2016 Convex Optimization Problems. Instructor: Shaddin Dughmi CS675: Convex and Combinatorial Optimization Fall 2016 Convex Optimization Problems Instructor: Shaddin Dughmi Outline 1 Convex Optimization Basics 2 Common Classes 3 Interlude: Positive Semi-Definite

More information

Sparse Matrix Theory and Semidefinite Optimization

Sparse Matrix Theory and Semidefinite Optimization Sparse Matrix Theory and Semidefinite Optimization Lieven Vandenberghe Department of Electrical Engineering University of California, Los Angeles Joint work with Martin S. Andersen and Yifan Sun Third

More information

Gradient Descent. Ryan Tibshirani Convex Optimization /36-725

Gradient Descent. Ryan Tibshirani Convex Optimization /36-725 Gradient Descent Ryan Tibshirani Convex Optimization 10-725/36-725 Last time: canonical convex programs Linear program (LP): takes the form min x subject to c T x Gx h Ax = b Quadratic program (QP): like

More information

Lecture 7: Convex Optimizations

Lecture 7: Convex Optimizations Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1

More information

Optimization in. Stephen Boyd. 3rd SIAM Conf. Control & Applications. and Control Theory. System. Convex

Optimization in. Stephen Boyd. 3rd SIAM Conf. Control & Applications. and Control Theory. System. Convex Optimization in Convex and Control Theory System Stephen Boyd Engineering Department Electrical University Stanford 3rd SIAM Conf. Control & Applications 1 Basic idea Many problems arising in system and

More information

Selected Examples of CONIC DUALITY AT WORK Robust Linear Optimization Synthesis of Linear Controllers Matrix Cube Theorem A.

Selected Examples of CONIC DUALITY AT WORK Robust Linear Optimization Synthesis of Linear Controllers Matrix Cube Theorem A. . Selected Examples of CONIC DUALITY AT WORK Robust Linear Optimization Synthesis of Linear Controllers Matrix Cube Theorem A. Nemirovski Arkadi.Nemirovski@isye.gatech.edu Linear Optimization Problem,

More information

A function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3

A function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3 Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is well-defined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all

More information

Interior Point Algorithms for Constrained Convex Optimization

Interior Point Algorithms for Constrained Convex Optimization Interior Point Algorithms for Constrained Convex Optimization Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Inequality constrained minimization problems

More information

18. Primal-dual interior-point methods

18. Primal-dual interior-point methods L. Vandenberghe EE236C (Spring 213-14) 18. Primal-dual interior-point methods primal-dual central path equations infeasible primal-dual method primal-dual method for self-dual embedding 18-1 Symmetric

More information

Convex Optimization Problems. Prof. Daniel P. Palomar

Convex Optimization Problems. Prof. Daniel P. Palomar Conve Optimization Problems Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R- Portfolio Optimization with R MSc in Financial Mathematics Fall 2018-19, HKUST,

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Convex Optimization Fourth lecture, 05.05.2010 Jun.-Prof. Matthias Hein Reminder from last time Convex functions: first-order condition: f(y) f(x) + f x,y x, second-order

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information