Convex Optimization in Communications and Signal Processing

Size: px
Start display at page:

Download "Convex Optimization in Communications and Signal Processing"

Transcription

1 Convex Optimization in Communications and Signal Processing Prof. Dr.-Ing. Wolfgang Gerstacker 1 University of Erlangen-Nürnberg Institute for Digital Communications National Technical University of Ukraine, KPI, Kiev, April These lecture slides are largely based on the book Convex Optimization by Steven Boyd and Lieven Vandenberghe and the corresponding slides. Many thanks to Prof. Boyd for the permission to use his materials for this course.

2 1 Introduction 2 Convex sets 3 Convex functions 4 Convex optimization problems 5 Duality 6 Applications in communications 7 Algorithms Equality constrained minimization 8 Conclusions 0

3 1. Introduction Outline 1 Introduction 2 Convex sets 1

4 1. Introduction Introduction mathematical optimization least-squares and linear programming convex optimization course goals and topics nonlinear optimization brief history of convex optimization 2

5 1. Introduction Mathematical optimization (mathematical) optimization problem minimize subject to f 0 (x) f i (x) b i, i = 1,..., m x = (x 1,..., x n ): optimization variables f 0 : R n R: objective function f i : R n R, i = 1,..., m: constraint functions optimal solution x has smallest value of f 0 among all vectors that satisfy the constraints 3

6 1. Introduction Examples portfolio optimization variables: amounts invested in different assets constraints: budget, max./min. investment per asset, minimum return objective: overall risk or return variance device sizing in electronic circuits variables: device widths and lengths constraints: manufacturing limits, timing requirements, maximum area objective: power consumption data fitting variables: model parameters constraints: prior information, parameter limits objective: measure of misfit or prediction error 4

7 1. Introduction Examples (communications and signal processing) channel estimation detection filter design beamformer design network optimization power control... 5

8 1. Introduction Solving optimization problems general optimization problem very difficult to solve methods involve some compromise, e.g., very long computation time, or not always finding the solution exceptions: certain problem classes can be solved efficiently and reliably least-squares problems linear programming problems convex optimization problems 6

9 1. Introduction Least-squares I minimize Ax b 2 2 interpretation linear system of equations Ax = b can be also written as a T i x = b i, i = 1,..., k, where a T i are the rows of A R k n and b i are the entries of b R k Ax b 2 2 is equal to k (a T i x b i) 2, i.e. the sum of i=1 squared equation errors solving least-squares problems analytical solution: x = (A T A) 1 A T b reliable and efficient algorithms and software 7

10 1. Introduction Least-squares II computation time proportional to n 2 k (A R k n ); less if structured a mature technology using least-squares least-squares problems are easy to recognize a few standard techniques increase flexibility (e.g., including weights, adding regularization terms) 8

11 1. Introduction Calculation of least-squares solution I cost function: J(x) = Ax b 2 2 = (Ax b)t (Ax b) = x T A T Ax b T Ax x T A T b + b T b we need the gradient vector J [ x = J x 1 J x 2... J x n ] T differentiation rules for some simple cost functions: J(x) = c T x = c 1 x c n x n J x = [ c 1... c n ] T = c J(x) = x T Cx J x = 2 Cx 9

12 1. Introduction Calculation of least-squares solution II for the least-squares cost function we get J x = 2 AT Ax 2 A T b setting the gradient vector to zero yields the solution x = (A T A) 1 A T b this is the global minimum since the function is differentiable, there is only one local extremum and x T A T Ax = (Ax) T (Ax) 0 x 10

13 1. Introduction Modified least-squares cost functions weighted cost function J(x) = k w i (a T i x b i) 2 i=1 with nonnegative weight factors w i regularized cost function J(x) = k (a T i x b i) 2 + ρ n i=1 i=1 x 2 i both cost functions can be minimized with similar calculations as the original least-squares cost function 11

14 1. Introduction Linear programming minimize subject to c T x a T i x b i, i = 1,..., m solving linear programs no analytical formula for solution reliable and efficient algorithms and software (e.g. Simplex algorithm by Dantzig) computation time proportional to n 2 m if m n; less if structured a mature technology using linear programming not as easy to recognize as least-squares problems a few standard tricks used to convert problems into linear programs (e.g., problems involving l 1 - or l -norms, piecewise-linear functions) 12

15 1. Introduction example for a linear programming problem: Chebyshev approximation problem minimize max i=1,...,m a T i x b i Problem is equivalent to the linear program minimize subject to t a T i x t b i, i = 1,..., m a T i x t b i, i = 1,..., m t 0 with variables x and t 13

16 1. Introduction Convex optimization problem minimize f 0 (x) subject to f i (x) b i, i = 1,..., m objective and constraint functions are convex: if α + β = 1, α 0, β 0 f i (αx + βy) αf i (x) + βf i (y) includes least-squares problems and linear programs as special cases 14

17 1. Introduction solving convex optimization problems no analytical solution reliable and efficient algorithms (e.g. interior point algorithms) computation time (roughly) proportional to max{n 3, n 2 m, F}, where F is cost of evaluating f i s and their first- and second-order derivatives almost a technology for subclasses of convex problems like second-order cone programming 15

18 1. Introduction using convex optimization often difficult to recognize (is a given function convex?) many tricks for transforming problems into convex form surprisingly many problems can be solved via convex optimization in particular, during the last years, a variety of problems in communications and signal processing could be solved via convex optimization 16

19 1. Introduction Course goals and topics goals 1 recognize/formulate problems as convex optimization problems 2 characterize optimal solution (optimal power distribution), give limits of performance, etc. 3 apply techniques to problems in communications and signal processing 4 understand the basic principles of convex optimization algorithms topics 1 convex sets, functions, optimization problems 2 examples and applications 3 algorithms 17

20 1. Introduction Nonlinear optimization traditional techniques for general nonconvex problems involve compromises local optimization methods (nonlinear programming) find a point that minimizes f 0 among feasible points near it fast, can handle large problems require initial guess provide no information about distance to (global) optimum Example: gradient search global optimization methods find the (global) solution worst-case complexity grows exponentially with problem size these algorithms are often based on solving convex subproblems 18

21 1. Introduction Brief history of convex optimization I theory (convex analysis): ca algorithms 1947: simplex algorithm for linear programming (Dantzig) 1960s: early interior-point methods (Fiacco & McCormick, Dikin,... ) 1970s: ellipsoid method and other subgradient methods 1980s: polynomial-time interior-point methods for linear programming (Karmarkar 1984) late 1980s now: polynomial-time interior-point methods for nonlinear convex optimization (Nesterov & Nemirovski 1994) 19

22 1. Introduction Brief history of convex optimization II applications before 1990: mostly in operations research; few in engineering since 1990: many new applications in engineering (control, signal processing, communications, circuit design,... ); new problem classes (semidefinite and second-order cone programming, robust optimization) 20

23 2. Convex sets Outline 1 Introduction 2 Convex sets 3 Convex functions 21

24 2. Convex sets Convex sets vector spaces and subspaces affine and convex sets some important examples operations that preserve convexity separating and supporting hyperplanes 22

25 2. Convex sets Vector spaces and subspaces variables to be optimized are typically collected in vectors we need the concept of a vector space from mathematics: collection of vectors for which vector addition and multiplication of vectors with scalars are defined; these operations must fulfill a number of requirements (axioms) such as associativity of addition, commutativity of addition etc. one important property of a vector space is closure: the result of addition and scalar multiplication, respectively, belongs also to the vector space subspace: subset of a vector space that is closed under addition and scalar multiplication 23

26 2. Convex sets Affine set line through x 1, x 2 : all points x = θx 1 + (1 θ)x 2 = x 2 + θ (x 1 x 2 ) (θ R) affine set: contains the line through any two distinct points in the set 24

27 2. Convex sets example: solution set of linear equations {x Ax = b} assume that we have two solutions x 1, x 2 Ax 1 = b, Ax 2 = b θ Ax 1 = θ b, (1 θ) Ax 2 = (1 θ) b A (θ x 1 + (1 θ) x 2 )) = (θ + (1 θ)) b = b θ x 1 + (1 θ) x 2, θ R is also a solution, set is affine (conversely, every affine set can be expressed as solution set of system of linear equations) 25

28 2. Convex sets Convex set line segment between x 1 and x 2 : all points with 0 θ 1 x = θx 1 + (1 θ)x 2 convex set: contains line segment between any two points in the set x 1, x 2 C, 0 θ 1 θx 1 + (1 θ)x 2 C examples: (one convex, two nonconvex sets) 26

29 2. Convex sets Convex combination and convex hull convex combination of x 1,..., x k : any point x of the form x = θ 1 x 1 + θ 2 x θ k x k with θ θ k = 1, θ i 0 convex hull conv S: set of all convex combinations of points in S conv S is also smallest convex set that contains S 27

30 2. Convex sets Cones and convex cones set C is called a cone if for any x C and θ 0, θ x C conic (nonnegative) combination of x 1 and x 2 : any point of the form x = θ 1 x 1 + θ 2 x 2 with θ 1 0, θ 2 0 two dimensional pie slice with apex 0 and edges passing through x 1 and x 2 28

31 2. Convex sets convex cone: set is convex and a cone; set that contains all conic combinations of points in the set for any x 1, x 2 C, θ 1 0, θ 2 0, we have θ 1 x 1 + θ 2 x 2 C 29

32 2. Convex sets Some important facts any line is affine; if it passes through zero, it is a subspace any line segment is convex, but not affine any subspace is affine, and a convex cone any ray, i.e. a set {x 0 + θ v θ 0}, v 0, is convex but not affine; if x 0 = 0, it is a convex cone 30

33 2. Convex sets Hyperplanes and halfspaces hyperplane: set of the form {x a T x = b}(a 0) halfspace: set of the form {x a T x b}(a 0) a is the normal vector hyperplanes are affine and convex; halfspaces are convex 31

34 2. Convex sets Alternative representation of hyperplanes hyperplane in a two-dimensional vector space (line): x = x 0 + θ c, θ R c is orthogonal to a, c T a = 0 a T x = a T x }{{} 0 +θ }{{} a T c b 0 hyperplane in a three-dimensional vector space (plane): x = x 0 + θ 1 c 1 + θ 2 c 2, θ 1, θ 2 R c 1, c 2 are orthogonal to a, c T 1 a = 0, ct 2 a = 0 a T x = a T x }{{} 0 +θ 1 a T c 1 +θ }{{} 2 a T c }{{} 2 b

35 2. Convex sets Euclidean balls and ellipsoids (Euclidian) ball with center x c and radius r: B(x c, r) = {x x x c 2 r} = {x c + ru u 2 1} ellipsoid: set of the form {x (x x c ) T P 1 (x x c ) 1} with P S n ++ (S n ++: set of all symmetric positive definite matrices) other representation: {x c + Au u 2 1} with A square and nonsingular (A = P 1/2 ) Euclidean balls and ellipsoids are convex sets 33

36 2. Convex sets Norm balls and norm cones norm: a function that satisfies x 0; x = 0 if and only if x = 0 tx = t x for t R x + y x + y notation: is general (unspecified) norm; symb is particular norm examples: l 1 -norm: x 1 = x 1 + x x n l 2 -norm: x 2 = x x x n 2 (Euclidean norm) l -norm: x = max i=1,...,n x i l p -norm: x p = p x 1 p + x 2 p x n p 34

37 2. Convex sets norm ball with center x c and radius r: {x x x c r} norm cone: {(x, t) x t} Euclidean norm cone is called second-order cone or ice-cream cone or quadratic cone (since it can be defined by a quadratic inequality) norm balls and cones are convex 35

38 2. Convex sets Polyhedra solution set of finitely many linear inequalities and equalities Ax b, Cx = d (A R m n, C R p n, is componentwise inequality) polyhedron is intersection of finite number of halfspaces and hyperplanes 36

39 2. Convex sets example: nonnegative orthant R n + = {x R n x i 0, i = 1,..., n} = {x R n x 0} R n + is polyhedron and convex cone affine sets (e.g. subspaces, hyperplanes, lines), rays, line segments, halfspaces are all polyhedra bounded polyhedron is called polytope all polyhedra are convex 37

40 2. Convex sets Positive semidefinite cone notation: S n is set of symmetric n n matrices and forms a linear space or vector space S n + = {X S n X 0}: positive semidefinite n n matrices S n + is a convex cone X S n + z T Xz 0 for all z S n ++ = {X S n X 0}: positive definite n n matrices X S n ++ z T Xz > 0 for all z 0 38

41 2. Convex sets Operations that preserve convexity practical methods for establishing convexity of a set C 1 apply definition x 1, x 2 C, 0 θ 1 θx 1 + (1 θ)x 2 C 2 show that C is obtained from simple convex sets (hyperplanes, halfspaces, norm balls,... ) by operations that preserve convexity intersection affine functions perspective function linear-fractional functions 39

42 2. Convex sets Intersection the intersection of (any number of) convex sets is convex 40

43 2. Convex sets Affine function suppose f : R n R m is affine (f(x) = Ax + b with A R m n, b R m ) the image of a convex set under f is convex S R n convex f(s) = {f(x) x S} convex the inverse image f 1 (C) of a convex set under f is convex C R m convex f 1 (C) = {x R n f(x) C} convex examples scaling, translation projection 41

44 2. Convex sets Perspective and linear-fractional function perspective function P : R n+1 R n : P(x, t) = x/t, dom P = {(x, t) t > 0} images and inverse images of convex sets under perspective functions are convex 42

45 2. Convex sets linear-fractional function f : R n R m : f(x) = Ax + b c T x + d, dom f = {x ct x + d > 0} f( ) may be viewed as concatenation of P( ) and a function g( ), f( ) = P( ) g( ), with g(x) = [ A c T ] [ b x + d images and inverse images of convex sets under linear-fractional functions are convex ] 43

46 2. Convex sets Separating hyperplane theorem if C and D are disjoint convex sets, then there exists a 0, b such that a T x b for x C, a T x b for x D the hyperplane {x a T x = b} separates C and D strict separation (i.e., at least in one of both inequalities, resp. can be replaced by < resp. > ) requires additional assumptions (e.g., C is closed, D is a singleton) 44

47 2. Convex sets Supporting hyperplane theorem supporting hyperplane to set C at boundary point x 0 : {x a T x = a T x 0 } where a 0 and a T x a T x 0 for all x C hyperplane is tangent to C at x 0 halfspace defined by the hyperplane (in opposite direction of a) contains C 45

48 2. Convex sets supporting hyperplane theorem: if C is convex, then there exists a supporting hyperplane at every boundary point of C 46

49 3. Convex functions Outline 2 Convex sets 3 Convex functions 4 Convex optimization problems 47

50 3. Convex functions Convex functions basic properties and examples operations that preserve convexity 48

51 3. Convex functions Definition f : R n R is convex if dom f is a convex set and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all x, y dom f, 0 θ 1 f is concave if f is convex f is strictly convex if dom f is convex and f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) for x, y dom f, x y, 0 < θ < 1 49

52 3. Convex functions convexity means that the line segment through any two points of the graph of the function is above the graph consider a function of one scalar variable, f(z) straight line through (x, f(x)) and (y, f(y)): g(z) = a z + b g(x) = a x + b! = f(x), g(y) = a y + b! = f(y) a = (f(y) f(x))/(y x), b = f(x) (f(y) f(x))/(y x) x g(z) = f(x) + (f(y) f(x))/(y x) (z x) g(θ x + (1 θ) y) = f(x) + (f(y) f(x))/(y x) (x (θ 1) + (1 θ) y) = f(x) + (1 θ) (f(y) f(x)) = θ f(x) + (1 θ) f(y) f(θ x + (1 θ) y) 50

53 3. Convex functions Examples on R convex: affine: ax + b on R, for any a, b R exponential: e ax, for any a R powers: x α on R ++, for α 1 or α 0 powers of absolute value: x p on R, for p 1 negative entropy: x log x on R ++ concave affine: ax + b on R, for any a, b R powers: x α on R ++, for 0 α 1 logarithm: log x on R ++ 51

54 3. Convex functions Examples on R n and R m n affine functions are convex and concave all norms are convex due to the following inequality valid for any norm function f(x): f(θ x + (1 θ) y) f(θ x) + f((1 θ) y) = θ f(x) + (1 θ) f(y) examples on R n affine function f(x) = a T x + b norms: x p = ( n i=1 x i p ) 1/p for p 1; x = max k x k 52

55 3. Convex functions examples on R m n (m n matrices) affine function f(x) = tr(a T X) + b = m n A ij X ij + b i=1 j=1 spectral (maximum singular value) norm f(x) = X 2 = σ max (X) = (λ max (X T X)) 1/2 53

56 3. Convex functions Restriction of a convex function to a line f : R n R is convex if and only if the function g : R R, g(t) = f(x + tv), dom g = {t x + tv dom f} is convex (in t) for any x dom f, v R n can check convexity of f by checking convexity of functions of one variable 54

57 3. Convex functions First-order condition f is differentiable if dom f is open and the gradient ( f(x) f(x) =, f(x),, f(x) ), x 1 x 2 x n exists at each x dom f 1st-order condition: differentiable f with convex domain is convex iff f(y) f(x) + f(x) T (y x) for all x, y dom f first-order Taylor series approximation of f is global underestimator 55

58 3. Convex functions first order approximation establishes a global lower bound from local information about the function (function value, gradient vector) we can derive global information (global underestimator) f(x) = 0 f(y) f(x) x is global minimizer of f 56

59 3. Convex functions Second-order conditions f is twice differentiable if dom f is open and the Hessian 2 f(x) S n, exists at each x dom f 2 f(x) ij = 2 f(x) x i x j, i, j = 1,..., n, 2nd-order conditions: for twice differentiable f with convex domain f is convex if and only if 2 f(x) 0 for all x dom f if 2 f(x) 0 for all x dom f, then f is strictly convex 57

60 3. Convex functions Examples quadratic function: f(x) = (1/2)x T Px + q T x + r (with P S n ) f(x) = Px + q, 2 f(x) = P convex if P 0 least-squares objective: f(x) = Ax b 2 2 f(x) = 2A T (Ax b), 2 f(x) = 2A T A convex (for any A) quadratic-over-linear: f(x, y) = x 2 /y 2 f(x, y) = 2 y 3 [ convex for y > 0 y x ] [ y x ] T 0 58

61 3. Convex functions expression for the Hessian can be verified via the partial derivatives: 2 f(x, y) x x = 2 y 2 f(x, y) y y 2 f(x, y) x y 2 f(x, y) y x = 2 x2 y 3 = 2 x y 2 = 2 x y 2 59

62 3. Convex functions log-sum-exp: f(x) = log n k=1 exp x k is convex 2 f(x) = 1 1 T z diag(z) 1 (1 T z) 2 zzt (z k = exp x k ) to show 2 f(x) 0, we must verify that v T 2 f(x)v 0 for all v: v T 2 f(x)v = ( k z kv 2 k )( k z k) ( k v kz k ) 2 ( k z k) 2 0 since ( k v kz k ) 2 ( k z kvk 2)( k z k) (from Cauchy-Schwarz inequality ( n i=1 a i b i ) 2 n i=1 a2 i n i=1 b2 i, use a i = z i, b i = z i v i ) geometric mean: f(x) = ( n k=1 x k) 1/n on R n ++ is concave (similar proof as for log-sum-exp) 60

63 3. Convex functions Epigraph and sublevel set α-sublevel set of f : R n R: C α = {x dom f f(x) α} sublevel sets of convex functions are convex (converse is false) epigraph of f : R n R: epi f = {(x, t) R n+1 x dom f, f(x) t} (epi means above ) f is convex if and only if epi f is a convex set 61

64 3. Convex functions Jensen s inequality basic inequality: if f is convex, then for 0 θ 1, f(θx + (1 θ)y) θf(x) + (1 θ)f(y) extension: if f is convex, then f(e z) E f(z) for any random variable z 62

65 3. Convex functions basic inequality is special case with discrete distribution prob(z = x) = θ, prob(z = y) = 1 θ general discrete distribution f ( k ) prob(z i ) z i i=1 k prob(z i ) f(z i ) i=1 f(θ 1 x 1 +θ 2 x θ k x k ) θ 1 f(x 1 )+θ 2 f(x 2 )+...+θ k f(x k ) with θ i 0, k θ i = 1 i=1 63

66 3. Convex functions continuous distribution p(x) ( ) f p(x) x dx p(x) f(x) dx S S with p(x) 0, S p(x) dx = 1, S dom f dithering with respect to a deterministic vector x Ef(x + z) f(x) with zero mean random vector z 64

67 3. Convex functions Operations that preserve convexity practical methods for establishing convexity of a function 1 verify definition (often simplified by restricting to a line) 2 for twice differentiable functions, show 2 f(x) 0 3 show that f is obtained from simple convex functions by operations that preserve convexity nonnegative weighted sum composition with affine function pointwise maximum and supremum composition minimization perspective 65

68 3. Convex functions Positive weighted sum & composition with affine function nonnegative multiple: αf is convex if f is convex, α 0 sum: f 1 + f 2 convex if f 1, f 2 convex (extends to infinite sums, integrals) f 1 (θ x + (1 θ) y) + f 2 (θ x + (1 θ) y) θ f 1 (x) + (1 θ) f 1 (y) + θ f 2 (x) + (1 θ) f 2 (y) = θ (f 1 (x) + f 2 (x)) + (1 θ) (f 1 (y) + f 2 (y)) 66

69 3. Convex functions composition with affine function: f(ax + b) is convex if f is convex f(a (θ x + (1 θ) y) + b) = f(θ (Ax + b) + (1 θ) (Ay + b)) θ f(ax + b) + (1 θ) f(ay + b) 67

70 3. Convex functions Pointwise maximum I if f 1,..., f m are convex, then f(x) = max{f 1 (x),..., f m (x)} is convex examples piecewise-linear function: f(x) = max i=1,,m (a T i x + b i) is convex sum of r largest components of x R n : f(x) = x [1] + x [2] + + x [r] is convex (x [i] is ith largest component of x) proof: f(x) = max{x i1 + x i2 + + x ir 1 i 1 < i 2 < < i r n} 68

71 3. Convex functions Pointwise maximum II consider e.g. x = [x 1 x 2 x 3 ] T (n = 2), f(x) = x [1] + x [2] (r = 2) equivalently, f can be expressed as f(x) = max{x 1 + x 2, x 2 + x 3, x 1 + x 3 } 69

72 3. Convex functions Pointwise supremum supremum: smallest upper bound if f(x, y) is convex in x for each y A, then is convex g(x) = sup f(x, y) y A 70

73 3. Convex functions Composition with scalar functions composition of g : R n R and h : R R: f is convex if f(x) = h(g(x)) g convex, h convex, h nondecreasing g concave, h convex, h nonincreasing proof (for n = 1, differentiable g, h on R (functions and derivatives exist for R)) f (x) = h (g(x))g (x) 2 + h (g(x))g (x) note: monotonicity must hold for extended-value extension h examples exp g(x) is convex if g is convex 1/g(x) is convex if g is concave and positive 71

74 3. Convex functions Vector composition composition of g : R n R k and h : R k R: f(x) = h(g(x)) = h(g 1 (x), g 2 (x),..., g k (x)) f is convex if: g i convex, h convex, h nondecreasing in each argument g i concave, h convex, h nonincreasing in each argument proof (for n = 1, differentiable g, h) f (x) = g (x) T 2 h(g(x))g (x) + h(g(x)) T g (x) examples m i=1 log g i(x) is concave if g i are concave and positive log m i=1 exp g i(x) is convex if g i are convex 72

75 3. Convex functions Minimization I if f(x, y) is convex in (x, y) and C is a convex set, then g(x) = inf f(x, y) y C is convex; infimum: "greatest lower bound" 73

76 3. Convex functions Perspective the perspective of a function f : R n R is the function g : R n R R, g(x, t) = t f(x/t), dom g = {(x, t) x/t dom f, t > 0} g is convex if f is convex this can be shown via the epigraph: (x, t, s) epi g t f(x/t) s f(x/t) s/t (x/t, s/t) epi f thus, epi g is the inverse image of epi f under the perspective mapping (u, v, w) (u/v, w/v); inverse images of convex sets under perspective are convex! 74

77 4. Convex optimization problems Outline 3 Convex functions 4 Convex optimization problems 5 Duality 75

78 4. Convex optimization problems Convex optimization problems optimization problem in standard form convex optimization problems linear optimization quadratic optimization semidefinite programming 76

79 4. Convex optimization problems Optimization problem in standard form minimize f 0 (x) subject to f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p x R n is the optimization variable f 0 : R n R is the objective or cost function f i : R n R, i = 1,, m, are the inequality constraint functions h i : R n R are the equality constraint functions optimal value: p = inf{f 0 (x) f i (x) 0, i = 1,, m, h i (x) = 0, i = 1,, p} 77 p = if problem is infeasible (no x satisfies the constraints) p = if problem is unbounded below

80 4. Convex optimization problems Optimal and locally optimal points x is feasible if x domf 0 and it satisfies the constraints a feasible x is optimal if f 0 (x) = p ; X opt is the set of optimal points x is locally optimal if there is an R > 0 such that x is optimal for minimize (over z) subject to f 0 (z) f i (z) 0, i = 1,..., m, h i (z) = 0, i = 1,, p z x 2 R examples (with n = 1, m = p = 0) f 0 (x) = 1/x, dom f 0 = R ++ : p = 0, no optimal point f 0 (x) = log(x), dom f 0 = R ++ : p = f 0 (x) = x log x, dom f 0 = R ++ : p = 1/e, x = 1/e is optimal f 0 (x) = x 3 3x, p =, local optimum at x = 1 78

81 4. Convex optimization problems Implicit constraints the standard form optimization problem has an implicit constraint m p x D = dom f i dom h i i=0 we call D the domain of the problem the constraints f i (x) 0, h i (x) = 0 are the explicit constraints a problem is unconstrained if it has no explicit constraints (m = p = 0) example minimize f 0 (x) = k log(b i a T i=1 i x) is an unconstrained problem with implicit constraints a T i x < b i i=1 79

82 4. Convex optimization problems Feasibility problem find x subject to f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,, p can be considered a special case of the general problem with f 0 (x) = 0 minimize 0 subject to f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,, p p = 0 if constraints are feasible; any feasible x is optimal p = if constraints are infeasible 80

83 4. Convex optimization problems Convex optimization problem standard form convex optimization problem minimize f 0 (x) subject to f i (x) 0, i = 1,..., m a T i x = b i, i = 1,..., p f 0, f 1,..., f m are convex; equality constraints are affine often written as minimize subject to f 0 (x) f i (x) 0, i = 1,..., m Ax = b important property: feasible set of a convex optimization problem is convex 81

84 4. Convex optimization problems example minimize f 0 (x) = x1 2 + x2 2 subject to f 1 (x) = x 1 /(1 + x2 2) 0 h 1 (x) = (x 1 + x 2 ) 2 = 0 f 0 is convex; feasible set {(x 1, x 2 ) x 1 = x 2 0} is convex not a convex problem (according to our definition): f 1 is not convex, h 1 is not affine equivalent (but not identical) to the convex problem minimize x x2 2 subject to x 1 0 x 1 + x 2 = 0 82

85 4. Convex optimization problems Local and global optima any locally optimal point of a convex problem is (globally) optimal proof: suppose x is locally optimal and y is optimal with f 0 (y) < f 0 (x) x locally optimal means there is an R > 0 such that z feasible, z x 2 R f 0 (z) f 0 (x) 83 consider z = θy + (1 θ)x with θ = R/(2 y x 2 ) y x 2 > R, so 0 < θ < 1/2 z is a convex combination of two feasible points, hence also feasible z x 2 = R/2 and f 0 (z) θf 0 (y) + (1 θ)f 0 (x) < f 0 (x) which contradicts our assumption that x is locally optimal

86 4. Convex optimization problems or use some simplified reasoning: consider z = θy + (1 θ)x with very small θ z lies in Euclidean ball around x with radius R z is feasible, since feasible set is convex f 0 (z) θ f 0 (y) + (1 θ) f 0 (x) < θ f 0 (x) + (1 θ) f 0 (x) = f 0 (x) contradiction to assumption that x is locally optimum f 0 (y) f 0 (x) 84

87 4. Convex optimization problems Optimality criterion for differentiable f 0 x is optimal if and only if it is feasible and f 0 (x) T (y x) 0 for all feasible y if nonzero, f 0 (x) defines a supporting hyperplane to feasible set X at x 85

88 4. Convex optimization problems proof: f 0 (x) T (y x) 0 is valid f 0 (y) f 0 (x) + f 0 (x) T (y x) f 0 (x) x is optimum point converse: see Boyd book 86

89 4. Convex optimization problems unconstrained problem: x is optimal if and only if x dom f 0, f 0 (x) = 0 equality constrained problem minimize f 0 (x) subject to Ax = b x is optimal if and only if there exists a ν such that x dom f 0, Ax = b, f 0 (x) + A T ν = 0 proof: f 0 (x) T (y x) 0 must hold for optimum x for all y satisfying Ay = b all admissible vectors y can be expressed as y = F z + x 0 where x 0 is a particular solution to the linear system of equations and the columns of F span the nullspace of A 87

90 4. Convex optimization problems f 0 (x) T F z 0 z f 0 (x) is orthogonal to the nullspace of A; orthogonal complement of nullspace of A is identical to the column space of A T f 0 (x) = A T v minimization over nonnegative orthant minimize f 0 (x) subject to x 0 x is optimal if and only if x dom f 0, x 0, { f0 (x) i 0 x i = 0 f 0 (x) i = 0 x i > 0 proof: f 0 (x) T y f 0 (x) T x y 0 for a particular x 0 f 0 (x) T y is unbounded below unless f 0 (x) 0 demand: f 0 (x) T x = 0 88

91 4. Convex optimization problems Equivalent convex problems two problems are (informally) equivalent if the solution of one is readily obtained from the solution of the other, and vice-versa some common transformations that preserve convexity: eliminating equality constraints minimize subject to is equivalent to f 0 (x) f i (x) 0, i = 1,..., m Ax = b minimize (over z) f 0 (Fz + x 0 ) subject to f i (Fz + x 0 ) 0, i = 1,..., m where F and x 0 are such that Ax = b x = Fz + x 0 for some z 89

92 4. Convex optimization problems introducing equality constraints minimize f 0 (A 0 x + b 0 ) subject to f i (A i x + b i ) 0, i = 1,..., m is equivalent to minimize (over x, y i ) f 0 (y 0 ) subject to f i (y i ) 0, i = 1,..., m y i = A i x + b i, i = 0, 1,..., m introducing slack variables for linear inequalities minimize subject to is equivalent to minimize (over x, s) subject to f 0 (x) a T i x b i, i = 1,..., m f 0 (x) a T i x + s i = b i, i = 1,..., m s i 0, i = 1,..., m 90

93 4. Convex optimization problems epigraph form: standard form convex problem is equivalent to minimize (over x, t) t subject to f 0 (x) t 0 f i (x) 0, i = 1,..., m Ax = b minimizing over some variables is equivalent to minimize f 0 (x 1, x 2 ) subject to f i (x 1 ) 0, i = 1,..., m minimize f 0 (x 1 ) subject to f i (x 1 ) 0, i = 1,..., m where f 0 (x 1 ) = inf x2 f 0 (x 1, x 2 ) 91

94 4. Convex optimization problems Linear program (LP) minimize subject to c T x + d Gx h Ax = b convex problem with affine objective and constraint functions feasible set is a polyhedron 92

95 4. Convex optimization problems Linear-fractional program minimize subject to linear-fractional program f 0 (x) Gx h Ax = b f 0 (x) = ct x + d e T x + f, dom f 0(x) = {x e T x + f > 0} a quasiconvex optimization problem; can be solved by bisection also equivalent to the LP (variables y, z ) minimize subject to c T y + dz Gy hz Ay = bz e T y + fz = 1 z 0 93

96 4. Convex optimization problems Quadratic program (QP) minimize subject to (1/2)x T Px + q T x + r Gx h Ax = b P S n +, so objective is convex quadratic minimize a convex quadratic function over a polyhedron 94

97 4. Convex optimization problems Examples least-squares minimize Ax b 2 2 analytical solution x = A b (A is pseudo-inverse) can add linear constraints, e.g., l x u linear program with random cost minimize c T x + γx T Σx = E (c T x) + γ Var (c T x) subject to Gx h, Ax = b c is random vector with mean c and covariance Σ hence, c T x is random variable with mean c T x and variance x T Σx γ > 0 is risk aversion parameter; controls the trade-off between expected cost and variance (risk) 95

98 4. Convex optimization problems Quadratically constrained quadratic program (QCQP) minimize (1/2)x T P 0 x + q T 0 x + r 0 subject to (1/2)x T P i x + q T i x + r i 0, i = 1,..., m Ax = b P i S n +; objective and constraints are convex quadratic if P 1,..., P m S n ++, feasible region is intersection of m ellipsoids and an affine set (standard form of ellipsoid: {x (x x c,i ) T B 1 i (x x c,i ) 1} with B i S n ++) 96

99 4. Convex optimization problems Second-order cone programming minimize f T x subject to A i x + b i 2 c T i x + d i, i = 1,..., m Fx = q (A i R n i n, F R p n ) closely related to quadratic programming inequalities are called second-order cone (SOC) constraints: (A i x + b i, c T i x + d i) second-order cone in R n i+1 (affine function (A i x + b i, c T i x + d i) has to lie in the second order cone in R n i+1 ) for A i = 0, SOCP reduces to an LP; if c i = 0, it reduces to a QCQP more general than QCQP and LP 97

100 4. Convex optimization problems Semidefinite program (SDP) minimize c T x subject to x 1 F 1 + x 2 F x n F n + G 0 Ax = b with F i, G S k inequality constraint is called linear matrix inequality (LMI) includes problems with multiple LMI constraints: for example, x 1 ˆF x n ˆF n + Ĝ 0, x 1 ˆF x n ˆF n + Ĝ 0 is equivalent to single LMI [ ] [ ] [ ˆF x 1 0 ˆF 1 +x 2 0 ˆF 2 + +x n 0 n 0 ˆF 1 0 ˆF 2 0 ˆF n ] [ Ĝ Ĝ ] 0 98

101 4. Convex optimization problems LP and SOCP as SDP LP and equivalent SDP LP: minimize c T x SDP: minimize c T x subject to Ax b subject to diag(ax b) 0 (note different interpretation of generalized inequality ) SOCP and equivalent SDP SOCP: minimize f T x subject to A i x + b i 2 c T i x + d i, i = 1,..., m SDP: minimize f [ T x (c T subject to i x + d i )I A i x + b i (A i x + b i ) T c T i x + d i ] 0, i = 1,..., m 99

102 4. Convex optimization problems proof of equivalence: we consider the block matrix [ ] A b P = b T c Schur complement of A in P: S = c b T A 1 b P is positive semidefinite if and only if S is positive semidefinite, provided A is positive definite For [ (c T P = i x + d i )I A i x + b i (A i x + b i ) T c T i x + d i ], and a positive value of c T i x + d i, the positive definiteness of P implies S = (c T i x + d 1 i) c T i x + d A i x + b i i A i x + b i 2 2 (ct i x + d i) 2 100

103 5. Duality Outline 4 Convex optimization problems 5 Duality 6 Applications in communications 101

104 5. Duality Duality Lagrange dual problem weak and strong duality geometric interpretation optimality conditions examples 102

105 5. Duality Lagrangian standard form problem (not necessarily convex) minimize f 0 (x) subject to f i (x) 0, i = 1,..., m h i (x) = 0, i = 1,..., p variable x R n, domain D, optimal value p Lagrangian: L : R n R m R p R, with dom L = D R m R p, L(x, λ, ν) = f 0 (x) + m λ i f i (x) + i=1 p ν i h i (x) weighted sum of objective and constraint functions λ i is Lagrange multiplier associated with f i (x) 0 ν i is Lagrange multiplier associated with h i (x) = 0 i=1 103

106 5. Duality Lagrange dual function Lagrange dual function: g : R m R p R, g(λ, ν) = inf L(x, λ, ν) x D ( m = inf f 0 (x) + λ i f i (x) + x D i=1 ) p ν i h i (x) λ and ν are the dual variables or Lagrange multiplier vectors g is concave since it is defined as pointwise infimum of affine functions, and it can be for some λ, ν lower bound property: if λ 0, then g(λ, ν) p i=1 104 proof: if x is feasible and λ 0, then f 0 ( x) L( x, λ, ν) inf L(x, λ, ν) = g(λ, ν) x D minimizing over all feasible x gives p g(λ, ν)

107 5. Duality Least-norm solution of linear equations dual function minimize subject to x T x Ax = b Lagrangian is L(x, ν) = x T x + ν T (Ax b) to minimize L over x, set gradient equal to zero: x L(x, ν) = 2x + A T ν = 0 x = (1/2)A T ν plug in L to obtain g: g(ν) = L(( 1/2)A T ν, ν) = 1 4 νt AA T ν b T ν a concave function of ν 105 lower bound property: p (1/4)ν T AA T ν b T ν for all ν

108 5. Duality Standard form LP dual function Lagrangian is minimize c T x subject to Ax = b, x 0 L(x, λ, ν) = c T x + ν T (Ax b) λ T x L is affine in x, hence g(λ, ν) = inf L(x, λ, ν) = x = b T ν + (c + A T ν λ) T x { b T ν A T ν λ + c = 0 otherwise g is linear on affine domain {(λ, ν) A T ν λ + c = 0}, hence concave lower bound property: p b T ν if A T ν + c 0 106

109 5. Duality The dual problem Lagrange dual problem maximize g(λ, ν) subject to λ 0 finds best lower bound on p, obtained from Lagrange dual function a convex optimization problem, also if original problem is not! optimal value denoted d λ, ν are dual feasible if λ 0, (λ, ν) dom g often simplified by making implicit constraint (λ, ν) dom g explicit example: standard form LP and its dual minimize c T x maximize b T ν subject to Ax = b subject to A T ν + c 0 x 0 107

110 5. Duality Weak and strong duality weak duality: d p always holds (for convex and nonconvex problems) can be used to find nontrivial lower bounds for difficult problems strong duality: d = p does not hold in general (usually) holds for convex problems conditions that guarantee strong duality in convex problems are called constraint qualifications 108

111 5. Duality Slater s constraint qualification strong duality holds for a convex problem minimize f 0 (x) subject to f i (x) 0, i = 1,..., m Ax = b if it is strictly feasible, i.e., x int D : f i (x) < 0, i = 1,..., m, Ax = b also guarantees that the dual optimum is attained (if p > ) can be sharpened: e.g., can replace int D with relint D (interior relative to affine hull); linear inequalities do not need to hold with strict inequality,... there exist many other types of constraint qualifications 109

112 5. Duality Inequality form LP primal problem minimize subject to c T x Ax b dual function g(λ) = inf x dual problem ( (c + A T λ) T x b T λ ) { b T λ A T λ + c = 0 = otherwise maximize b T λ subject to A T λ + c = 0, λ 0 from Slater s condition: p = d if A x b for some x in fact, p = d except when primal and dual are infeasible 110

113 5. Duality Quadratic program primal problem (assume P S n ++) dual function g(λ) = inf x maximize subject to ( x T Px + λ T (Ax b) x T Px Ax b ) = 1 4 λt AP 1 A T λ b T λ (infimum is attained at x for which 2 P x = A T λ x = 1 2 P 1 A T λ) dual problem maximize (1/4)λ T AP 1 A T λ b T λ subject to λ 0 from Slater s condition: p = d if A x b for some x in fact, p = d always 111

114 5. Duality Geometric interpretation for simplicity, consider problem with one constraint f 1 (x) 0 interpretation of dual function: g(λ) = t + λu = [ λ inf (t + λu), where G = {(f 1(x), f 0 (x) x D} (u,t) G 1 ] [ u t ] 112

115 5. Duality λu + t = g(λ) is (non-vertical) supporting hyperplane to G hyperplane intersects t-axis at t = g(λ) 113

116 5. Duality Complementary slackness I assume strong duality holds, x is primal optimal, (λ, ν ) is dual optimal ( ) f 0 (x ) = g(λ, ν ) = inf x D f 0 (x ) + f 0 (x ) f 0 (x) + m λi f i(x) + i=1 m λi f i(x ) + i=1 (λ i 0, f i (x ) 0, λ i f i(x ) 0, h i (x ) = 0) p νi h i(x) i=1 p νi h i(x ) i=1 114

117 5. Duality Complementary slackness II hence, the two inequalities hold with equality x minimizes L(x, λ, ν ) λ i f i(x ) = 0 for i = 1,..., m (known as complementary slackness): λ i > 0 f i (x ) = 0, f i (x ) < 0 λ i = 0 ith Lagrange multiplier λ i can be only nonzero if ith constraint is active at optimum 115

118 5. Duality Karush-Kuhn-Tucker (KKT) conditions the following four conditions are called KKT conditions (for a problem with differentiable f i, h i, D = R n ): 1 primal constraints: f i (x) 0, i = 1,..., m, h i (x) = 0, i = 1,..., p 2 dual constraints: λ 0 3 complementary slackness: λ i f i (x) = 0, i = 1,..., m 4 gradient of Lagrangian with respect to x vanishes: m p f 0 (x) + λ i f i (x) + ν i h i (x) = 0 i=1 from previous considerations: if strong duality holds and x, λ, ν are optimal, then they must satisfy the KKT conditions i=1 116

119 5. Duality KKT conditions for convex problem I if x, λ, ν satisfy KKT for a convex problem, then they are optimal: from complementary slackness: f 0 ( x) = L( x, λ, ν) from 4th condition (and convexity): g( λ, ν) = L( x, λ, ν) (L( x, λ, ν) is convex in x and differentiable) hence, f 0 ( x) = g( λ, ν) (g( λ, ν) = L( x, λ, ν) = f 0 ( x) + m λ i f i ( x) + i=1 g( λ, ν) = f 0 ( x) p p = f 0 ( x) p ν i h i ( x) = f 0 ( x)) i=1 117

120 5. Duality KKT conditions for convex problem II if Slater s condition is satisfied: x is optimal if and only if there exist λ, ν that satisfy KKT conditions recall that Slater implies strong duality, and dual optimum is attained generalizes optimality condition f 0 (x) = 0 for unconstrained problem 118

121 5. Duality example: water-filling (assume α i > 0) for signal transmission over n independent subchannels with signal power x i and noise power α i, the total channel capacity is C = n i=1 log(1+x i/α i ) = n i=1 (log(1+x i/α i )+log(α i ))+const. hence, for maximization of the channel capacity under a total power constraint the following optimization problem results minimize n i=1 log(x i + α i ) subject to x 0, 1 T x = 1 x is optimal iff x 0, 1 T x = 1, and there exist λ R n, ν R such that λ 0, λ i x i = 0, 1 x i + α i + λ i = ν 119

122 5. Duality if ν < 1/α i : λ i = 0 and x i = 1/ν α i (x i = 0 would cause negative λ i ) if ν 1/α i : λ i = ν 1/α i and x i = 0 (x i > 0 would cause λ i > 0 violating complementary slackness) combining the above two results gives x i = max{0, 1/ν α i } determine ν from 1 T x = n i=1 max{0, 1/ν α i} = 1 interpretation n patches; level of patch i is at height α i flood area with unit amount of water resulting level is 1/ν 120

123 5. Duality Duality and problem reformulations equivalent formulations of a problem can lead to very different duals reformulating the primal problem can be useful when the dual is difficult to derive, or uninteresting common reformulations introduce new variables and equality constraints make explicit constraints implicit or vice-versa transform objective or constraint functions e.g., replace f 0 (x) by φ(f 0 (x)) with φ convex, increasing 121

124 5. Duality Semidefinite program primal SDP (F i, G S k ) minimize subject to c T x x 1 F x n F n G Lagrange multiplier is matrix Z S k Lagrangian L(x, Z) = c T x + tr(z(x 1 F x n F n G)) = c 1 x c n x n + tr(z F 1 ) x tr(z F n ) x n tr(z G) (scalar product between two symmetric matrices A and B: tr(a B)) dual function g(z) = inf L(x, Z) = x { tr(gz) tr(f i Z) + c i = 0, i = 1,..., n otherwise 122

125 5. Duality dual SDP maximize tr(gz) subject to Z 0, tr(f i Z) + c i = 0, i = 1,..., n p = d if primal SDP is strictly feasible ( x with x 1 F x n F n G) 123

126 6. Applications in communications Outline 5 Duality 6 Applications in communications 7 Algorithms Equality constrained minimization 124

127 6. Applications in communications Examples for applications in communications downlink beamforming uplink-downlink duality multiuser detection 125

128 6. Applications in communications Downlink beamforming problem [Luo et al. 2006] transmitter beamforming problem in downlink of wireless communications base station is equipped with multiple antennas and each of K mobile terminal with a single antenna block diagram for K = 2 in equivalent complex baseband: Base station Mobile terminals z 1 ~N (0,σ 2 ) u 1 w 1 h 1 H y 1 x z 2 ~N (0,σ 2 ) u 2 w 2 h 2 H y 2 126

129 6. Applications in communications w i : transmit beamforming vector for ith user (N T 1; N T : number of transmit antennas) u i : information symbol of ith user of variance E{ u i 2 } = 1 x: transmit vector of base station in current time step (N T 1) 127

130 6. Applications in communications received signal of ith user: y i = h H i x + z i, i = 1,..., K h i : channel vector of ith user (µth entry (µ = 1,..., N T ): overall (conjugate) channel coefficient from µth transmit antenna of BS to user i) h i is assumed to be known at BS and receiver z i : i.i.d. additive complex Gaussian noise of variance σ 2 y i is a scalar (single antenna receivers) 128

131 6. Applications in communications transmit vector of beamformer in the BS: x = K u i w i i=1 representation for received signal at lth terminal: ( K ) y l = h H l u i w i + z l, l = 1,..., K i=1 = u l h H l w l + K i=1,i l u i h H l w i + z l signal-to-interference-and-noise ratio (SINR) of lth user: h H l SINR l = w l 2 k l hh l w k 2 + σ 2 129

132 6. Applications in communications design criterion for beamforming vectors: minimization of total transmit power, while satisfying a given set of SINR constraints γ l for the users (assuming the set is feasible) minimize subject to E{x H x} = E{ x 2 2 } = K i=1 w i 2 2 w H l h lh H l w l k l wh k h lh H l w k + σ 2 γ l l however: SINR constraint is not convex not a convex optimization problem, but one that can be relaxed or transformed into a convex problem 130

133 6. Applications in communications Relaxation approach reformulation define B i = w i w H i (N T N T positive semidef. matrix) H i = h i h H i B i is of rank 1 (dyadic product) minimize subject to K i=1 tr(b i) tr(h l B l ) γ l tr(h l B k ) γ l σ 2 k l B i 0, B i Hermitian, rank (B i ) = 1 l i 131

134 6. Applications in communications dropping the rank-1 constraint results in a convex semidefinite programming (SDP) problem (SDP relaxation) it can be shown that the SDP relaxation is guaranteed to have at least one optimal solution with rank 1 it can be used to optimally solve to original, nonconvex problem 132

135 6. Applications in communications Transformation into a convex problem observe that an arbitrary phase rotation can be added to the beamforming vectors without affecting the transmit power or the constraints h H l w l can be chosen to be real, 0 without any loss in generality l constraints: (1 + 1 γ l ) h H l w l 2 K k=1 hh l w k 2 + σ 2 l 133

136 6. Applications in communications define following vector and matrix [ ] T w = w T 1 wt 2... wt K h H l h H l... 0 H l = h H l h H l w 1 h H l w 2 H l w =. h H l w K constraint can be written written as (1 + 1 ) [ ] h H l γ w l 2 Hl w 2 l σ 2 134

137 6. Applications in communications h H l w l can be assumed to be real, non-negative taking the square root yields: [ ] γ l h H l w l Hl w l σ or Aw + b 2 c H w + d 0 [ ] Hl with A = 0 T ; b =. 0 σ c H = [ 0 T 0 T... h H l 0 T... 0 T ] γ l ; d = 0 second-order cone constraint! original optimization problem is equivalent to the following second-order cone program 2 135

138 6. Applications in communications SOCP minimize subject to τ [ γ l h H l w l Hl w σ w 2 τ ] 2 l K (this minimizes i=1 w i 2 2 under the constraints; ( ) is a monotonically increasing function and does not change the optimum solution for the beamforming filters) 136

139 6. Applications in communications Uplink-downlink duality via Lagrangian duality [Luo et al. 2006] exploring the dual of convex optimization problems in engineering often reveals the structure of the optimum solution Lagrangian dual of the above SOCP problem has an engineering interpretation, which is known as uplink-downlink duality several different versions of uplink-downlink duality have been developed in the literature, referring to different figures of merit, e.g. channel capacity 137

140 6. Applications in communications duality in beamforming context: minimum power needed to achieve a set of SINR targets in a downlink multiple-input multiple-output (MIMO) channel is the same as the minimum power needed to achieve the same set of SINR targets in the uplink channel, where the uplink channel is derived by reversing the input and output of the downlink these results can be proven via Lagrangian duality in convex optimization in a unified manner 138

141 6. Applications in communications Lagrangian of the downlink beamforming problem: K K L(w, λ)= w H i w i + h H i w k 2 + σ 2 = i=1 K λ i σ 2 + i=1 i=1 K i=1 λ=[λ 1, λ 2,..., λ K ] T λ i w H i 1 γ i h H i w i 2 + k i Lagrangian dual function: g(λ) = I λ i γ i h i h H i inf L(w, λ) w C N T K + µ i λ µ h µ h H µ w i 139

142 6. Applications in communications K λ g(λ) = i σ 2 i=1 if ( else I λ i γ i h i h H i + µ i λ µ h µ h H µ ) 0 i dual problem (SDP): maximize subject to K λ i σ 2 i=1 I + K λ µ h µ h H µ µ=1 (1 + 1 γi ) λ i h i h H i i 140

143 6. Applications in communications dual problem can be shown to correspond to an uplink problem with λ i as the the (scaled) uplink power, h i as the uplink channel, and γ i as the SINR constraint block diagram of uplink transmission (K = 2) Mobile terminals Base station x 1 h 1 z~n (0,σ 2 I) w 1 H ỹ 1 x 2 h 2 w 2 H ỹ 2 w i : receiver filter vector of base station for ith user (N 1) x i : information symbol of ith user of variance E{ x i 2 } = ρ i 141

144 6. Applications in communications h i : channel vector of ith user (µth entry: overall channel coefficient from user i to µth receive antenna of BS); N 1 z: i.i.d. additive complex Gaussian noise vector with covariance matrix σ 2 I received signal for user l after filtering at BS: ỹ l = w H l ( K ) x i h i + z i=1 = x l w H l h l + K i=1,i l l = 1,..., K x i w H l h i + w H l z 142

145 6. Applications in communications SINR of lth user: SINR l = ρ l w H l h l 2 ρ k w H l h k 2 + σ 2 w H l k l w l design criterion for receiver vectors: minimize total sum transmit power of mobile terminals, while satisfying a given set of SINR constraints γ l for the users minimize subject to K ρ i i=1 ρ l w H l h lh H l w l ρ k w H l h kh H k w l + σ 2 w H γ l l l w l k l 143

146 6. Applications in communications optimal w i is the minimum mean-squared error (MMSE) filter ( K 1 w i = ρ k h k h H k I) + σ 2 h i k=1 substituting the MMSE filters into the constraints and performing some matrix manipulations, it can be shown that the uplink problem is equivalent to minimize subject to K ρ i i=1 ) (1 + 1 ρ γi i h i h H i σ 2 I + K ρ µ h µ h H µ µ=1 i 144

147 6. Applications in communications with λ i = ρ i σ 2, the dual downlink problem and the uplink problem are identical, except that maximization and minimization are reversed and right hand side and left hand side in the constraint inequalities have been interchanged finally, both problems turn out to have the same solution strong duality holds for original and dual downlink problem (convex optimization problem) original and dual problem must have the same solution solution of dual uplink problem solves also original downlink problem 145

148 6. Applications in communications dual variables of the downlink problem can be interpreted as uplink powers scaled by the noise variance there is a quite general uplink-downlink duality which is useful because the uplink problem is easier to solve (e.g., iteratively) 146

149 6. Applications in communications Multiuser detection [Luo et al. 2006] consider a multiple-input multiple-output (MIMO) transmission with received vector ρ y = n Hs + z ρ: total signal-to-noise ratio, SNR H: n m channel matrix entry (ν, µ): channel coefficient from µth transmit antenna to νth receive antenna m: number of transmit antennas n: number of receive antennas s: transmitted symbol vector with BPSK entries s { 1, +1} m z: additive complex Gaussian noise vector with i.i.d. entries 147

CSCI : Optimization and Control of Networks. Review on Convex Optimization

CSCI : Optimization and Control of Networks. Review on Convex Optimization CSCI7000-016: Optimization and Control of Networks Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one

More information

A Brief Review on Convex Optimization

A Brief Review on Convex Optimization A Brief Review on Convex Optimization 1 Convex set S R n is convex if x,y S, λ,µ 0, λ+µ = 1 λx+µy S geometrically: x,y S line segment through x,y S examples (one convex, two nonconvex sets): A Brief Review

More information

Convex Optimization M2

Convex Optimization M2 Convex Optimization M2 Lecture 3 A. d Aspremont. Convex Optimization M2. 1/49 Duality A. d Aspremont. Convex Optimization M2. 2/49 DMs DM par email: dm.daspremont@gmail.com A. d Aspremont. Convex Optimization

More information

5. Duality. Lagrangian

5. Duality. Lagrangian 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Convex Optimization Boyd & Vandenberghe. 5. Duality

Convex Optimization Boyd & Vandenberghe. 5. Duality 5. Duality Convex Optimization Boyd & Vandenberghe Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Lecture: Duality.

Lecture: Duality. Lecture: Duality http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/35 Lagrange dual problem weak and strong

More information

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions 3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions

More information

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions

3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions 3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions

More information

4. Convex optimization problems

4. Convex optimization problems Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

More information

Convex optimization problems. Optimization problem in standard form

Convex optimization problems. Optimization problem in standard form Convex optimization problems optimization problem in standard form convex optimization problems linear optimization quadratic optimization geometric programming quasiconvex optimization generalized inequality

More information

Lecture: Convex Optimization Problems

Lecture: Convex Optimization Problems 1/36 Lecture: Convex Optimization Problems http://bicmr.pku.edu.cn/~wenzw/opt-2015-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghe s lecture notes Introduction 2/36 optimization

More information

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization

Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

More information

Optimisation convexe: performance, complexité et applications.

Optimisation convexe: performance, complexité et applications. Optimisation convexe: performance, complexité et applications. Introduction, convexité, dualité. A. d Aspremont. M2 OJME: Optimisation convexe. 1/128 Today Convex optimization: introduction Course organization

More information

Lecture: Duality of LP, SOCP and SDP

Lecture: Duality of LP, SOCP and SDP 1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:

More information

4. Convex optimization problems

4. Convex optimization problems Convex Optimization Boyd & Vandenberghe 4. Convex optimization problems optimization problem in standard form convex optimization problems quasiconvex optimization linear optimization quadratic optimization

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

Constrained Optimization and Lagrangian Duality

Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

More information

EE/AA 578, Univ of Washington, Fall Duality

EE/AA 578, Univ of Washington, Fall Duality 7. Duality EE/AA 578, Univ of Washington, Fall 2016 Lagrange dual problem weak and strong duality geometric interpretation optimality conditions perturbation and sensitivity analysis examples generalized

More information

Convex Optimization & Lagrange Duality

Convex Optimization & Lagrange Duality Convex Optimization & Lagrange Duality Chee Wei Tan CS 8292 : Advanced Topics in Convex Optimization and its Applications Fall 2010 Outline Convex optimization Optimality condition Lagrange duality KKT

More information

Tutorial on Convex Optimization for Engineers Part II

Tutorial on Convex Optimization for Engineers Part II Tutorial on Convex Optimization for Engineers Part II M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de

More information

Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)

Lagrange Duality. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Lagrange Duality Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Lagrangian Dual function Dual

More information

January 29, Introduction to optimization and complexity. Outline. Introduction. Problem formulation. Convexity reminder. Optimality Conditions

January 29, Introduction to optimization and complexity. Outline. Introduction. Problem formulation. Convexity reminder. Optimality Conditions Olga Galinina olga.galinina@tut.fi ELT-53656 Network Analysis Dimensioning II Department of Electronics Communications Engineering Tampere University of Technology, Tampere, Finl January 29, 2014 1 2 3

More information

Convex Optimization and l 1 -minimization

Convex Optimization and l 1 -minimization Convex Optimization and l 1 -minimization Sangwoon Yun Computational Sciences Korea Institute for Advanced Study December 11, 2009 2009 NIMS Thematic Winter School Outline I. Convex Optimization II. l

More information

CS-E4830 Kernel Methods in Machine Learning

CS-E4830 Kernel Methods in Machine Learning CS-E4830 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 27. September, 2017 Juho Rousu 27. September, 2017 1 / 45 Convex optimization Convex optimisation This

More information

12. Interior-point methods

12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

More information

Lecture 2: Convex functions

Lecture 2: Convex functions Lecture 2: Convex functions f : R n R is convex if dom f is convex and for all x, y dom f, θ [0, 1] f is concave if f is convex f(θx + (1 θ)y) θf(x) + (1 θ)f(y) x x convex concave neither x examples (on

More information

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010

I.3. LMI DUALITY. Didier HENRION EECI Graduate School on Control Supélec - Spring 2010 I.3. LMI DUALITY Didier HENRION henrion@laas.fr EECI Graduate School on Control Supélec - Spring 2010 Primal and dual For primal problem p = inf x g 0 (x) s.t. g i (x) 0 define Lagrangian L(x, z) = g 0

More information

Convex functions. Definition. f : R n! R is convex if dom f is a convex set and. f ( x +(1 )y) < f (x)+(1 )f (y) f ( x +(1 )y) apple f (x)+(1 )f (y)

Convex functions. Definition. f : R n! R is convex if dom f is a convex set and. f ( x +(1 )y) < f (x)+(1 )f (y) f ( x +(1 )y) apple f (x)+(1 )f (y) Convex functions I basic properties and I operations that preserve convexity I quasiconvex functions I log-concave and log-convex functions IOE 611: Nonlinear Programming, Fall 2017 3. Convex functions

More information

Lecture Note 5: Semidefinite Programming for Stability Analysis

Lecture Note 5: Semidefinite Programming for Stability Analysis ECE7850: Hybrid Systems:Theory and Applications Lecture Note 5: Semidefinite Programming for Stability Analysis Wei Zhang Assistant Professor Department of Electrical and Computer Engineering Ohio State

More information

4. Convex optimization problems (part 1: general)

4. Convex optimization problems (part 1: general) EE/AA 578, Univ of Washington, Fall 2016 4. Convex optimization problems (part 1: general) optimization problem in standard form convex optimization problems quasiconvex optimization 4 1 Optimization problem

More information

ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications

ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications ELE539A: Optimization of Communication Systems Lecture 15: Semidefinite Programming, Detection and Estimation Applications Professor M. Chiang Electrical Engineering Department, Princeton University March

More information

Convex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)

Convex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST) Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples

More information

subject to (x 2)(x 4) u,

subject to (x 2)(x 4) u, Exercises Basic definitions 5.1 A simple example. Consider the optimization problem with variable x R. minimize x 2 + 1 subject to (x 2)(x 4) 0, (a) Analysis of primal problem. Give the feasible set, the

More information

Tutorial on Convex Optimization: Part II

Tutorial on Convex Optimization: Part II Tutorial on Convex Optimization: Part II Dr. Khaled Ardah Communications Research Laboratory TU Ilmenau Dec. 18, 2018 Outline Convex Optimization Review Lagrangian Duality Applications Optimal Power Allocation

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Duality Theory and Optimality Conditions 5th lecture, 12.05.2010 Jun.-Prof. Matthias Hein Program of today/next lecture Lagrangian and duality: the Lagrangian the dual

More information

Convex Functions. Pontus Giselsson

Convex Functions. Pontus Giselsson Convex Functions Pontus Giselsson 1 Today s lecture lower semicontinuity, closure, convex hull convexity preserving operations precomposition with affine mapping infimal convolution image function supremum

More information

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient

Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian

More information

A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution:

A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution: 1-5: Least-squares I A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution: f (x) =(Ax b) T (Ax b) =x T A T Ax 2b T Ax + b T b f (x) = 2A T Ax 2A T b = 0 Chih-Jen Lin (National

More information

Introduction to Convex Optimization

Introduction to Convex Optimization Introduction to Convex Optimization Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Optimization

More information

Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications

Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications Convex Optimization for Signal Processing and Communications: From Fundamentals to Applications Chong-Yung Chi Institute of Communications Engineering & Department of Electrical Engineering National Tsing

More information

Subgradient. Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes. definition. subgradient calculus

Subgradient. Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes. definition. subgradient calculus 1/41 Subgradient Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes definition subgradient calculus duality and optimality conditions directional derivative Basic inequality

More information

Subgradients. subgradients and quasigradients. subgradient calculus. optimality conditions via subgradients. directional derivatives

Subgradients. subgradients and quasigradients. subgradient calculus. optimality conditions via subgradients. directional derivatives Subgradients subgradients and quasigradients subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE392o, Stanford University Basic inequality recall basic

More information

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.

HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given. HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

More information

Course Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979)

Course Outline. FRTN10 Multivariable Control, Lecture 13. General idea for Lectures Lecture 13 Outline. Example 1 (Doyle Stein, 1979) Course Outline FRTN Multivariable Control, Lecture Automatic Control LTH, 6 L-L Specifications, models and loop-shaping by hand L6-L8 Limitations on achievable performance L9-L Controller optimization:

More information

Convex Optimization & Machine Learning. Introduction to Optimization

Convex Optimization & Machine Learning. Introduction to Optimization Convex Optimization & Machine Learning Introduction to Optimization mcuturi@i.kyoto-u.ac.jp CO&ML 1 Why do we need optimization in machine learning We want to find the best possible decision w.r.t. a problem

More information

Exercises. Exercises. Basic terminology and optimality conditions. 4.2 Consider the optimization problem

Exercises. Exercises. Basic terminology and optimality conditions. 4.2 Consider the optimization problem Exercises Basic terminology and optimality conditions 4.1 Consider the optimization problem f 0(x 1, x 2) 2x 1 + x 2 1 x 1 + 3x 2 1 x 1 0, x 2 0. Make a sketch of the feasible set. For each of the following

More information

Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities

Duality. Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Duality Lagrange dual problem weak and strong duality optimality conditions perturbation and sensitivity analysis generalized inequalities Lagrangian Consider the optimization problem in standard form

More information

ICS-E4030 Kernel Methods in Machine Learning

ICS-E4030 Kernel Methods in Machine Learning ICS-E4030 Kernel Methods in Machine Learning Lecture 3: Convex optimization and duality Juho Rousu 28. September, 2016 Juho Rousu 28. September, 2016 1 / 38 Convex optimization Convex optimisation This

More information

The Q-parametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Spring-mass System

The Q-parametrization (Youla) Lecture 13: Synthesis by Convex Optimization. Lecture 13: Synthesis by Convex Optimization. Example: Spring-mass System The Q-parametrization (Youla) Lecture 3: Synthesis by Convex Optimization controlled variables z Plant distubances w Example: Spring-mass system measurements y Controller control inputs u Idea for lecture

More information

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17

EE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 17 EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 17 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory May 29, 2012 Andre Tkacenko

More information

Convex Optimization and Modeling

Convex Optimization and Modeling Convex Optimization and Modeling Convex Optimization Fourth lecture, 05.05.2010 Jun.-Prof. Matthias Hein Reminder from last time Convex functions: first-order condition: f(y) f(x) + f x,y x, second-order

More information

Lecture 6: Conic Optimization September 8

Lecture 6: Conic Optimization September 8 IE 598: Big Data Optimization Fall 2016 Lecture 6: Conic Optimization September 8 Lecturer: Niao He Scriber: Juan Xu Overview In this lecture, we finish up our previous discussion on optimality conditions

More information

CS295: Convex Optimization. Xiaohui Xie Department of Computer Science University of California, Irvine

CS295: Convex Optimization. Xiaohui Xie Department of Computer Science University of California, Irvine CS295: Convex Optimization Xiaohui Xie Department of Computer Science University of California, Irvine Course information Prerequisites: multivariate calculus and linear algebra Textbook: Convex Optimization

More information

Geometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as

Geometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.

More information

A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution:

A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution: 1-5: Least-squares I A : k n. Usually k > n otherwise easily the minimum is zero. Analytical solution: f (x) =(Ax b) T (Ax b) =x T A T Ax 2b T Ax + b T b f (x) = 2A T Ax 2A T b = 0 Chih-Jen Lin (National

More information

FRTN10 Multivariable Control, Lecture 13. Course outline. The Q-parametrization (Youla) Example: Spring-mass System

FRTN10 Multivariable Control, Lecture 13. Course outline. The Q-parametrization (Youla) Example: Spring-mass System FRTN Multivariable Control, Lecture 3 Anders Robertsson Automatic Control LTH, Lund University Course outline The Q-parametrization (Youla) L-L5 Purpose, models and loop-shaping by hand L6-L8 Limitations

More information

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem:

Motivation. Lecture 2 Topics from Optimization and Duality. network utility maximization (NUM) problem: CDS270 Maryam Fazel Lecture 2 Topics from Optimization and Duality Motivation network utility maximization (NUM) problem: consider a network with S sources (users), each sending one flow at rate x s, through

More information

12. Interior-point methods

12. Interior-point methods 12. Interior-point methods Convex Optimization Boyd & Vandenberghe inequality constrained minimization logarithmic barrier function and central path barrier method feasibility and phase I methods complexity

More information

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014

Convex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014 Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,

More information

Lagrangian Duality and Convex Optimization

Lagrangian Duality and Convex Optimization Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?

More information

The proximal mapping

The proximal mapping The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction ESE 605 Modern Convex Optimization mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

Subgradients. subgradients. strong and weak subgradient calculus. optimality conditions via subgradients. directional derivatives

Subgradients. subgradients. strong and weak subgradient calculus. optimality conditions via subgradients. directional derivatives Subgradients subgradients strong and weak subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE364b, Stanford University Basic inequality recall basic inequality

More information

Convex Functions. Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK)

Convex Functions. Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK) Convex Functions Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK) Course on Convex Optimization for Wireless Comm. and Signal Proc. Jointly taught by Daniel P. Palomar and Wing-Kin (Ken) Ma

More information

1. Gradient method. gradient method, first-order methods. quadratic bounds on convex functions. analysis of gradient method

1. Gradient method. gradient method, first-order methods. quadratic bounds on convex functions. analysis of gradient method L. Vandenberghe EE236C (Spring 2016) 1. Gradient method gradient method, first-order methods quadratic bounds on convex functions analysis of gradient method 1-1 Approximate course outline First-order

More information

Appendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS

Appendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS Appendix PRELIMINARIES 1. THEOREMS OF ALTERNATIVES FOR SYSTEMS OF LINEAR CONSTRAINTS Here we consider systems of linear constraints, consisting of equations or inequalities or both. A feasible solution

More information

Solution to EE 617 Mid-Term Exam, Fall November 2, 2017

Solution to EE 617 Mid-Term Exam, Fall November 2, 2017 Solution to EE 67 Mid-erm Exam, Fall 207 November 2, 207 EE 67 Solution to Mid-erm Exam - Page 2 of 2 November 2, 207 (4 points) Convex sets (a) (2 points) Consider the set { } a R k p(0) =, p(t) for t

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction Convex Optimization Boyd & Vandenberghe mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history

More information

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics

1. Introduction. mathematical optimization. least-squares and linear programming. convex optimization. example. course goals and topics 1. Introduction mathematical optimization least-squares and linear programming convex optimization example course goals and topics nonlinear optimization brief history of convex optimization 1 1 Mathematical

More information

On the Method of Lagrange Multipliers

On the Method of Lagrange Multipliers On the Method of Lagrange Multipliers Reza Nasiri Mahalati November 6, 2016 Most of what is in this note is taken from the Convex Optimization book by Stephen Boyd and Lieven Vandenberghe. This should

More information

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44

Convex Optimization. Newton s method. ENSAE: Optimisation 1/44 Convex Optimization Newton s method ENSAE: Optimisation 1/44 Unconstrained minimization minimize f(x) f convex, twice continuously differentiable (hence dom f open) we assume optimal value p = inf x f(x)

More information

IOE 611/Math 663: Nonlinear Programming

IOE 611/Math 663: Nonlinear Programming 1. Introduction I course logistics I mathematical optimization I least-squares and linear programming I convex optimization I example I course goals and topics I nonlinear optimization I brief history

More information

A function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3

A function(al) f is convex if dom f is a convex set, and. f(θx + (1 θ)y) < θf(x) + (1 θ)f(y) f(x) = x 3 Convex functions The domain dom f of a functional f : R N R is the subset of R N where f is well-defined. A function(al) f is convex if dom f is a convex set, and f(θx + (1 θ)y) θf(x) + (1 θ)f(y) for all

More information

Linear and non-linear programming

Linear and non-linear programming Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)

More information

CO 250 Final Exam Guide

CO 250 Final Exam Guide Spring 2017 CO 250 Final Exam Guide TABLE OF CONTENTS richardwu.ca CO 250 Final Exam Guide Introduction to Optimization Kanstantsin Pashkovich Spring 2017 University of Waterloo Last Revision: March 4,

More information

Lecture 18: Optimization Programming

Lecture 18: Optimization Programming Fall, 2016 Outline Unconstrained Optimization 1 Unconstrained Optimization 2 Equality-constrained Optimization Inequality-constrained Optimization Mixture-constrained Optimization 3 Quadratic Programming

More information

CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.

CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008. 1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function

More information

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research

Introduction to Machine Learning Lecture 7. Mehryar Mohri Courant Institute and Google Research Introduction to Machine Learning Lecture 7 Mehryar Mohri Courant Institute and Google Research mohri@cims.nyu.edu Convex Optimization Differentiation Definition: let f : X R N R be a differentiable function,

More information

Lecture 7: Convex Optimizations

Lecture 7: Convex Optimizations Lecture 7: Convex Optimizations Radu Balan, David Levermore March 29, 2018 Convex Sets. Convex Functions A set S R n is called a convex set if for any points x, y S the line segment [x, y] := {tx + (1

More information

Lagrange duality. The Lagrangian. We consider an optimization program of the form

Lagrange duality. The Lagrangian. We consider an optimization program of the form Lagrange duality Another way to arrive at the KKT conditions, and one which gives us some insight on solving constrained optimization problems, is through the Lagrange dual. The dual is a maximization

More information

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem

Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R

More information

minimize x subject to (x 2)(x 4) u,

minimize x subject to (x 2)(x 4) u, Math 6366/6367: Optimization and Variational Methods Sample Preliminary Exam Questions 1. Suppose that f : [, L] R is a C 2 -function with f () on (, L) and that you have explicit formulae for

More information

N. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form:

N. L. P. NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP. Optimization. Models of following form: 0.1 N. L. P. Katta G. Murty, IOE 611 Lecture slides Introductory Lecture NONLINEAR PROGRAMMING (NLP) deals with optimization models with at least one nonlinear function. NLP does not include everything

More information

Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version

Convex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

More information

Chapter 2: Preliminaries and elements of convex analysis

Chapter 2: Preliminaries and elements of convex analysis Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-14-15.shtml Academic year 2014-15

More information

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness.

14. Duality. ˆ Upper and lower bounds. ˆ General duality. ˆ Constraint qualifications. ˆ Counterexample. ˆ Complementary slackness. CS/ECE/ISyE 524 Introduction to Optimization Spring 2016 17 14. Duality ˆ Upper and lower bounds ˆ General duality ˆ Constraint qualifications ˆ Counterexample ˆ Complementary slackness ˆ Examples ˆ Sensitivity

More information

Optimization and Optimal Control in Banach Spaces

Optimization and Optimal Control in Banach Spaces Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,

More information

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications

ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications ELE539A: Optimization of Communication Systems Lecture 6: Quadratic Programming, Geometric Programming, and Applications Professor M. Chiang Electrical Engineering Department, Princeton University February

More information

Optimality Conditions for Constrained Optimization

Optimality Conditions for Constrained Optimization 72 CHAPTER 7 Optimality Conditions for Constrained Optimization 1. First Order Conditions In this section we consider first order optimality conditions for the constrained problem P : minimize f 0 (x)

More information

Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings

Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

More information

Optimality, Duality, Complementarity for Constrained Optimization

Optimality, Duality, Complementarity for Constrained Optimization Optimality, Duality, Complementarity for Constrained Optimization Stephen Wright University of Wisconsin-Madison May 2014 Wright (UW-Madison) Optimality, Duality, Complementarity May 2014 1 / 41 Linear

More information

Primal/Dual Decomposition Methods

Primal/Dual Decomposition Methods Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients

More information

Optimization for Machine Learning

Optimization for Machine Learning Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html

More information

Semidefinite Programming Basics and Applications

Semidefinite Programming Basics and Applications Semidefinite Programming Basics and Applications Ray Pörn, principal lecturer Åbo Akademi University Novia University of Applied Sciences Content What is semidefinite programming (SDP)? How to represent

More information

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima

Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

More information

EE 227A: Convex Optimization and Applications October 14, 2008

EE 227A: Convex Optimization and Applications October 14, 2008 EE 227A: Convex Optimization and Applications October 14, 2008 Lecture 13: SDP Duality Lecturer: Laurent El Ghaoui Reading assignment: Chapter 5 of BV. 13.1 Direct approach 13.1.1 Primal problem Consider

More information

Lecture 15 Newton Method and Self-Concordance. October 23, 2008

Lecture 15 Newton Method and Self-Concordance. October 23, 2008 Newton Method and Self-Concordance October 23, 2008 Outline Lecture 15 Self-concordance Notion Self-concordant Functions Operations Preserving Self-concordance Properties of Self-concordant Functions Implications

More information

Semidefinite Programming

Semidefinite Programming Semidefinite Programming Notes by Bernd Sturmfels for the lecture on June 26, 208, in the IMPRS Ringvorlesung Introduction to Nonlinear Algebra The transition from linear algebra to nonlinear algebra has

More information