# In English, this means that if we travel on a straight line between any two points in C, then we never leave C.

Save this PDF as:

Size: px
Start display at page:

Download "In English, this means that if we travel on a straight line between any two points in C, then we never leave C." ## Transcription

1 Convex sets In this section, we will be introduced to some of the mathematical fundamentals of convex sets. In order to motivate some of the definitions, we will look at the closest point problem from several different angles. The tools and concepts we develop here, however, have many other applications both in this course and beyond. A set C R N is convex if x, y C (1 θ)x + θy C for all θ [0, 1]. In English, this means that if we travel on a straight line between any two points in C, then we never leave C. These sets in R are convex: These sets are not: 1

2 Examples of convex (and nonconvex) sets Subspaces. Recall that if S is a subspace of R N, then x, y S ax + by S for all a, b R. So S is clearly convex. Affine sets. Affine sets are just subspaces that have been offset by the origin: {x R N : x = y + v, y T }, T = subspace, for some fixed vector v. An equivalent definition is that x, y C θx + (1 θ)y C for all θ R the difference between this definition and that for a subspace is that subspaces must include the origin. Bound constraints. Rectangular sets of the form C = {x R N : l 1 x 1 u 1, l x u,..., l N x N u N } for some l 1,..., l N, u 1,..., u N R are convex. The simplex in R N {x R N : x 1 + x + + x N 1, x 1, x,..., x N 0} is convex. Any subset of R N that can be expressed as a set of linear inequality constraints {x R N : Ax b} is convex. Notice that both rectangular sets and the simplex

3 fall into this category for the simplex, take A = , b = In general, when sets like these are bounded, the result is a polyhedron. Norm balls. If is a valid norm on R N, then is a convex set. B r = {x : x r}, Ellipsoids. An ellipsoid is a set of the form E = {x : (x x 0 ) T P 1 (x x 0 ) r}, for a symmetric positive-definite matrix P. Geometrically, the ellipsoid is centered at x 0, its axes are oriented with the eigenvectors of P, and the relative widths along these axes are proportional to the eigenvalues of P. A single point {x} is convex. The empty set is convex. The set is convex. (Sketch it!) The set is not convex. {x R : x 1 x 1 x + 1 0} {x R : x 1 x 1 x + 1 0} 3

4 The set is certainly not convex. {x R : x 1 x 1 x + 1 = 0} Sets defined by linear equality constraints where only some of the constraints have to hold are in general not convex. For example {x R : x 1 x 1 and x 1 + x 1} is convex, while is not convex. {x R : x 1 x 1 or x 1 + x 1} Cones A cone is a set C such that x C θx C for all θ 0. Convex cones are sets which are both convex and a cone. C is a convex cone if x 1, x C θ 1 x 1 + θ x C for all θ 1, θ 0. Given an x 1, x, the set of all linear combinations with positive weights makes a wedge. For practice, sketch the region below that consists of all such combinations of x 1 and x : 4

5 x x 1 We will mostly be interested in proper cones, which in addition to being convex, are closed, have a non-empty interior 1 ( solid ), and do not contain entire lines ( pointed ). Examples: Non-negative orthant. The set of vectors whose entries are nonnegative, R N + = {x R N : x n 0, for n = 1,..., N}, is a proper cone. Positive semi-definite cone. The set of N N symmetric matrices with non-negative eigenvalues, S N + = {X R N N : X = V ΛV T, is a proper cone. V T V = V V T = I, Λ diagonal and non-negative} Non-negative polynomials. Vectors of coefficients of non-negative polynomials on [0, 1], {x R N : x 1 +x t+x 3 t + +x N t N 1 0 for all 0 t 1}, 1 See Technical Details for precise definition. 5

6 form a proper cone. Notice that it is not necessary that all the x n 0; for example t t (x 1 = 0, x = 1, x 3 = 1) is non-negative on [0, 1]. Norm cones. The subset of R N+1 defined by {(x, t), x R N, t R : x t} is a proper cone for any valid norm and t > 0. Sketch what this looks like for the Euclidean norm and N = : Every proper cone K defines a partial ordering or generalized inequality. We write x K y when y x K. For example, for vectors x, y R N, we say x R N + y when x n y n for all n = 1,..., N. For symmetric matrices X, Y, we say X S N + Y when Y X has non-negative eigenvalues. We will typically just use when the context makes it clear. In fact, for R N + we will just write x y (as we did above) to mean that 6

7 the entries in x are component-by-component upper-bounded by the entries in y. Partial orderings obey share of the properties of the standard on the real line. For example: x y, u v x + u y + v. But other properties do not hold; for example, it is not necessary that either x y or y x. For an extensive list of properties of partial orderings (most of which will make perfect sense on sight) can be found in [BV04, Chapter.4]. Hyperplanes and halfspaces Hyperplanes and halfspaces are both very simple constructs, but they will be crucial to our understanding to convex sets, functions, and optimization problems. A hyperplane is a set of the form {x R N : x, c = t} for some fixed vector c 0 and scalar t. When t = 0, this set is a subspace of dimension N 1, and contains all vectors that are orthogonal to c. For t 0, this is an affine space consisting of all the vectors orthogonal to c (call this set C ) offset to some x 0 : {x R N : x, c = t} = {x R N : x = x 0 + C }, for any x 0 with x 0, c = t. We might take x 0 = t c/ c, for instance. The point is, c is a normal vector of the set. Here are some examples in R : 7

8 5 5 4 t =1 4 t = apple 1 c = 0 1 apple c = t = 1 t =1 t =5 It should be clear that hyperplanes are convex sets. A halfspace is a set of the form {x R N : x, c t} for some fixed vector c 0 and scalar t. For t = 0, the halfspace contains all vectors whose inner product with c is negative (i.e. the angle between x and c is greater than 90 ). Here is a simple example: 8

9 5 4 t =5 3 apple c = Separating hyperplanes If two convex sets are disjoint, then there is a hyperplane that separates them. C c d D {x : hx, ai = b} a = d c b = kdk kck 9

10 This fact is intuitive, and is incredibly useful in understanding the solutions to convex optimization programs (we will see this even in the next section). It also easy to prove under mild additional assumptions; below we will show how to explicitly construct a separating hyperplane between two convex sets if there are points in the two sets that are definitively closest to one another. The result extends to general convex sets, and the proof is based on the same basic idea, but requires some additional toplogical considerations Let C, D be disjoint convex sets and suppose that there exist points c C and d D that achieve the minimum distance c d = inf x y. x C,y D Define Then a = d c, b = d c. x, a b for x C, x, a b for x D. That is the hyperplane {x : x, a = b} separates C and D. To see this, set f(x) = x, a b, and show that for and point u in D, we have f(u) 0. First, we prove the basic geometric fact that for any two vectors x, y, if x + θy x for all θ [0, 1] then x, y 0. (1) This holds, for example, when the sets are closed, and at least one of them is bounded. 10

11 To establish this, we expand the norm as x + θy = x + θ y + θ x, y, from which we can immediately deduce that θ y + x, y 0 for all θ [0, 1] x, y 0. Now let u be an arbitrary point in D. Since D is convex, we know that d + θ(u d) D for all θ [0, 1]. Since d is as close to c as any other point in D, we have and so by (1), we know that This means d + θ(u d) c d c, d c, u d 0. f(u) = u, d c d c d + c, d c = u, d c = u (d + c)/, d c = u d + d/ c/, d c = u d, d c + c d 0. The proof that for every v C, f(v) 0 is exactly the same. 11

12 Note the following: We actually only need the interiors of sets C and D to be disjoint they can intersect at one or more points along their boundaries. Viz C D But you have to be more careful in defining the hyperplane which separates them (as we would have c = d a = 0 in the argument above). There can be many hyperplanes that separate the two sets, but there is not necessarily more than one. The hyperplane {x : x, a = b} strictly separates sets C and D if x, a < b for all x C and x, a > b for all x D. There are examples of closed, convex, disjoint sets C, D that are separable but not strictly separable. It is a good exercise to think of an example. 1

13 Supporting hyperplanes A direct consequence of the separating hyperplane theorem is that every point on the boundary of a convex set C can be separated from its interior. If a 0 satisfies x, a x 0, a for all x C, then {x : x, a = x 0, a } is called a supporting hyperplane to C at x 0. Here s a picture: x 0 The hyperplane is tangent to C at x 0, and the halfspace {x : x, a x 0, a } contains C. The supporting hyperplane theorem says that a supporting hyperplane exists at every point x 0 on the boundary of a (non-empty) convex set C. Note that there might be more than one: x 0 13

14 Given the separating hyperplane theorem, this is easy to prove. If C has a non-empty interior, then it follows from just realizing that x 0 and int(c) are disjoint. If int(c) =, then C must be contained in an affine set, and hence also in (possibly many) hyperplanes, any of which will be supporting. 14

15 The closest point problem Let x 0 R N be given, and let C be a non-empty, closed, convex set. The projection of x 0 onto C is the closest point (in the standard Euclidean distance, for now) in C to x 0 : P C (x 0 ) = arg min y C x 0 y We will see below that there is a unique minimizer to this problem, and that the solution has geometric properties that are analogous to the case where C is a subspace. Projection onto a subspace Let s recall how we solve this problem in the special case where C := T is a K-dimensional subspace. In this case, the solution ˆx = P T (x 0 ) is unique, and is characterized by the orthogonality principle: x 0 ˆx T meaning that y, x 0 ˆx = 0 for all y T. The proof of this fact is reviewed in the Technical Details section at the end of these notes. The orthogonality principle leads immediately to an algorithm for calculating P T (x 0 ). Let v 1,..., v K be a basis for T ; we can write the solution as K ˆx = α k v k ; k=1 solving for the expansion coefficients α k is the same as solving for ˆx. We know that K x 0 α j v j, v k = 0, for k = 1,..., K, j=1 15

16 and so the α k must obey the linear system of equations K α j v j, v k = x 0, v k, for k = 1,..., K. j=1 Concatenating the v k as columns in the N K matrix V, and the entries α k into the vector α R K, we can write the equations above as V T V α = V T x 0. Since the {v k } are a basis for T (i.e. they are linearly independent), V T V is invertible, and we can solve for the best expansion coefficients: ˆα = (V T V ) 1 V T x 0. Using these expansion coefficient to reconstruct ˆx yields ˆx = V ˆα = V (V T V ) 1 V T x 0. In this case, the projector P T ( ) is a linear map, specified by N N matrix V (V T V ) 1 V T. Projection onto an affine set Affine sets are not fundamentally different than subspaces. Any affine set C can be written as a subspace T plus an offset v 0 : C = T + v 0 = {x : x = y + v 0, y C}. It is easy to translate the results for subspaces above to say that the projection onto an affine set is unique, and obeys the orthogonality principle y ˆx, x 0 ˆx = 0, for all y C. () 16

17 You can solve this problem by shifting x 0 and C by negative v 0, projecting x 0 v 0 onto the subspace C v 0, and then shifting the answer back. Projection onto a general convex set In general, there is no closed-form expression for the projector onto a given convex set. However, the concepts above (orthogonality, projection onto a subspace) can help us understand the solution for an arbitrary convex set. Uniqueness If C is closed 3 and convex, then for any x 0, the program has a unique solution. min y C x 0 y (3) First of all, that there is at least one point in C such that the minimum is obtained is a consequence of C being closed and of x 0 y being a continuous function of y. Let ˆx be one such minimizer; we will show that x 0 y > x 0 ˆx for all y C, y ˆx. Consider first all the points in C which are co-aligned with ˆx. Let I = {α R : αˆx C}. Since C is convex and closed, this is a closed interval of the real line (that contains at least the point α = 1). The function g(α) = x 0 αˆx = α ˆx α ˆx, x 0 + x 0, 3 We review basic concepts in topology for R N in the Technical Details section at the end of the notes. 17

18 is minimized at α = 1 by construction, and since its second derivative is strictly positive, we know that this is the unique minima. So if there is another minimizer of (3), then it is not co-aligned with ˆx. Now let y be any point in C that is not co-aligned with ˆx. We will show that y cannot minimize (3) because the point ˆx/ + y/ C is definitively closer to x 0. We have x 0 ˆx y = x 0 ˆx = x 0 ˆx 4 < x 0 ˆx 4 ( x0 ˆx = x 0 y. + x 0 y + x 0 ŷ 4 + x 0 ŷ 4 + x 0 y + x 0 ˆx, x 0 y + x 0 ˆx x 0 y The strict inequality above follows from Cauchy-Schwarz, while the last inequality follows from the fact that ˆx is a minimizer. This shows that no y ˆx can also minimize (3), and so ˆx is unique. ) 18

19 Obtuseness. Relative to the solution ˆx, every vector in C is at an obtuse angle to the error x 0 ˆx. More precisely, P C (x 0 ) = ˆx if and only if y ˆx, x 0 ˆx 0. (4) Compare with () above. Here is a picture: ˆx x 0 y We first prove that (4) P C (x 0 ) = ˆx. For any y C, we have y x 0 = y ˆx + ˆx x 0 = y ˆx + ˆx x 0 + y ˆx, ˆx x 0 ˆx x 0 + y ˆx, ˆx x 0. Note that the inner product term above is the same as (4), but with the sign of the second argument flipped, so we know this term must be non-negative. Thus y x 0 ˆx x 0. Since this holds uniformly over all y C, ˆx must be the closest point in C to x 0. 19

20 We now show that P C (x 0 ) = ˆx (4). Let y be an arbitrary point in C. Since C is convex, the point ˆx + θ(y ˆx) must also be in C for all θ [0, 1]. Since ˆx = P C (x 0 ), ˆx + θ(y ˆx) x 0 ˆx x 0 for all θ [0, 1]. By our intermediate result (1) a few pages ago, this means y ˆx, ˆx x 0 0 y ˆx, x 0 ˆx 0. 0

21 Technical Details: closest point to a subspace In this section, we establish the orthogonality principle for projection of a point x 0 onto a subspace T. Let ˆx be a vector which obeys ê = x ˆx T. We will show that ˆx is the unique closest point to x 0 in T. Let y be any other vector in T, and set We will show that e = x y. e > ê (i.e. that x y > x ˆx ). Note that e = x y = ê (y ˆx) = ê (y ˆx), ê (y ˆx) = ê + y ˆx ê, y ˆx y ˆx, ê. Since y ˆx T and ê T, and so ê, y ˆx = 0, and y ˆx, ê = 0, e = ê + y ˆx. Since all three quantities in the expression above are positive and we see that y ˆx > 0 y ˆx, y ˆx e > ê. We leave it as an exercise to establish the converse; that if y, ˆx x 0 = 0 for all y T, then ˆx is the projection of x 0 onto T. 1

22 Technical Details: basic topology in R N This section contains a brief review of basic topological concepts in R N. Our discussion will take place using the standard Euclidean distance measure (i.e. l norm), but all of these definitions can be generalized to other metrics. An excellent source for this material is [Rud76]. We say that a sequence of vectors {x k, k = 1,,...} converges to ˆx if x k ˆx ˆx as k. More precisely, this means that for every ɛ > 0, there exists an n ɛ such that x k ˆx ɛ for all k n ɛ. It is easy to show that a sequence of vectors converge if and only if their individual components converge point-by-point. A set X is open if we can draw a small ball around every point in X which is also entirely contained in X. More precisely, let B(x, ɛ) be the set of all points within ɛ of x: B(x, ɛ) = {y R N : x y ɛ}. Then X is open if for every x X, there exists an ɛ x > 0 such that B(x, ɛ x ) X. The standard example here is open intervals of the real line, e.g. (0, 1). There are many ways to define closed sets. The easiest is that a set X is closed if its complement is open. A more illuminating (and equivalent) definition is that X is closed if it contains all of its limit points. A vector ˆx is a limit point of X if there exists a sequence of vectors {x k } X that converge to ˆx.

23 The closure of general set X, denoted cl(x ), is the set of all limit points of X. Note that every x X is trivially a limit point (take the sequence x k = x), so X cl(x ). By construction, cl(x ) is the smallest closed set that contains X. Related to the definition of open and closed sets are the technical definitions of boundary and interior. The interior of a set X is the collection of points around which we can place a ball of finite width which remains in the set: int(x ) = {x X : ɛ > 0 such that B(x, ɛ) X }. The boundary of X is the set of points in cl(x ) that are not in the interior: bd(x ) = cl(x )\ int(x ). Another (equivalent) way of defining this is the set of points that are in both the closure of X and the closure of its complement X c. Note that if the set is not closed, there may be boundary points that are not in the set itself. The set X is bounded if we can find a uniform upper bound on the distance between two points it contains; this upper bound is commonly referred to as the diameter of the set: diam X = sup x y. x,y X The set X R N is compact if it is closed and bounded. A key fact about compact sets is that every sequence has a convergent subsequence this is known as the Bolzano-Weierstrauss theorem. 3

24 References [Ber03] D. Bertsekas. Convex Analysis and Optimization. Athena Scientific, 003. [BV04] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 004. [Rud76] W. Rudin. Principles of Mathematical Analysis. McGraw-Hill,

### Mathematics 530. Practice Problems. n + 1 } Department of Mathematical Sciences University of Delaware Prof. T. Angell October 19, 2015 Mathematics 530 Practice Problems 1. Recall that an indifference relation on a partially ordered set is defined

### Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The

### IE 521 Convex Optimization Lecture 1: 16th January 2019 Outline 1 / 20 Which set is different from others? Figure: Four sets 2 / 20 Which set is different from others? Figure: Four sets 3 / 20 Interior, Closure, Boundary Definition.

### Inner product spaces. Layers of structure: Inner product spaces Layers of structure: vector space normed linear space inner product space The abstract definition of an inner product, which we will see very shortly, is simple (and by itself is pretty

### CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008. 1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function

### Optimization and Optimal Control in Banach Spaces Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,

### Lecture 1: Convex Sets January 23 IE 521: Convex Optimization Instructor: Niao He Lecture 1: Convex Sets January 23 Spring 2017, UIUC Scribe: Niao He Courtesy warning: These notes do not necessarily cover everything discussed in the class.

### Maths 212: Homework Solutions Maths 212: Homework Solutions 1. The definition of A ensures that x π for all x A, so π is an upper bound of A. To show it is the least upper bound, suppose x < π and consider two cases. If x < 1, then

### Math 5210, Definitions and Theorems on Metric Spaces Math 5210, Definitions and Theorems on Metric Spaces Let (X, d) be a metric space. We will use the following definitions (see Rudin, chap 2, particularly 2.18) 1. Let p X and r R, r > 0, The ball of radius

### Math 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces. Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,...,

### Convex Analysis and Optimization Chapter 2 Solutions Convex Analysis and Optimization Chapter 2 Solutions Dimitri P. Bertsekas with Angelia Nedić and Asuman E. Ozdaglar Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com

### Optimality Conditions for Nonsmooth Convex Optimization Optimality Conditions for Nonsmooth Convex Optimization Sangkyun Lee Oct 22, 2014 Let us consider a convex function f : R n R, where R is the extended real field, R := R {, + }, which is proper (f never

### The proximal mapping The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function

### Linear Algebra Massoud Malek CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product

### Introduction to Real Analysis Alternative Chapter 1 Christopher Heil Introduction to Real Analysis Alternative Chapter 1 A Primer on Norms and Banach Spaces Last Updated: March 10, 2018 c 2018 by Christopher Heil Chapter 1 A Primer on Norms and Banach Spaces

### DS-GA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra. DS-GA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1

### Chapter 2: Preliminaries and elements of convex analysis Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt-14-15.shtml Academic year 2014-15

### Set, functions and Euclidean space. Seungjin Han Set, functions and Euclidean space Seungjin Han September, 2018 1 Some Basics LOGIC A is necessary for B : If B holds, then A holds. B A A B is the contraposition of B A. A is sufficient for B: If A holds,

### On John type ellipsoids On John type ellipsoids B. Klartag Tel Aviv University Abstract Given an arbitrary convex symmetric body K R n, we construct a natural and non-trivial continuous map u K which associates ellipsoids to

### Chapter 1. Preliminaries. The purpose of this chapter is to provide some basic background information. Linear Space. Hilbert Space. Chapter 1 Preliminaries The purpose of this chapter is to provide some basic background information. Linear Space Hilbert Space Basic Principles 1 2 Preliminaries Linear Space The notion of linear space

### Some Background Material Chapter 1 Some Background Material In the first chapter, we present a quick review of elementary - but important - material as a way of dipping our toes in the water. This chapter also introduces important

### Lecture 7: Positive Semidefinite Matrices Lecture 7: Positive Semidefinite Matrices Rajat Mittal IIT Kanpur The main aim of this lecture note is to prepare your background for semidefinite programming. We have already seen some linear algebra.

### Convex Sets. Prof. Dan A. Simovici UMB Convex Sets Prof. Dan A. Simovici UMB 1 / 57 Outline 1 Closures, Interiors, Borders of Sets in R n 2 Segments and Convex Sets 3 Properties of the Class of Convex Sets 4 Closure and Interior Points of Convex

### Geometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### EE 546, Univ of Washington, Spring Proximal mapping. introduction. review of conjugate functions. proximal mapping. Proximal mapping 6 1 EE 546, Univ of Washington, Spring 2012 6. Proximal mapping introduction review of conjugate functions proximal mapping Proximal mapping 6 1 Proximal mapping the proximal mapping (prox-operator) of a convex

### Optimization Theory. A Concise Introduction. Jiongmin Yong October 11, 017 16:5 ws-book9x6 Book Title Optimization Theory 017-08-Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 017-08-Lecture Notes page Optimization

### Spectral Theory, with an Introduction to Operator Means. William L. Green Spectral Theory, with an Introduction to Operator Means William L. Green January 30, 2008 Contents Introduction............................... 1 Hilbert Space.............................. 4 Linear Maps

### Lecture 1 Introduction L. Vandenberghe EE236A (Fall 2013-14) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation

### Constrained Optimization and Lagrangian Duality CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may

### Dot Products. K. Behrend. April 3, Abstract A short review of some basic facts on the dot product. Projections. The spectral theorem. Dot Products K. Behrend April 3, 008 Abstract A short review of some basic facts on the dot product. Projections. The spectral theorem. Contents The dot product 3. Length of a vector........................

### Lecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained

### Elements of Convex Optimization Theory Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a self-contained overview of elements of convex optimization

### Appendix B Convex analysis This version: 28/02/2014 Appendix B Convex analysis In this appendix we review a few basic notions of convexity and related notions that will be important for us at various times. B.1 The Hausdorff distance

### HW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given. HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a real-valued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard

### Lecture Notes on Metric Spaces Lecture Notes on Metric Spaces Math 117: Summer 2007 John Douglas Moore Our goal of these notes is to explain a few facts regarding metric spaces not included in the first few chapters of the text ,

### Functional Analysis. Franck Sueur Metric spaces Definitions Completeness Compactness Separability... Functional Analysis Franck Sueur 2018-2019 Contents 1 Metric spaces 1 1.1 Definitions........................................ 1 1.2 Completeness...................................... 3 1.3 Compactness......................................

### Math 341: Convex Geometry. Xi Chen Math 341: Convex Geometry Xi Chen 479 Central Academic Building, University of Alberta, Edmonton, Alberta T6G 2G1, CANADA E-mail address: xichen@math.ualberta.ca CHAPTER 1 Basics 1. Euclidean Geometry

### Convex Analysis and Economic Theory Winter 2018 Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory Winter 2018 Supplement A: Mathematical background A.1 Extended real numbers The extended real number

### Handout 2: Elements of Convex Analysis ENGG 5501: Foundations of Optimization 2018 19 First Term Handout 2: Elements of Convex Analysis Instructor: Anthony Man Cho So September 10, 2018 As briefly mentioned in Handout 1, the notion of convexity

### Exercise Solutions to Functional Analysis Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n

### DO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO QUESTION BOOKLET EECS 227A Fall 2009 Midterm Tuesday, Ocotober 20, 11:10-12:30pm DO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO You have 80 minutes to complete the midterm. The midterm consists

### Lecture 2: Linear Algebra Review EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 2-6 of the web textbook 1 2.1 Vectors 2.1.1

### U.C. Berkeley CS294: Spectral Methods and Expanders Handout 11 Luca Trevisan February 29, 2016 U.C. Berkeley CS294: Spectral Methods and Expanders Handout Luca Trevisan February 29, 206 Lecture : ARV In which we introduce semi-definite programming and a semi-definite programming relaxation of sparsest

### Normed & Inner Product Vector Spaces Normed & Inner Product Vector Spaces ECE 174 Introduction to Linear & Nonlinear Optimization Ken Kreutz-Delgado ECE Department, UC San Diego Ken Kreutz-Delgado (UC San Diego) ECE 174 Fall 2016 1 / 27 Normed

### The Inverse Function Theorem 1 John Nachbar Washington University April 11, 2014 1 Overview. The Inverse Function Theorem 1 If a function f : R R is C 1 and if its derivative is strictly positive at some x R, then, by continuity of

### Linear Analysis Lecture 5 Linear Analysis Lecture 5 Inner Products and V Let dim V < with inner product,. Choose a basis B and let v, w V have coordinates in F n given by x 1. x n and y 1. y n, respectively. Let A F n n be the

### Course Notes for EE227C (Spring 2018): Convex Optimization and Approximation Course Notes for EE7C (Spring 018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee7c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee7c@berkeley.edu February

### UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction

### POLARS AND DUAL CONES POLARS AND DUAL CONES VERA ROSHCHINA Abstract. The goal of this note is to remind the basic definitions of convex sets and their polars. For more details see the classic references [1, 2] and  for polytopes.

### Functional Analysis Review Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all

### Introduction to Convex Analysis Microeconomics II - Tutoring Class Introduction to Convex Analysis Microeconomics II - Tutoring Class Professor: V. Filipe Martins-da-Rocha TA: Cinthia Konichi April 2010 1 Basic Concepts and Results This is a first glance on basic convex

### Topology, Math 581, Fall 2017 last updated: November 24, Topology 1, Math 581, Fall 2017: Notes and homework Krzysztof Chris Ciesielski Topology, Math 581, Fall 2017 last updated: November 24, 2017 1 Topology 1, Math 581, Fall 2017: Notes and homework Krzysztof Chris Ciesielski Class of August 17: Course and syllabus overview. Topology

### Math 320-2: Midterm 2 Practice Solutions Northwestern University, Winter 2015 Math 30-: Midterm Practice Solutions Northwestern University, Winter 015 1. Give an example of each of the following. No justification is needed. (a) A metric on R with respect to which R is bounded. (b)

### Convex Analysis and Economic Theory Winter 2018 Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory Winter 2018 Topic 0: Vector spaces 0.1 Basic notation Here are some of the fundamental sets and spaces

### Homework set 4 - Solutions Homework set 4 - Solutions Math 407 Renato Feres 1. Exercise 4.1, page 49 of notes. Let W := T0 m V and denote by GLW the general linear group of W, defined as the group of all linear isomorphisms of W

### FUNCTIONAL ANALYSIS-NORMED SPACE MAT641- MSC Mathematics, MNIT Jaipur FUNCTIONAL ANALYSIS-NORMED SPACE DR. RITU AGARWAL MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY JAIPUR 1. Normed space Norm generalizes the concept of length in an arbitrary

### Some Background Math Notes on Limsups, Sets, and Convexity EE599 STOCHASTIC NETWORK OPTIMIZATION, MICHAEL J. NEELY, FALL 2008 1 Some Background Math Notes on Limsups, Sets, and Convexity I. LIMITS Let f(t) be a real valued function of time. Suppose f(t) converges

### Chapter 2 Convex Analysis Chapter 2 Convex Analysis The theory of nonsmooth analysis is based on convex analysis. Thus, we start this chapter by giving basic concepts and results of convexity (for further readings see also [202,

### Jensen s inequality for multivariate medians Jensen s inequality for multivariate medians Milan Merkle University of Belgrade, Serbia emerkle@etf.rs Given a probability measure µ on Borel sigma-field of R d, and a function f : R d R, the main issue

### Convex Functions. Pontus Giselsson Convex Functions Pontus Giselsson 1 Today s lecture lower semicontinuity, closure, convex hull convexity preserving operations precomposition with affine mapping infimal convolution image function supremum

### Math 209B Homework 2 Math 29B Homework 2 Edward Burkard Note: All vector spaces are over the field F = R or C 4.6. Two Compactness Theorems. 4. Point Set Topology Exercise 6 The product of countably many sequentally compact

### NORMS ON SPACE OF MATRICES NORMS ON SPACE OF MATRICES. Operator Norms on Space of linear maps Let A be an n n real matrix and x 0 be a vector in R n. We would like to use the Picard iteration method to solve for the following system

### REVIEW OF ESSENTIAL MATH 346 TOPICS REVIEW OF ESSENTIAL MATH 346 TOPICS 1. AXIOMATIC STRUCTURE OF R Doğan Çömez The real number system is a complete ordered field, i.e., it is a set R which is endowed with addition and multiplication operations

### MAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012 (Homework 1: Chapter 1: Exercises 1-7, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions

### Spectral theory for compact operators on Banach spaces 68 Chapter 9 Spectral theory for compact operators on Banach spaces Recall that a subset S of a metric space X is precompact if its closure is compact, or equivalently every sequence contains a Cauchy

### Statistics 612: L p spaces, metrics on spaces of probabilites, and connections to estimation Statistics 62: L p spaces, metrics on spaces of probabilites, and connections to estimation Moulinath Banerjee December 6, 2006 L p spaces and Hilbert spaces We first formally define L p spaces. Consider

### USING FUNCTIONAL ANALYSIS AND SOBOLEV SPACES TO SOLVE POISSON S EQUATION USING FUNCTIONAL ANALYSIS AND SOBOLEV SPACES TO SOLVE POISSON S EQUATION YI WANG Abstract. We study Banach and Hilbert spaces with an eye towards defining weak solutions to elliptic PDE. Using Lax-Milgram

### Analysis and Linear Algebra. Lectures 1-3 on the mathematical tools that will be used in C103 Analysis and Linear Algebra Lectures 1-3 on the mathematical tools that will be used in C103 Set Notation A, B sets AcB union A1B intersection A\B the set of objects in A that are not in B N. Empty set

### PROBLEMS. (b) (Polarization Identity) Show that in any inner product space 1 Professor Carl Cowen Math 54600 Fall 09 PROBLEMS 1. (Geometry in Inner Product Spaces) (a) (Parallelogram Law) Show that in any inner product space x + y 2 + x y 2 = 2( x 2 + y 2 ). (b) (Polarization

### Inequality Constraints Chapter 2 Inequality Constraints 2.1 Optimality Conditions Early in multivariate calculus we learn the significance of differentiability in finding minimizers. In this section we begin our study of the

### Lecture Notes in Advanced Calculus 1 (80315) Raz Kupferman Institute of Mathematics The Hebrew University Lecture Notes in Advanced Calculus 1 (80315) Raz Kupferman Institute of Mathematics The Hebrew University February 7, 2007 2 Contents 1 Metric Spaces 1 1.1 Basic definitions...........................

### Math 302 Outcome Statements Winter 2013 Math 302 Outcome Statements Winter 2013 1 Rectangular Space Coordinates; Vectors in the Three-Dimensional Space (a) Cartesian coordinates of a point (b) sphere (c) symmetry about a point, a line, and a

### Existence of minimizers Existence of imizers We have just talked a lot about how to find the imizer of an unconstrained convex optimization problem. We have not talked too much, at least not in concrete mathematical terms, about

### Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem 56 Chapter 7 Locally convex spaces, the hyperplane separation theorem, and the Krein-Milman theorem Recall that C(X) is not a normed linear space when X is not compact. On the other hand we could use semi

### Analysis Preliminary Exam Workshop: Hilbert Spaces Analysis Preliminary Exam Workshop: Hilbert Spaces 1. Hilbert spaces A Hilbert space H is a complete real or complex inner product space. Consider complex Hilbert spaces for definiteness. If (, ) : H H

### Chapter 7. Extremal Problems. 7.1 Extrema and Local Extrema Chapter 7 Extremal Problems No matter in theoretical context or in applications many problems can be formulated as problems of finding the maximum or minimum of a function. Whenever this is the case, advanced

### Course Summary Math 211 Course Summary Math 211 table of contents I. Functions of several variables. II. R n. III. Derivatives. IV. Taylor s Theorem. V. Differential Geometry. VI. Applications. 1. Best affine approximations.

### Summer School: Semidefinite Optimization Summer School: Semidefinite Optimization Christine Bachoc Université Bordeaux I, IMB Research Training Group Experimental and Constructive Algebra Haus Karrenberg, Sept. 3 - Sept. 7, 2012 Duality Theory

### Math Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88 Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant

### Helly's Theorem and its Equivalences via Convex Analysis Portland State University PDXScholar University Honors Theses University Honors College 2014 Helly's Theorem and its Equivalences via Convex Analysis Adam Robinson Portland State University Let us know

### NOTES ON MULTIVARIABLE CALCULUS: DIFFERENTIAL CALCULUS NOTES ON MULTIVARIABLE CALCULUS: DIFFERENTIAL CALCULUS SAMEER CHAVAN Abstract. This is the first part of Notes on Multivariable Calculus based on the classical texts  and . We present here the geometric

### A NICE PROOF OF FARKAS LEMMA A NICE PROOF OF FARKAS LEMMA DANIEL VICTOR TAUSK Abstract. The goal of this short note is to present a nice proof of Farkas Lemma which states that if C is the convex cone spanned by a finite set and if

### Linear Algebra I. Ronald van Luijk, 2015 Linear Algebra I Ronald van Luijk, 2015 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents Dependencies among sections 3 Chapter 1. Euclidean space: lines and hyperplanes 5 1.1. Definition

### (1) Consider the space S consisting of all continuous real-valued functions on the closed interval [0, 1]. For f, g S, define Homework, Real Analysis I, Fall, 2010. (1) Consider the space S consisting of all continuous real-valued functions on the closed interval [0, 1]. For f, g S, define ρ(f, g) = 1 0 f(x) g(x) dx. Show that

### LINEAR ALGEBRA W W L CHEN LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008. This chapter is available free to all individuals, on the understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,

### Chapter 2 Metric Spaces Chapter 2 Metric Spaces The purpose of this chapter is to present a summary of some basic properties of metric and topological spaces that play an important role in the main body of the book. 2.1 Metrics

### Structural and Multidisciplinary Optimization. P. Duysinx and P. Tossings Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 2018-2019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be

### Topological vectorspaces (July 25, 2011) Topological vectorspaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ Natural non-fréchet spaces Topological vector spaces Quotients and linear maps More topological

### Chapter 6: Orthogonality Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products

### Vector Spaces, Affine Spaces, and Metric Spaces Vector Spaces, Affine Spaces, and Metric Spaces 2 This chapter is only meant to give a short overview of the most important concepts in linear algebra, affine spaces, and metric spaces and is not intended

### 10725/36725 Optimization Homework 2 Solutions 10725/36725 Optimization Homework 2 Solutions 1 Convexity (Kevin) 1.1 Sets Let A R n be a closed set with non-empty interior that has a supporting hyperplane at every point on its boundary. (a) Show that

### Chapter 5. Basics of Euclidean Geometry Chapter 5 Basics of Euclidean Geometry 5.1 Inner Products, Euclidean Spaces In Affine geometry, it is possible to deal with ratios of vectors and barycenters of points, but there is no way to express the

### Convex Functions and Optimization Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized

### Convex Geometry. Carsten Schütt Convex Geometry Carsten Schütt November 25, 2006 2 Contents 0.1 Convex sets... 4 0.2 Separation.... 9 0.3 Extreme points..... 15 0.4 Blaschke selection principle... 18 0.5 Polytopes and polyhedra.... 23 WEEK In general terms, our aim in this first part of the course is to use vector space theory to study the geometry of Euclidean space A good knowledge of the subject matter of the Matrix Applications Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plus-properties and F-uniqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FP-I book: