In English, this means that if we travel on a straight line between any two points in C, then we never leave C.


 Andra Barnett
 1 years ago
 Views:
Transcription
1 Convex sets In this section, we will be introduced to some of the mathematical fundamentals of convex sets. In order to motivate some of the definitions, we will look at the closest point problem from several different angles. The tools and concepts we develop here, however, have many other applications both in this course and beyond. A set C R N is convex if x, y C (1 θ)x + θy C for all θ [0, 1]. In English, this means that if we travel on a straight line between any two points in C, then we never leave C. These sets in R are convex: These sets are not: 1
2 Examples of convex (and nonconvex) sets Subspaces. Recall that if S is a subspace of R N, then x, y S ax + by S for all a, b R. So S is clearly convex. Affine sets. Affine sets are just subspaces that have been offset by the origin: {x R N : x = y + v, y T }, T = subspace, for some fixed vector v. An equivalent definition is that x, y C θx + (1 θ)y C for all θ R the difference between this definition and that for a subspace is that subspaces must include the origin. Bound constraints. Rectangular sets of the form C = {x R N : l 1 x 1 u 1, l x u,..., l N x N u N } for some l 1,..., l N, u 1,..., u N R are convex. The simplex in R N {x R N : x 1 + x + + x N 1, x 1, x,..., x N 0} is convex. Any subset of R N that can be expressed as a set of linear inequality constraints {x R N : Ax b} is convex. Notice that both rectangular sets and the simplex
3 fall into this category for the simplex, take A = , b = In general, when sets like these are bounded, the result is a polyhedron. Norm balls. If is a valid norm on R N, then is a convex set. B r = {x : x r}, Ellipsoids. An ellipsoid is a set of the form E = {x : (x x 0 ) T P 1 (x x 0 ) r}, for a symmetric positivedefinite matrix P. Geometrically, the ellipsoid is centered at x 0, its axes are oriented with the eigenvectors of P, and the relative widths along these axes are proportional to the eigenvalues of P. A single point {x} is convex. The empty set is convex. The set is convex. (Sketch it!) The set is not convex. {x R : x 1 x 1 x + 1 0} {x R : x 1 x 1 x + 1 0} 3
4 The set is certainly not convex. {x R : x 1 x 1 x + 1 = 0} Sets defined by linear equality constraints where only some of the constraints have to hold are in general not convex. For example {x R : x 1 x 1 and x 1 + x 1} is convex, while is not convex. {x R : x 1 x 1 or x 1 + x 1} Cones A cone is a set C such that x C θx C for all θ 0. Convex cones are sets which are both convex and a cone. C is a convex cone if x 1, x C θ 1 x 1 + θ x C for all θ 1, θ 0. Given an x 1, x, the set of all linear combinations with positive weights makes a wedge. For practice, sketch the region below that consists of all such combinations of x 1 and x : 4
5 x x 1 We will mostly be interested in proper cones, which in addition to being convex, are closed, have a nonempty interior 1 ( solid ), and do not contain entire lines ( pointed ). Examples: Nonnegative orthant. The set of vectors whose entries are nonnegative, R N + = {x R N : x n 0, for n = 1,..., N}, is a proper cone. Positive semidefinite cone. The set of N N symmetric matrices with nonnegative eigenvalues, S N + = {X R N N : X = V ΛV T, is a proper cone. V T V = V V T = I, Λ diagonal and nonnegative} Nonnegative polynomials. Vectors of coefficients of nonnegative polynomials on [0, 1], {x R N : x 1 +x t+x 3 t + +x N t N 1 0 for all 0 t 1}, 1 See Technical Details for precise definition. 5
6 form a proper cone. Notice that it is not necessary that all the x n 0; for example t t (x 1 = 0, x = 1, x 3 = 1) is nonnegative on [0, 1]. Norm cones. The subset of R N+1 defined by {(x, t), x R N, t R : x t} is a proper cone for any valid norm and t > 0. Sketch what this looks like for the Euclidean norm and N = : Every proper cone K defines a partial ordering or generalized inequality. We write x K y when y x K. For example, for vectors x, y R N, we say x R N + y when x n y n for all n = 1,..., N. For symmetric matrices X, Y, we say X S N + Y when Y X has nonnegative eigenvalues. We will typically just use when the context makes it clear. In fact, for R N + we will just write x y (as we did above) to mean that 6
7 the entries in x are componentbycomponent upperbounded by the entries in y. Partial orderings obey share of the properties of the standard on the real line. For example: x y, u v x + u y + v. But other properties do not hold; for example, it is not necessary that either x y or y x. For an extensive list of properties of partial orderings (most of which will make perfect sense on sight) can be found in [BV04, Chapter.4]. Hyperplanes and halfspaces Hyperplanes and halfspaces are both very simple constructs, but they will be crucial to our understanding to convex sets, functions, and optimization problems. A hyperplane is a set of the form {x R N : x, c = t} for some fixed vector c 0 and scalar t. When t = 0, this set is a subspace of dimension N 1, and contains all vectors that are orthogonal to c. For t 0, this is an affine space consisting of all the vectors orthogonal to c (call this set C ) offset to some x 0 : {x R N : x, c = t} = {x R N : x = x 0 + C }, for any x 0 with x 0, c = t. We might take x 0 = t c/ c, for instance. The point is, c is a normal vector of the set. Here are some examples in R : 7
8 5 5 4 t =1 4 t = apple 1 c = 0 1 apple c = t = 1 t =1 t =5 It should be clear that hyperplanes are convex sets. A halfspace is a set of the form {x R N : x, c t} for some fixed vector c 0 and scalar t. For t = 0, the halfspace contains all vectors whose inner product with c is negative (i.e. the angle between x and c is greater than 90 ). Here is a simple example: 8
9 5 4 t =5 3 apple c = Separating hyperplanes If two convex sets are disjoint, then there is a hyperplane that separates them. C c d D {x : hx, ai = b} a = d c b = kdk kck 9
10 This fact is intuitive, and is incredibly useful in understanding the solutions to convex optimization programs (we will see this even in the next section). It also easy to prove under mild additional assumptions; below we will show how to explicitly construct a separating hyperplane between two convex sets if there are points in the two sets that are definitively closest to one another. The result extends to general convex sets, and the proof is based on the same basic idea, but requires some additional toplogical considerations Let C, D be disjoint convex sets and suppose that there exist points c C and d D that achieve the minimum distance c d = inf x y. x C,y D Define Then a = d c, b = d c. x, a b for x C, x, a b for x D. That is the hyperplane {x : x, a = b} separates C and D. To see this, set f(x) = x, a b, and show that for and point u in D, we have f(u) 0. First, we prove the basic geometric fact that for any two vectors x, y, if x + θy x for all θ [0, 1] then x, y 0. (1) This holds, for example, when the sets are closed, and at least one of them is bounded. 10
11 To establish this, we expand the norm as x + θy = x + θ y + θ x, y, from which we can immediately deduce that θ y + x, y 0 for all θ [0, 1] x, y 0. Now let u be an arbitrary point in D. Since D is convex, we know that d + θ(u d) D for all θ [0, 1]. Since d is as close to c as any other point in D, we have and so by (1), we know that This means d + θ(u d) c d c, d c, u d 0. f(u) = u, d c d c d + c, d c = u, d c = u (d + c)/, d c = u d + d/ c/, d c = u d, d c + c d 0. The proof that for every v C, f(v) 0 is exactly the same. 11
12 Note the following: We actually only need the interiors of sets C and D to be disjoint they can intersect at one or more points along their boundaries. Viz C D But you have to be more careful in defining the hyperplane which separates them (as we would have c = d a = 0 in the argument above). There can be many hyperplanes that separate the two sets, but there is not necessarily more than one. The hyperplane {x : x, a = b} strictly separates sets C and D if x, a < b for all x C and x, a > b for all x D. There are examples of closed, convex, disjoint sets C, D that are separable but not strictly separable. It is a good exercise to think of an example. 1
13 Supporting hyperplanes A direct consequence of the separating hyperplane theorem is that every point on the boundary of a convex set C can be separated from its interior. If a 0 satisfies x, a x 0, a for all x C, then {x : x, a = x 0, a } is called a supporting hyperplane to C at x 0. Here s a picture: x 0 The hyperplane is tangent to C at x 0, and the halfspace {x : x, a x 0, a } contains C. The supporting hyperplane theorem says that a supporting hyperplane exists at every point x 0 on the boundary of a (nonempty) convex set C. Note that there might be more than one: x 0 13
14 Given the separating hyperplane theorem, this is easy to prove. If C has a nonempty interior, then it follows from just realizing that x 0 and int(c) are disjoint. If int(c) =, then C must be contained in an affine set, and hence also in (possibly many) hyperplanes, any of which will be supporting. 14
15 The closest point problem Let x 0 R N be given, and let C be a nonempty, closed, convex set. The projection of x 0 onto C is the closest point (in the standard Euclidean distance, for now) in C to x 0 : P C (x 0 ) = arg min y C x 0 y We will see below that there is a unique minimizer to this problem, and that the solution has geometric properties that are analogous to the case where C is a subspace. Projection onto a subspace Let s recall how we solve this problem in the special case where C := T is a Kdimensional subspace. In this case, the solution ˆx = P T (x 0 ) is unique, and is characterized by the orthogonality principle: x 0 ˆx T meaning that y, x 0 ˆx = 0 for all y T. The proof of this fact is reviewed in the Technical Details section at the end of these notes. The orthogonality principle leads immediately to an algorithm for calculating P T (x 0 ). Let v 1,..., v K be a basis for T ; we can write the solution as K ˆx = α k v k ; k=1 solving for the expansion coefficients α k is the same as solving for ˆx. We know that K x 0 α j v j, v k = 0, for k = 1,..., K, j=1 15
16 and so the α k must obey the linear system of equations K α j v j, v k = x 0, v k, for k = 1,..., K. j=1 Concatenating the v k as columns in the N K matrix V, and the entries α k into the vector α R K, we can write the equations above as V T V α = V T x 0. Since the {v k } are a basis for T (i.e. they are linearly independent), V T V is invertible, and we can solve for the best expansion coefficients: ˆα = (V T V ) 1 V T x 0. Using these expansion coefficient to reconstruct ˆx yields ˆx = V ˆα = V (V T V ) 1 V T x 0. In this case, the projector P T ( ) is a linear map, specified by N N matrix V (V T V ) 1 V T. Projection onto an affine set Affine sets are not fundamentally different than subspaces. Any affine set C can be written as a subspace T plus an offset v 0 : C = T + v 0 = {x : x = y + v 0, y C}. It is easy to translate the results for subspaces above to say that the projection onto an affine set is unique, and obeys the orthogonality principle y ˆx, x 0 ˆx = 0, for all y C. () 16
17 You can solve this problem by shifting x 0 and C by negative v 0, projecting x 0 v 0 onto the subspace C v 0, and then shifting the answer back. Projection onto a general convex set In general, there is no closedform expression for the projector onto a given convex set. However, the concepts above (orthogonality, projection onto a subspace) can help us understand the solution for an arbitrary convex set. Uniqueness If C is closed 3 and convex, then for any x 0, the program has a unique solution. min y C x 0 y (3) First of all, that there is at least one point in C such that the minimum is obtained is a consequence of C being closed and of x 0 y being a continuous function of y. Let ˆx be one such minimizer; we will show that x 0 y > x 0 ˆx for all y C, y ˆx. Consider first all the points in C which are coaligned with ˆx. Let I = {α R : αˆx C}. Since C is convex and closed, this is a closed interval of the real line (that contains at least the point α = 1). The function g(α) = x 0 αˆx = α ˆx α ˆx, x 0 + x 0, 3 We review basic concepts in topology for R N in the Technical Details section at the end of the notes. 17
18 is minimized at α = 1 by construction, and since its second derivative is strictly positive, we know that this is the unique minima. So if there is another minimizer of (3), then it is not coaligned with ˆx. Now let y be any point in C that is not coaligned with ˆx. We will show that y cannot minimize (3) because the point ˆx/ + y/ C is definitively closer to x 0. We have x 0 ˆx y = x 0 ˆx = x 0 ˆx 4 < x 0 ˆx 4 ( x0 ˆx = x 0 y. + x 0 y + x 0 ŷ 4 + x 0 ŷ 4 + x 0 y + x 0 ˆx, x 0 y + x 0 ˆx x 0 y The strict inequality above follows from CauchySchwarz, while the last inequality follows from the fact that ˆx is a minimizer. This shows that no y ˆx can also minimize (3), and so ˆx is unique. ) 18
19 Obtuseness. Relative to the solution ˆx, every vector in C is at an obtuse angle to the error x 0 ˆx. More precisely, P C (x 0 ) = ˆx if and only if y ˆx, x 0 ˆx 0. (4) Compare with () above. Here is a picture: ˆx x 0 y We first prove that (4) P C (x 0 ) = ˆx. For any y C, we have y x 0 = y ˆx + ˆx x 0 = y ˆx + ˆx x 0 + y ˆx, ˆx x 0 ˆx x 0 + y ˆx, ˆx x 0. Note that the inner product term above is the same as (4), but with the sign of the second argument flipped, so we know this term must be nonnegative. Thus y x 0 ˆx x 0. Since this holds uniformly over all y C, ˆx must be the closest point in C to x 0. 19
20 We now show that P C (x 0 ) = ˆx (4). Let y be an arbitrary point in C. Since C is convex, the point ˆx + θ(y ˆx) must also be in C for all θ [0, 1]. Since ˆx = P C (x 0 ), ˆx + θ(y ˆx) x 0 ˆx x 0 for all θ [0, 1]. By our intermediate result (1) a few pages ago, this means y ˆx, ˆx x 0 0 y ˆx, x 0 ˆx 0. 0
21 Technical Details: closest point to a subspace In this section, we establish the orthogonality principle for projection of a point x 0 onto a subspace T. Let ˆx be a vector which obeys ê = x ˆx T. We will show that ˆx is the unique closest point to x 0 in T. Let y be any other vector in T, and set We will show that e = x y. e > ê (i.e. that x y > x ˆx ). Note that e = x y = ê (y ˆx) = ê (y ˆx), ê (y ˆx) = ê + y ˆx ê, y ˆx y ˆx, ê. Since y ˆx T and ê T, and so ê, y ˆx = 0, and y ˆx, ê = 0, e = ê + y ˆx. Since all three quantities in the expression above are positive and we see that y ˆx > 0 y ˆx, y ˆx e > ê. We leave it as an exercise to establish the converse; that if y, ˆx x 0 = 0 for all y T, then ˆx is the projection of x 0 onto T. 1
22 Technical Details: basic topology in R N This section contains a brief review of basic topological concepts in R N. Our discussion will take place using the standard Euclidean distance measure (i.e. l norm), but all of these definitions can be generalized to other metrics. An excellent source for this material is [Rud76]. We say that a sequence of vectors {x k, k = 1,,...} converges to ˆx if x k ˆx ˆx as k. More precisely, this means that for every ɛ > 0, there exists an n ɛ such that x k ˆx ɛ for all k n ɛ. It is easy to show that a sequence of vectors converge if and only if their individual components converge pointbypoint. A set X is open if we can draw a small ball around every point in X which is also entirely contained in X. More precisely, let B(x, ɛ) be the set of all points within ɛ of x: B(x, ɛ) = {y R N : x y ɛ}. Then X is open if for every x X, there exists an ɛ x > 0 such that B(x, ɛ x ) X. The standard example here is open intervals of the real line, e.g. (0, 1). There are many ways to define closed sets. The easiest is that a set X is closed if its complement is open. A more illuminating (and equivalent) definition is that X is closed if it contains all of its limit points. A vector ˆx is a limit point of X if there exists a sequence of vectors {x k } X that converge to ˆx.
23 The closure of general set X, denoted cl(x ), is the set of all limit points of X. Note that every x X is trivially a limit point (take the sequence x k = x), so X cl(x ). By construction, cl(x ) is the smallest closed set that contains X. Related to the definition of open and closed sets are the technical definitions of boundary and interior. The interior of a set X is the collection of points around which we can place a ball of finite width which remains in the set: int(x ) = {x X : ɛ > 0 such that B(x, ɛ) X }. The boundary of X is the set of points in cl(x ) that are not in the interior: bd(x ) = cl(x )\ int(x ). Another (equivalent) way of defining this is the set of points that are in both the closure of X and the closure of its complement X c. Note that if the set is not closed, there may be boundary points that are not in the set itself. The set X is bounded if we can find a uniform upper bound on the distance between two points it contains; this upper bound is commonly referred to as the diameter of the set: diam X = sup x y. x,y X The set X R N is compact if it is closed and bounded. A key fact about compact sets is that every sequence has a convergent subsequence this is known as the BolzanoWeierstrauss theorem. 3
24 References [Ber03] D. Bertsekas. Convex Analysis and Optimization. Athena Scientific, 003. [BV04] S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 004. [Rud76] W. Rudin. Principles of Mathematical Analysis. McGrawHill,
Mathematics 530. Practice Problems. n + 1 }
Department of Mathematical Sciences University of Delaware Prof. T. Angell October 19, 2015 Mathematics 530 Practice Problems 1. Recall that an indifference relation on a partially ordered set is defined
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very wellwritten and a pleasure to read. The
More informationIE 521 Convex Optimization
Lecture 1: 16th January 2019 Outline 1 / 20 Which set is different from others? Figure: Four sets 2 / 20 Which set is different from others? Figure: Four sets 3 / 20 Interior, Closure, Boundary Definition.
More informationInner product spaces. Layers of structure:
Inner product spaces Layers of structure: vector space normed linear space inner product space The abstract definition of an inner product, which we will see very shortly, is simple (and by itself is pretty
More informationCHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.
1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function
More informationOptimization and Optimal Control in Banach Spaces
Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex nonsmooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,
More informationLecture 1: Convex Sets January 23
IE 521: Convex Optimization Instructor: Niao He Lecture 1: Convex Sets January 23 Spring 2017, UIUC Scribe: Niao He Courtesy warning: These notes do not necessarily cover everything discussed in the class.
More informationMaths 212: Homework Solutions
Maths 212: Homework Solutions 1. The definition of A ensures that x π for all x A, so π is an upper bound of A. To show it is the least upper bound, suppose x < π and consider two cases. If x < 1, then
More informationMath 5210, Definitions and Theorems on Metric Spaces
Math 5210, Definitions and Theorems on Metric Spaces Let (X, d) be a metric space. We will use the following definitions (see Rudin, chap 2, particularly 2.18) 1. Let p X and r R, r > 0, The ball of radius
More informationMath 350 Fall 2011 Notes about inner product spaces. In this notes we state and prove some important properties of inner product spaces.
Math 350 Fall 2011 Notes about inner product spaces In this notes we state and prove some important properties of inner product spaces. First, recall the dot product on R n : if x, y R n, say x = (x 1,...,
More informationConvex Analysis and Optimization Chapter 2 Solutions
Convex Analysis and Optimization Chapter 2 Solutions Dimitri P. Bertsekas with Angelia Nedić and Asuman E. Ozdaglar Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationOptimality Conditions for Nonsmooth Convex Optimization
Optimality Conditions for Nonsmooth Convex Optimization Sangkyun Lee Oct 22, 2014 Let us consider a convex function f : R n R, where R is the extended real field, R := R {, + }, which is proper (f never
More informationThe proximal mapping
The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt2016fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function
More informationLinear Algebra Massoud Malek
CSUEB Linear Algebra Massoud Malek Inner Product and Normed Space In all that follows, the n n identity matrix is denoted by I n, the n n zero matrix by Z n, and the zero vector by θ n An inner product
More informationIntroduction to Real Analysis Alternative Chapter 1
Christopher Heil Introduction to Real Analysis Alternative Chapter 1 A Primer on Norms and Banach Spaces Last Updated: March 10, 2018 c 2018 by Christopher Heil Chapter 1 A Primer on Norms and Banach Spaces
More informationDSGA 1002 Lecture notes 0 Fall Linear Algebra. These notes provide a review of basic concepts in linear algebra.
DSGA 1002 Lecture notes 0 Fall 2016 Linear Algebra These notes provide a review of basic concepts in linear algebra. 1 Vector spaces You are no doubt familiar with vectors in R 2 or R 3, i.e. [ ] 1.1
More informationChapter 2: Preliminaries and elements of convex analysis
Chapter 2: Preliminaries and elements of convex analysis Edoardo Amaldi DEIB Politecnico di Milano edoardo.amaldi@polimi.it Website: http://home.deib.polimi.it/amaldi/opt1415.shtml Academic year 201415
More informationSet, functions and Euclidean space. Seungjin Han
Set, functions and Euclidean space Seungjin Han September, 2018 1 Some Basics LOGIC A is necessary for B : If B holds, then A holds. B A A B is the contraposition of B A. A is sufficient for B: If A holds,
More informationOn John type ellipsoids
On John type ellipsoids B. Klartag Tel Aviv University Abstract Given an arbitrary convex symmetric body K R n, we construct a natural and nontrivial continuous map u K which associates ellipsoids to
More informationChapter 1. Preliminaries. The purpose of this chapter is to provide some basic background information. Linear Space. Hilbert Space.
Chapter 1 Preliminaries The purpose of this chapter is to provide some basic background information. Linear Space Hilbert Space Basic Principles 1 2 Preliminaries Linear Space The notion of linear space
More informationSome Background Material
Chapter 1 Some Background Material In the first chapter, we present a quick review of elementary  but important  material as a way of dipping our toes in the water. This chapter also introduces important
More informationLecture 7: Positive Semidefinite Matrices
Lecture 7: Positive Semidefinite Matrices Rajat Mittal IIT Kanpur The main aim of this lecture note is to prepare your background for semidefinite programming. We have already seen some linear algebra.
More informationConvex Sets. Prof. Dan A. Simovici UMB
Convex Sets Prof. Dan A. Simovici UMB 1 / 57 Outline 1 Closures, Interiors, Borders of Sets in R n 2 Segments and Convex Sets 3 Properties of the Class of Convex Sets 4 Closure and Interior Points of Convex
More informationGeometric problems. Chapter Projection on a set. The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as
Chapter 8 Geometric problems 8.1 Projection on a set The distance of a point x 0 R n to a closed set C R n, in the norm, is defined as dist(x 0,C) = inf{ x 0 x x C}. The infimum here is always achieved.
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2009 Copyright 2009 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationEE 546, Univ of Washington, Spring Proximal mapping. introduction. review of conjugate functions. proximal mapping. Proximal mapping 6 1
EE 546, Univ of Washington, Spring 2012 6. Proximal mapping introduction review of conjugate functions proximal mapping Proximal mapping 6 1 Proximal mapping the proximal mapping (proxoperator) of a convex
More informationOptimization Theory. A Concise Introduction. Jiongmin Yong
October 11, 017 16:5 wsbook9x6 Book Title Optimization Theory 01708Lecture Notes page 1 1 Optimization Theory A Concise Introduction Jiongmin Yong Optimization Theory 01708Lecture Notes page Optimization
More informationSpectral Theory, with an Introduction to Operator Means. William L. Green
Spectral Theory, with an Introduction to Operator Means William L. Green January 30, 2008 Contents Introduction............................... 1 Hilbert Space.............................. 4 Linear Maps
More informationLecture 1 Introduction
L. Vandenberghe EE236A (Fall 201314) Lecture 1 Introduction course overview linear optimization examples history approximate syllabus basic definitions linear optimization in vector and matrix notation
More informationConstrained Optimization and Lagrangian Duality
CIS 520: Machine Learning Oct 02, 2017 Constrained Optimization and Lagrangian Duality Lecturer: Shivani Agarwal Disclaimer: These notes are designed to be a supplement to the lecture. They may or may
More informationDot Products. K. Behrend. April 3, Abstract A short review of some basic facts on the dot product. Projections. The spectral theorem.
Dot Products K. Behrend April 3, 008 Abstract A short review of some basic facts on the dot product. Projections. The spectral theorem. Contents The dot product 3. Length of a vector........................
More informationLecture 1: Introduction. Outline. B9824 Foundations of Optimization. Fall Administrative matters. 2. Introduction. 3. Existence of optima
B9824 Foundations of Optimization Lecture 1: Introduction Fall 2010 Copyright 2010 Ciamac Moallemi Outline 1. Administrative matters 2. Introduction 3. Existence of optima 4. Local theory of unconstrained
More informationElements of Convex Optimization Theory
Elements of Convex Optimization Theory Costis Skiadas August 2015 This is a revised and extended version of Appendix A of Skiadas (2009), providing a selfcontained overview of elements of convex optimization
More informationAppendix B Convex analysis
This version: 28/02/2014 Appendix B Convex analysis In this appendix we review a few basic notions of convexity and related notions that will be important for us at various times. B.1 The Hausdorff distance
More informationHW1 solutions. 1. α Ef(x) β, where Ef(x) is the expected value of f(x), i.e., Ef(x) = n. i=1 p if(a i ). (The function f : R R is given.
HW1 solutions Exercise 1 (Some sets of probability distributions.) Let x be a realvalued random variable with Prob(x = a i ) = p i, i = 1,..., n, where a 1 < a 2 < < a n. Of course p R n lies in the standard
More informationLecture Notes on Metric Spaces
Lecture Notes on Metric Spaces Math 117: Summer 2007 John Douglas Moore Our goal of these notes is to explain a few facts regarding metric spaces not included in the first few chapters of the text [1],
More informationFunctional Analysis. Franck Sueur Metric spaces Definitions Completeness Compactness Separability...
Functional Analysis Franck Sueur 20182019 Contents 1 Metric spaces 1 1.1 Definitions........................................ 1 1.2 Completeness...................................... 3 1.3 Compactness......................................
More informationMath 341: Convex Geometry. Xi Chen
Math 341: Convex Geometry Xi Chen 479 Central Academic Building, University of Alberta, Edmonton, Alberta T6G 2G1, CANADA Email address: xichen@math.ualberta.ca CHAPTER 1 Basics 1. Euclidean Geometry
More informationConvex Analysis and Economic Theory Winter 2018
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory Winter 2018 Supplement A: Mathematical background A.1 Extended real numbers The extended real number
More informationHandout 2: Elements of Convex Analysis
ENGG 5501: Foundations of Optimization 2018 19 First Term Handout 2: Elements of Convex Analysis Instructor: Anthony Man Cho So September 10, 2018 As briefly mentioned in Handout 1, the notion of convexity
More informationExercise Solutions to Functional Analysis
Exercise Solutions to Functional Analysis Note: References refer to M. Schechter, Principles of Functional Analysis Exersize that. Let φ,..., φ n be an orthonormal set in a Hilbert space H. Show n f n
More informationDO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO
QUESTION BOOKLET EECS 227A Fall 2009 Midterm Tuesday, Ocotober 20, 11:1012:30pm DO NOT OPEN THIS QUESTION BOOKLET UNTIL YOU ARE TOLD TO DO SO You have 80 minutes to complete the midterm. The midterm consists
More informationLecture 2: Linear Algebra Review
EE 227A: Convex Optimization and Applications January 19 Lecture 2: Linear Algebra Review Lecturer: Mert Pilanci Reading assignment: Appendix C of BV. Sections 26 of the web textbook 1 2.1 Vectors 2.1.1
More informationU.C. Berkeley CS294: Spectral Methods and Expanders Handout 11 Luca Trevisan February 29, 2016
U.C. Berkeley CS294: Spectral Methods and Expanders Handout Luca Trevisan February 29, 206 Lecture : ARV In which we introduce semidefinite programming and a semidefinite programming relaxation of sparsest
More informationNormed & Inner Product Vector Spaces
Normed & Inner Product Vector Spaces ECE 174 Introduction to Linear & Nonlinear Optimization Ken KreutzDelgado ECE Department, UC San Diego Ken KreutzDelgado (UC San Diego) ECE 174 Fall 2016 1 / 27 Normed
More informationThe Inverse Function Theorem 1
John Nachbar Washington University April 11, 2014 1 Overview. The Inverse Function Theorem 1 If a function f : R R is C 1 and if its derivative is strictly positive at some x R, then, by continuity of
More informationLinear Analysis Lecture 5
Linear Analysis Lecture 5 Inner Products and V Let dim V < with inner product,. Choose a basis B and let v, w V have coordinates in F n given by x 1. x n and y 1. y n, respectively. Let A F n n be the
More informationCourse Notes for EE227C (Spring 2018): Convex Optimization and Approximation
Course Notes for EE7C (Spring 018): Convex Optimization and Approximation Instructor: Moritz Hardt Email: hardt+ee7c@berkeley.edu Graduate Instructor: Max Simchowitz Email: msimchow+ee7c@berkeley.edu February
More informationUNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems
UNDERGROUND LECTURE NOTES 1: Optimality Conditions for Constrained Optimization Problems Robert M. Freund February 2016 c 2016 Massachusetts Institute of Technology. All rights reserved. 1 1 Introduction
More informationPOLARS AND DUAL CONES
POLARS AND DUAL CONES VERA ROSHCHINA Abstract. The goal of this note is to remind the basic definitions of convex sets and their polars. For more details see the classic references [1, 2] and [3] for polytopes.
More informationFunctional Analysis Review
Outline 9.520: Statistical Learning Theory and Applications February 8, 2010 Outline 1 2 3 4 Vector Space Outline A vector space is a set V with binary operations +: V V V and : R V V such that for all
More informationIntroduction to Convex Analysis Microeconomics II  Tutoring Class
Introduction to Convex Analysis Microeconomics II  Tutoring Class Professor: V. Filipe MartinsdaRocha TA: Cinthia Konichi April 2010 1 Basic Concepts and Results This is a first glance on basic convex
More informationTopology, Math 581, Fall 2017 last updated: November 24, Topology 1, Math 581, Fall 2017: Notes and homework Krzysztof Chris Ciesielski
Topology, Math 581, Fall 2017 last updated: November 24, 2017 1 Topology 1, Math 581, Fall 2017: Notes and homework Krzysztof Chris Ciesielski Class of August 17: Course and syllabus overview. Topology
More informationMath 3202: Midterm 2 Practice Solutions Northwestern University, Winter 2015
Math 30: Midterm Practice Solutions Northwestern University, Winter 015 1. Give an example of each of the following. No justification is needed. (a) A metric on R with respect to which R is bounded. (b)
More informationConvex Analysis and Economic Theory Winter 2018
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory Winter 2018 Topic 0: Vector spaces 0.1 Basic notation Here are some of the fundamental sets and spaces
More informationHomework set 4  Solutions
Homework set 4  Solutions Math 407 Renato Feres 1. Exercise 4.1, page 49 of notes. Let W := T0 m V and denote by GLW the general linear group of W, defined as the group of all linear isomorphisms of W
More informationFUNCTIONAL ANALYSISNORMED SPACE
MAT641 MSC Mathematics, MNIT Jaipur FUNCTIONAL ANALYSISNORMED SPACE DR. RITU AGARWAL MALAVIYA NATIONAL INSTITUTE OF TECHNOLOGY JAIPUR 1. Normed space Norm generalizes the concept of length in an arbitrary
More informationSome Background Math Notes on Limsups, Sets, and Convexity
EE599 STOCHASTIC NETWORK OPTIMIZATION, MICHAEL J. NEELY, FALL 2008 1 Some Background Math Notes on Limsups, Sets, and Convexity I. LIMITS Let f(t) be a real valued function of time. Suppose f(t) converges
More informationChapter 2 Convex Analysis
Chapter 2 Convex Analysis The theory of nonsmooth analysis is based on convex analysis. Thus, we start this chapter by giving basic concepts and results of convexity (for further readings see also [202,
More informationJensen s inequality for multivariate medians
Jensen s inequality for multivariate medians Milan Merkle University of Belgrade, Serbia emerkle@etf.rs Given a probability measure µ on Borel sigmafield of R d, and a function f : R d R, the main issue
More informationConvex Functions. Pontus Giselsson
Convex Functions Pontus Giselsson 1 Today s lecture lower semicontinuity, closure, convex hull convexity preserving operations precomposition with affine mapping infimal convolution image function supremum
More informationMath 209B Homework 2
Math 29B Homework 2 Edward Burkard Note: All vector spaces are over the field F = R or C 4.6. Two Compactness Theorems. 4. Point Set Topology Exercise 6 The product of countably many sequentally compact
More informationNORMS ON SPACE OF MATRICES
NORMS ON SPACE OF MATRICES. Operator Norms on Space of linear maps Let A be an n n real matrix and x 0 be a vector in R n. We would like to use the Picard iteration method to solve for the following system
More informationREVIEW OF ESSENTIAL MATH 346 TOPICS
REVIEW OF ESSENTIAL MATH 346 TOPICS 1. AXIOMATIC STRUCTURE OF R Doğan Çömez The real number system is a complete ordered field, i.e., it is a set R which is endowed with addition and multiplication operations
More informationMAT 419 Lecture Notes Transcribed by Eowyn Cenek 6/1/2012
(Homework 1: Chapter 1: Exercises 17, 9, 11, 19, due Monday June 11th See also the course website for lectures, assignments, etc) Note: today s lecture is primarily about definitions Lots of definitions
More informationSpectral theory for compact operators on Banach spaces
68 Chapter 9 Spectral theory for compact operators on Banach spaces Recall that a subset S of a metric space X is precompact if its closure is compact, or equivalently every sequence contains a Cauchy
More informationStatistics 612: L p spaces, metrics on spaces of probabilites, and connections to estimation
Statistics 62: L p spaces, metrics on spaces of probabilites, and connections to estimation Moulinath Banerjee December 6, 2006 L p spaces and Hilbert spaces We first formally define L p spaces. Consider
More informationUSING FUNCTIONAL ANALYSIS AND SOBOLEV SPACES TO SOLVE POISSON S EQUATION
USING FUNCTIONAL ANALYSIS AND SOBOLEV SPACES TO SOLVE POISSON S EQUATION YI WANG Abstract. We study Banach and Hilbert spaces with an eye towards defining weak solutions to elliptic PDE. Using LaxMilgram
More informationAnalysis and Linear Algebra. Lectures 13 on the mathematical tools that will be used in C103
Analysis and Linear Algebra Lectures 13 on the mathematical tools that will be used in C103 Set Notation A, B sets AcB union A1B intersection A\B the set of objects in A that are not in B N. Empty set
More informationPROBLEMS. (b) (Polarization Identity) Show that in any inner product space
1 Professor Carl Cowen Math 54600 Fall 09 PROBLEMS 1. (Geometry in Inner Product Spaces) (a) (Parallelogram Law) Show that in any inner product space x + y 2 + x y 2 = 2( x 2 + y 2 ). (b) (Polarization
More informationInequality Constraints
Chapter 2 Inequality Constraints 2.1 Optimality Conditions Early in multivariate calculus we learn the significance of differentiability in finding minimizers. In this section we begin our study of the
More informationLecture Notes in Advanced Calculus 1 (80315) Raz Kupferman Institute of Mathematics The Hebrew University
Lecture Notes in Advanced Calculus 1 (80315) Raz Kupferman Institute of Mathematics The Hebrew University February 7, 2007 2 Contents 1 Metric Spaces 1 1.1 Basic definitions...........................
More informationMath 302 Outcome Statements Winter 2013
Math 302 Outcome Statements Winter 2013 1 Rectangular Space Coordinates; Vectors in the ThreeDimensional Space (a) Cartesian coordinates of a point (b) sphere (c) symmetry about a point, a line, and a
More informationExistence of minimizers
Existence of imizers We have just talked a lot about how to find the imizer of an unconstrained convex optimization problem. We have not talked too much, at least not in concrete mathematical terms, about
More informationLocally convex spaces, the hyperplane separation theorem, and the KreinMilman theorem
56 Chapter 7 Locally convex spaces, the hyperplane separation theorem, and the KreinMilman theorem Recall that C(X) is not a normed linear space when X is not compact. On the other hand we could use semi
More informationAnalysis Preliminary Exam Workshop: Hilbert Spaces
Analysis Preliminary Exam Workshop: Hilbert Spaces 1. Hilbert spaces A Hilbert space H is a complete real or complex inner product space. Consider complex Hilbert spaces for definiteness. If (, ) : H H
More informationChapter 7. Extremal Problems. 7.1 Extrema and Local Extrema
Chapter 7 Extremal Problems No matter in theoretical context or in applications many problems can be formulated as problems of finding the maximum or minimum of a function. Whenever this is the case, advanced
More informationCourse Summary Math 211
Course Summary Math 211 table of contents I. Functions of several variables. II. R n. III. Derivatives. IV. Taylor s Theorem. V. Differential Geometry. VI. Applications. 1. Best affine approximations.
More informationSummer School: Semidefinite Optimization
Summer School: Semidefinite Optimization Christine Bachoc Université Bordeaux I, IMB Research Training Group Experimental and Constructive Algebra Haus Karrenberg, Sept. 3  Sept. 7, 2012 Duality Theory
More informationMath Camp Lecture 4: Linear Algebra. Xiao Yu Wang. Aug 2010 MIT. Xiao Yu Wang (MIT) Math Camp /10 1 / 88
Math Camp 2010 Lecture 4: Linear Algebra Xiao Yu Wang MIT Aug 2010 Xiao Yu Wang (MIT) Math Camp 2010 08/10 1 / 88 Linear Algebra Game Plan Vector Spaces Linear Transformations and Matrices Determinant
More informationHelly's Theorem and its Equivalences via Convex Analysis
Portland State University PDXScholar University Honors Theses University Honors College 2014 Helly's Theorem and its Equivalences via Convex Analysis Adam Robinson Portland State University Let us know
More informationNOTES ON MULTIVARIABLE CALCULUS: DIFFERENTIAL CALCULUS
NOTES ON MULTIVARIABLE CALCULUS: DIFFERENTIAL CALCULUS SAMEER CHAVAN Abstract. This is the first part of Notes on Multivariable Calculus based on the classical texts [6] and [5]. We present here the geometric
More informationA NICE PROOF OF FARKAS LEMMA
A NICE PROOF OF FARKAS LEMMA DANIEL VICTOR TAUSK Abstract. The goal of this short note is to present a nice proof of Farkas Lemma which states that if C is the convex cone spanned by a finite set and if
More informationLinear Algebra I. Ronald van Luijk, 2015
Linear Algebra I Ronald van Luijk, 2015 With many parts from Linear Algebra I by Michael Stoll, 2007 Contents Dependencies among sections 3 Chapter 1. Euclidean space: lines and hyperplanes 5 1.1. Definition
More information(1) Consider the space S consisting of all continuous realvalued functions on the closed interval [0, 1]. For f, g S, define
Homework, Real Analysis I, Fall, 2010. (1) Consider the space S consisting of all continuous realvalued functions on the closed interval [0, 1]. For f, g S, define ρ(f, g) = 1 0 f(x) g(x) dx. Show that
More informationLINEAR ALGEBRA W W L CHEN
LINEAR ALGEBRA W W L CHEN c W W L Chen, 1997, 2008. This chapter is available free to all individuals, on the understanding that it is not to be used for financial gain, and may be downloaded and/or photocopied,
More informationChapter 2 Metric Spaces
Chapter 2 Metric Spaces The purpose of this chapter is to present a summary of some basic properties of metric and topological spaces that play an important role in the main body of the book. 2.1 Metrics
More informationStructural and Multidisciplinary Optimization. P. Duysinx and P. Tossings
Structural and Multidisciplinary Optimization P. Duysinx and P. Tossings 20182019 CONTACTS Pierre Duysinx Institut de Mécanique et du Génie Civil (B52/3) Phone number: 04/366.91.94 Email: P.Duysinx@uliege.be
More informationTopological vectorspaces
(July 25, 2011) Topological vectorspaces Paul Garrett garrett@math.umn.edu http://www.math.umn.edu/ garrett/ Natural nonfréchet spaces Topological vector spaces Quotients and linear maps More topological
More informationChapter 6: Orthogonality
Chapter 6: Orthogonality (Last Updated: November 7, 7) These notes are derived primarily from Linear Algebra and its applications by David Lay (4ed). A few theorems have been moved around.. Inner products
More informationVector Spaces, Affine Spaces, and Metric Spaces
Vector Spaces, Affine Spaces, and Metric Spaces 2 This chapter is only meant to give a short overview of the most important concepts in linear algebra, affine spaces, and metric spaces and is not intended
More information10725/36725 Optimization Homework 2 Solutions
10725/36725 Optimization Homework 2 Solutions 1 Convexity (Kevin) 1.1 Sets Let A R n be a closed set with nonempty interior that has a supporting hyperplane at every point on its boundary. (a) Show that
More informationChapter 5. Basics of Euclidean Geometry
Chapter 5 Basics of Euclidean Geometry 5.1 Inner Products, Euclidean Spaces In Affine geometry, it is possible to deal with ratios of vectors and barycenters of points, but there is no way to express the
More informationConvex Functions and Optimization
Chapter 5 Convex Functions and Optimization 5.1 Convex Functions Our next topic is that of convex functions. Again, we will concentrate on the context of a map f : R n R although the situation can be generalized
More informationConvex Geometry. Carsten Schütt
Convex Geometry Carsten Schütt November 25, 2006 2 Contents 0.1 Convex sets... 4 0.2 Separation.... 9 0.3 Extreme points..... 15 0.4 Blaschke selection principle... 18 0.5 Polytopes and polyhedra.... 23
More informationx 1 x 2. x 1, x 2,..., x n R. x n
WEEK In general terms, our aim in this first part of the course is to use vector space theory to study the geometry of Euclidean space A good knowledge of the subject matter of the Matrix Applications
More informationLecture 8 Plus properties, merit functions and gap functions. September 28, 2008
Lecture 8 Plus properties, merit functions and gap functions September 28, 2008 Outline Plusproperties and Funiqueness Equation reformulations of VI/CPs Merit functions Gap merit functions FPI book:
More information