Convex Optimization. (EE227A: UC Berkeley) Lecture 4. Suvrit Sra. (Conjugates, subdifferentials) 31 Jan, 2013
|
|
- Elinor Patrick
- 6 years ago
- Views:
Transcription
1 Convex Optimization (EE227A: UC Berkeley) Lecture 4 (Conjugates, subdifferentials) 31 Jan, 2013 Suvrit Sra
2 Organizational HW1 due: 14th Feb 2013 in class. Please L A TEX your solutions (contact TA if this is an issue) Discussion with classmates is ok Each person must submit his/her individual solutions Acknowledge any help you receive Do not copy! Make sure you understand the solution you submit Cite any source that you use Have fun solving problems! 2 / 30
3 Recap Eigenvalues, singular values, positive definiteness Convex sets, θ 1 x + θ 2 y C, θ 1 + θ 2 = 1, θ i 0 Convex functions, midpoint convex, recognizing convexity Norms, mixed-norms, matrix norms, dual norms Indicator, distance function, minimum of jointly convex Brief mention of other forms of convexity f ( x+y 2 ) 1 2 [f(x) + f(y)] + continuity = f is cvx 2 f(x) 0 implies f is cvx. 3 / 30
4 Fenchel Conjugate 4 / 30
5 Dual norms Def. Let be a norm on R n. Its dual norm is u := sup { u T x x 1 }. 5 / 30
6 Dual norms Def. Let be a norm on R n. Its dual norm is u := sup { u T x x 1 }. Exercise: Verify that we may write u = sup x 0 u T x x 5 / 30
7 Dual norms Def. Let be a norm on R n. Its dual norm is u := sup { u T x x 1 }. Exercise: Verify that we may write u = sup x 0 u T x x Exercise: Verify that u is a norm. 5 / 30
8 Dual norms Def. Let be a norm on R n. Its dual norm is u := sup { u T x x 1 }. Exercise: Verify that we may write u = sup x 0 u T x x Exercise: Verify that u is a norm. u + v = sup { (u + v) T x x 1 } But sup (A + B) sup A + sup B 5 / 30
9 Dual norms Def. Let be a norm on R n. Its dual norm is u := sup { u T x x 1 }. Exercise: Verify that we may write u = sup x 0 u T x x Exercise: Verify that u is a norm. u + v = sup { (u + v) T x x 1 } But sup (A + B) sup A + sup B Exercise: Let 1/p + 1/q = 1, where p, q 1. Show that q is dual to p. In particular, the l 2 -norm is self-dual. 5 / 30
10 Dual norms Def. Let be a norm on R n. Its dual norm is u := sup { u T x x 1 }. Exercise: Verify that we may write u = sup x 0 u T x x Exercise: Verify that u is a norm. u + v = sup { (u + v) T x x 1 } But sup (A + B) sup A + sup B Exercise: Let 1/p + 1/q = 1, where p, q 1. Show that q is dual to p. In particular, the l 2 -norm is self-dual. Hint: Use Hölder s inequality: u T v u p v q 5 / 30
11 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). 6 / 30
12 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! 6 / 30
13 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. 6 / 30
14 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 6 / 30
15 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 6 / 30
16 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α 6 / 30
17 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α Then, z T x x = αz T u αu = α(z T u u ); 6 / 30
18 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α Then, z T x x = αz T u αu = α(z T u u ); 6 / 30
19 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α Then, z T x x = αz T u αu = α(z T u u ); Case (ii): Since z T x x z, 6 / 30
20 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α Then, z T x x = αz T u αu = α(z T u u ); Case (ii): Since z T x x z, x T z x x ( z 1) 0. 6 / 30
21 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α Then, z T x x = αz T u αu = α(z T u u ); Case (ii): Since z T x x z, x T z x x ( z 1) 0. x = 0 maximizes x ( z 1), hence f(z) = 0. 6 / 30
22 Fenchel conjugate Def. The Fenchel conjugate of a function f is f (z) := sup x dom f x T z f(x). Note: f pointwise (over x) sup of linear functions of z, so cvx! Example Let f(x) = x. We have f (z) = I 1(z). That is, conjugate of norm is the indicator function of dual norm ball. Consider two cases: (i) z > 1; (ii) z 1 Case (i), by definition of dual norm (sup over z T u) there is a u s.t. u 1 and z T u > 1 f (z) = sup x x T z f(x). Rewrite x = αu, and let α Then, z T x x = αz T u αu = α(z T u u ); Case (ii): Since z T x x z, x T z x x ( z 1) 0. x = 0 maximizes x ( z 1), hence f(z) = 0. Thus, f(z) = + if (i), and 0 if (ii), as desired. 6 / 30
23 Fenchel conjugate Example f(x) = ax + b; then, f (z) = sup zx (ax + b) x 7 / 30
24 Fenchel conjugate Example f(x) = ax + b; then, f (z) = sup zx (ax + b) x =, if (z a) 0. 7 / 30
25 Fenchel conjugate Example f(x) = ax + b; then, f (z) = sup zx (ax + b) x =, if (z a) 0. Thus, dom f = {a}, and f (a) = b. 7 / 30
26 Fenchel conjugate Example f(x) = ax + b; then, f (z) = sup zx (ax + b) x =, if (z a) 0. Thus, dom f = {a}, and f (a) = b. Example Let a 0, and set f(x) = a 2 x 2 if x a, and + otherwise. Then, f (z) = a 1 + z 2. 7 / 30
27 Fenchel conjugate Example f(x) = ax + b; then, f (z) = sup zx (ax + b) x =, if (z a) 0. Thus, dom f = {a}, and f (a) = b. Example Let a 0, and set f(x) = a 2 x 2 if x a, and + otherwise. Then, f (z) = a 1 + z 2. Example f(x) = 1 2 xt Ax, where A 0. Then, f (z) = 1 2 zt A 1 z. 7 / 30
28 Fenchel conjugate Example f(x) = ax + b; then, f (z) = sup zx (ax + b) x =, if (z a) 0. Thus, dom f = {a}, and f (a) = b. Example Let a 0, and set f(x) = a 2 x 2 if x a, and + otherwise. Then, f (z) = a 1 + z 2. Example f(x) = 1 2 xt Ax, where A 0. Then, f (z) = 1 2 zt A 1 z. Exercise: If f(x) = max(0, 1 x), then dom f is [ 1, 0], and within this domain, f (z) = z. Hint: Analyze cases: max(0, 1 x) = 0; and max(0, 1 x) = 1 x 7 / 30
29 Subdifferentials 8 / 30
30 First order global underestimator f(y) f(x) f(y) + f(y), x y y x f(x) f(y) + f(y), x y 9 / 30
31 First order global underestimator f(x) g 1 f(y) f(y) + g 1, x y g 2 g 3 y f(x) f(y) + g, x y 9 / 30
32 Subgradients g 2 g 3 f(x) g 1 f(y) f(y) + g 1, x y g 1, g 2, g 3 are subgradients at y y 10 / 30
33 Subgradients basic facts f is convex, differentiable: f(y) the unique subgradient at y A vector g is a subgradient at a point y if and only if f(y) + g, x y is globally smaller than f(x). Usually, one subgradient costs approx. as much as f(x) 11 / 30
34 Subgradients basic facts f is convex, differentiable: f(y) the unique subgradient at y A vector g is a subgradient at a point y if and only if f(y) + g, x y is globally smaller than f(x). Usually, one subgradient costs approx. as much as f(x) Determining all subgradients at a given point difficult. Subgradient calculus a great achievement in convex analysis 11 / 30
35 Subgradients basic facts f is convex, differentiable: f(y) the unique subgradient at y A vector g is a subgradient at a point y if and only if f(y) + g, x y is globally smaller than f(x). Usually, one subgradient costs approx. as much as f(x) Determining all subgradients at a given point difficult. Subgradient calculus a great achievement in convex analysis Without convexity, things become wild! advanced course 11 / 30
36 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable 12 / 30
37 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f 1 (x) 12 / 30
38 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f 1 (x) f 2 (x) 12 / 30
39 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f(x) f 1 (x) f 2 (x) 12 / 30
40 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f(x) f 1 (x) f 2 (x) y 12 / 30
41 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f(x) f 1 (x) f 2 (x) f 1 (x) > f 2 (x): unique subgradient of f is f 1 (x) y 12 / 30
42 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f(x) f 1 (x) f 2 (x) f 1 (x) > f 2 (x): unique subgradient of f is f 1 (x) f 1 (x) < f 2 (x): unique subgradient of f is f 2 (x) y 12 / 30
43 Subgradients example f(x) := max(f 1 (x), f 2 (x)); both f 1, f 2 convex, differentiable f(x) f 1 (x) f 2 (x) f 1 (x) > f 2 (x): unique subgradient of f is f 1 (x) f 1 (x) < f 2 (x): unique subgradient of f is f 2 (x) f 1 (y) = f 2 (y): subgradients, the segment [f 1 (y), f 2 (y)] (imagine all supporting lines turning about point y) y 12 / 30
44 Subgradients Def. A vector g R n is called a subgradient at a point y, if for all x dom f, it holds that f(x) f(y) + g, x y 13 / 30
45 Subdifferential Def. The set of all subgradients at y denoted by f(y). This set is called subdifferential of f at y 14 / 30
46 Subdifferential Def. The set of all subgradients at y denoted by f(y). This set is called subdifferential of f at y If f is convex, f(x) is nice: If x relative interior of dom f, then f(x) nonempty 14 / 30
47 Subdifferential Def. The set of all subgradients at y denoted by f(y). This set is called subdifferential of f at y If f is convex, f(x) is nice: If x relative interior of dom f, then f(x) nonempty If f differentiable at x, then f(x) = { f(x)} 14 / 30
48 Subdifferential Def. The set of all subgradients at y denoted by f(y). This set is called subdifferential of f at y If f is convex, f(x) is nice: If x relative interior of dom f, then f(x) nonempty If f differentiable at x, then f(x) = { f(x)} If f(x) = {g}, then f is differentiable and g = f(x) 14 / 30
49 Subdifferential example f(x) = x 15 / 30
50 Subdifferential example f(x) = x +1 f(x) 1 x 15 / 30
51 Subdifferential example f(x) = x +1 f(x) 1 x 1 x < 0, x = +1 x > 0, [ 1, 1] x = / 30
52 More examples Example f(x) = x 2. Then, { x 1 2 f(x) := x x 0, {z z 2 1} x = / 30
53 More examples Example f(x) = x 2. Then, { x 1 2 f(x) := x x 0, {z z 2 1} x = 0. Proof. z 2 x 2 + g, z x 16 / 30
54 More examples Example f(x) = x 2. Then, { x 1 2 f(x) := x x 0, {z z 2 1} x = 0. Proof. z 2 x 2 + g, z x z 2 g, z 16 / 30
55 More examples Example f(x) = x 2. Then, { x 1 2 f(x) := x x 0, {z z 2 1} x = 0. Proof. z 2 x 2 + g, z x z 2 g, z = g / 30
56 More examples Example A convex function need not be subdifferentiable everywhere. Let { (1 x 2 2 f(x) := )1/2 if x 2 1, + otherwise. f diff. for all x with x 2 < 1, but f(x) = whenever x / 30
57 Calculus 18 / 30
58 Recall basic calculus If f and k are differentiable, we know that Addition: (f + k)(x) = f(x) + k(x) Scaling: (αf(x)) = α f(x) 19 / 30
59 Recall basic calculus If f and k are differentiable, we know that Addition: (f + k)(x) = f(x) + k(x) Scaling: (αf(x)) = α f(x) Chain rule If f : R n R m, and k : R m R p. Let h : R n R p be the composition h(x) = (k f)(x) = k(f(x)). Then, Dh(x) = Dk(f(x))Df(x). 19 / 30
60 Recall basic calculus If f and k are differentiable, we know that Addition: (f + k)(x) = f(x) + k(x) Scaling: (αf(x)) = α f(x) Chain rule If f : R n R m, and k : R m R p. Let h : R n R p be the composition h(x) = (k f)(x) = k(f(x)). Then, Dh(x) = Dk(f(x))Df(x). Example If f : R n R and k : R R, then using the fact that h(x) = [Dh(x)] T, we obtain h(x) = k (f(x)) f(x). 19 / 30
61 Subgradient calculus Finding one subgradient within f(x) 20 / 30
62 Subgradient calculus Finding one subgradient within f(x) Determining entire subdifferential f(x) at a point x 20 / 30
63 Subgradient calculus Finding one subgradient within f(x) Determining entire subdifferential f(x) at a point x Do we have the chain rule? 20 / 30
64 Subgradient calculus Finding one subgradient within f(x) Determining entire subdifferential f(x) at a point x Do we have the chain rule? Usually not easy! 20 / 30
65 Subgradient calculus If f is differentiable, f(x) = { f(x)} 21 / 30
66 Subgradient calculus If f is differentiable, f(x) = { f(x)} Scaling α > 0, (αf)(x) = α f(x) = {αg g f(x)} 21 / 30
67 Subgradient calculus If f is differentiable, f(x) = { f(x)} Scaling α > 0, (αf)(x) = α f(x) = {αg g f(x)} Addition : (f + k)(x) = f(x) + k(x) (set addition) 21 / 30
68 Subgradient calculus If f is differentiable, f(x) = { f(x)} Scaling α > 0, (αf)(x) = α f(x) = {αg g f(x)} Addition : (f + k)(x) = f(x) + k(x) (set addition) Chain rule : Let A R m n, b R m, f : R m R, and h : R n R be given by h(x) = f(ax + b). Then, h(x) = A T f(ax + b). 21 / 30
69 Subgradient calculus If f is differentiable, f(x) = { f(x)} Scaling α > 0, (αf)(x) = α f(x) = {αg g f(x)} Addition : (f + k)(x) = f(x) + k(x) (set addition) Chain rule : Let A R m n, b R m, f : R m R, and h : R n R be given by h(x) = f(ax + b). Then, h(x) = A T f(ax + b). Chain rule : h(x) = f k, where k : X Y is diff. h(x) = f(k(x)) Dk(x) = [Dk(x)] T f(k(x)) 21 / 30
70 Subgradient calculus If f is differentiable, f(x) = { f(x)} Scaling α > 0, (αf)(x) = α f(x) = {αg g f(x)} Addition : (f + k)(x) = f(x) + k(x) (set addition) Chain rule : Let A R m n, b R m, f : R m R, and h : R n R be given by h(x) = f(ax + b). Then, h(x) = A T f(ax + b). Chain rule : h(x) = f k, where k : X Y is diff. h(x) = f(k(x)) Dk(x) = [Dk(x)] T f(k(x)) Max function : If f(x) := max 1 i m f i (x), then f(x) = conv { f i (x) f i (x) = f(x)}, convex hull over subdifferentials of active functions at x 21 / 30
71 Subgradient calculus If f is differentiable, f(x) = { f(x)} Scaling α > 0, (αf)(x) = α f(x) = {αg g f(x)} Addition : (f + k)(x) = f(x) + k(x) (set addition) Chain rule : Let A R m n, b R m, f : R m R, and h : R n R be given by h(x) = f(ax + b). Then, h(x) = A T f(ax + b). Chain rule : h(x) = f k, where k : X Y is diff. h(x) = f(k(x)) Dk(x) = [Dk(x)] T f(k(x)) Max function : If f(x) := max 1 i m f i (x), then f(x) = conv { f i (x) f i (x) = f(x)}, convex hull over subdifferentials of active functions at x Conjugation: z f(x) if and only if x f (z) 21 / 30
72 Examples It can happen that (f 1 + f 2 ) f 1 + f 2 22 / 30
73 Examples It can happen that (f 1 + f 2 ) f 1 + f 2 Example Define f 1 and f 2 by { 2 x if x 0, f 1 (x) := + if x < 0, and f 2 (x) := { + if x > 0, 2 x if x 0. Then, f = max {f 1, f 2 } = I 0, whereby f(0) = R But f 1 (0) = f 2 (0) =. 22 / 30
74 Examples It can happen that (f 1 + f 2 ) f 1 + f 2 Example Define f 1 and f 2 by { 2 x if x 0, f 1 (x) := + if x < 0, and f 2 (x) := { + if x > 0, 2 x if x 0. Then, f = max {f 1, f 2 } = I 0, whereby f(0) = R But f 1 (0) = f 2 (0) =. However, f 1 (x) + f 2 (x) (f 1 + f 2 )(x) always holds. 22 / 30
75 Examples Example f(x) = x. Then, f(0) = conv {±e 1,..., ±e n }, where e i is i-th canonical basis vector (column of identity matrix). 23 / 30
76 Examples Example f(x) = x. Then, f(0) = conv {±e 1,..., ±e n }, where e i is i-th canonical basis vector (column of identity matrix). { To prove, notice that f(x) = max 1 i n e T i x }. 23 / 30
77 Example (S. Boyd) Example Let f(x) = max { s T x s i { 1, 1} } (2 n members) 24 / 30
78 Example (S. Boyd) Example Let f(x) = max { s T x s i { 1, 1} } (2 n members) ( 1, 1) (1, 1) f at x = (0, 0) 24 / 30
79 Example (S. Boyd) Example Let f(x) = max { s T x s i { 1, 1} } (2 n members) ( 1, 1) +1 1 (1, 1) 1 f at x = (0, 0) f at x = (1, 0) 24 / 30
80 Example (S. Boyd) Example Let f(x) = max { s T x s i { 1, 1} } (2 n members) ( 1, 1) (1, 1) 1 (1, 1) 1 1 f at x = (0, 0) f at x = (1, 0) f at x = (1, 1) 24 / 30
81 Rules for subgradients 25 / 30
82 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! 26 / 30
83 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! Simple way to obtain some g f(x): 26 / 30
84 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! Simple way to obtain some g f(x): Pick any y for which h(x, y ) = f(x) 26 / 30
85 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! Simple way to obtain some g f(x): Pick any y for which h(x, y ) = f(x) Pick any subgradient g h(x, y ) 26 / 30
86 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! Simple way to obtain some g f(x): Pick any y for which h(x, y ) = f(x) Pick any subgradient g h(x, y ) This g f(x) 26 / 30
87 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! Simple way to obtain some g f(x): Pick any y for which h(x, y ) = f(x) Pick any subgradient g h(x, y ) This g f(x) h(z, y ) h(x, y ) + g T (z x) h(z, y ) f(x) + g T (z x) 26 / 30
88 Subgradient for pointwise sup f(x) := sup y Y h(x, y) Getting f(x) is complicated! Simple way to obtain some g f(x): Pick any y for which h(x, y ) = f(x) Pick any subgradient g h(x, y ) This g f(x) h(z, y ) h(x, y ) + g T (z x) h(z, y ) f(x) + g T (z x) f(z) h(z, y) (because of sup) f(z) f(x) + g T (z x). 26 / 30
89 Example Suppose a i R n and b i R. And f(x) := max 1 i n (at i x + b i ). This f a max (in fact, over a finite number of terms) 27 / 30
90 Example Suppose a i R n and b i R. And f(x) := max 1 i n (at i x + b i ). This f a max (in fact, over a finite number of terms) Suppose f(x) = a T k x + b k for some index k 27 / 30
91 Example Suppose a i R n and b i R. And f(x) := max 1 i n (at i x + b i ). This f a max (in fact, over a finite number of terms) Suppose f(x) = a T k x + b k for some index k Here f(x; y) = f k (x) = a T k x + b k, and f k (x) = { f k (x)} 27 / 30
92 Example Suppose a i R n and b i R. And f(x) := max 1 i n (at i x + b i ). This f a max (in fact, over a finite number of terms) Suppose f(x) = a T k x + b k for some index k Here f(x; y) = f k (x) = a T k x + b k, and f k (x) = { f k (x)} Hence, a k f(x) works! 27 / 30
93 Subgradient of expectation Suppose f = Ef(x, u), where f is convex in x for each u (an r.v.) f(x) := f(x, u)p(u)du 28 / 30
94 Subgradient of expectation Suppose f = Ef(x, u), where f is convex in x for each u (an r.v.) f(x) := f(x, u)p(u)du For each u choose any g(x, u) x f(x, u) 28 / 30
95 Subgradient of expectation Suppose f = Ef(x, u), where f is convex in x for each u (an r.v.) f(x) := f(x, u)p(u)du For each u choose any g(x, u) x f(x, u) Then, g = g(x, u)p(u)du = Eg(x, u) f(x) 28 / 30
96 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). 29 / 30
97 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). To find a vector g f(x), we may: 29 / 30
98 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). To find a vector g f(x), we may: For i = 1 to n, compute g i f i (x) 29 / 30
99 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). To find a vector g f(x), we may: For i = 1 to n, compute g i f i (x) Compute u h(f 1 (x),..., f n (x)) 29 / 30
100 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). To find a vector g f(x), we may: For i = 1 to n, compute g i f i (x) Compute u h(f 1 (x),..., f n (x)) Set g = u 1 g 1 + u 2 g u n g n ; this g f(x) 29 / 30
101 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). To find a vector g f(x), we may: For i = 1 to n, compute g i f i (x) Compute u h(f 1 (x),..., f n (x)) Set g = u 1 g 1 + u 2 g u n g n ; this g f(x) Compare with f(x) = J h(x), where J matrix of f i (x) 29 / 30
102 Subgradient of composition Suppose h : R n R cvx and nondecreasing; each f i cvx f(x) := h(f 1 (x), f 2 (x),..., f n (x)). To find a vector g f(x), we may: For i = 1 to n, compute g i f i (x) Compute u h(f 1 (x),..., f n (x)) Set g = u 1 g 1 + u 2 g u n g n ; this g f(x) Compare with f(x) = J h(x), where J matrix of f i (x) Exercise: Verify g f(x) by showing f(z) f(x) + g T (z x) 29 / 30
103 References 1 R. T. Rockafellar. Convex Analysis 2 S. Boyd (Stanford); EE364b Lecture Notes. 30 / 30
Subgradients. subgradients. strong and weak subgradient calculus. optimality conditions via subgradients. directional derivatives
Subgradients subgradients strong and weak subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE364b, Stanford University Basic inequality recall basic inequality
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 4. Subgradient
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 4 Subgradient Shiqian Ma, MAT-258A: Numerical Optimization 2 4.1. Subgradients definition subgradient calculus duality and optimality conditions Shiqian
More informationSubgradients. subgradients and quasigradients. subgradient calculus. optimality conditions via subgradients. directional derivatives
Subgradients subgradients and quasigradients subgradient calculus optimality conditions via subgradients directional derivatives Prof. S. Boyd, EE392o, Stanford University Basic inequality recall basic
More informationLecture 1: Background on Convex Analysis
Lecture 1: Background on Convex Analysis John Duchi PCMI 2016 Outline I Convex sets 1.1 Definitions and examples 2.2 Basic properties 3.3 Projections onto convex sets 4.4 Separating and supporting hyperplanes
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - A) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationSubgradient. Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes. definition. subgradient calculus
1/41 Subgradient Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes definition subgradient calculus duality and optimality conditions directional derivative Basic inequality
More informationLECTURE 12 LECTURE OUTLINE. Subgradients Fenchel inequality Sensitivity in constrained optimization Subdifferential calculus Optimality conditions
LECTURE 12 LECTURE OUTLINE Subgradients Fenchel inequality Sensitivity in constrained optimization Subdifferential calculus Optimality conditions Reading: Section 5.4 All figures are courtesy of Athena
More informationLecture 6: September 12
10-725: Optimization Fall 2013 Lecture 6: September 12 Lecturer: Ryan Tibshirani Scribes: Micol Marchetti-Bowick Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not
More informationOptimality Conditions for Nonsmooth Convex Optimization
Optimality Conditions for Nonsmooth Convex Optimization Sangkyun Lee Oct 22, 2014 Let us consider a convex function f : R n R, where R is the extended real field, R := R {, + }, which is proper (f never
More informationConvex Analysis Background
Convex Analysis Background John C. Duchi Stanford University Park City Mathematics Institute 206 Abstract In this set of notes, we will outline several standard facts from convex analysis, the study of
More informationLecture 6: September 17
10-725/36-725: Convex Optimization Fall 2015 Lecturer: Ryan Tibshirani Lecture 6: September 17 Scribes: Scribes: Wenjun Wang, Satwik Kottur, Zhiding Yu Note: LaTeX template courtesy of UC Berkeley EECS
More informationExtreme Abridgment of Boyd and Vandenberghe s Convex Optimization
Extreme Abridgment of Boyd and Vandenberghe s Convex Optimization Compiled by David Rosenberg Abstract Boyd and Vandenberghe s Convex Optimization book is very well-written and a pleasure to read. The
More informationLecture 2: Convex Sets and Functions
Lecture 2: Convex Sets and Functions Hyang-Won Lee Dept. of Internet & Multimedia Eng. Konkuk University Lecture 2 Network Optimization, Fall 2015 1 / 22 Optimization Problems Optimization problems are
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 6. Suvrit Sra. (Conic optimization) 07 Feb, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 6 (Conic optimization) 07 Feb, 2013 Suvrit Sra Organizational Info Quiz coming up on 19th Feb. Project teams by 19th Feb Good if you can mix your research
More information8. Conjugate functions
L. Vandenberghe EE236C (Spring 2013-14) 8. Conjugate functions closed functions conjugate function 8-1 Closed set a set C is closed if it contains its boundary: x k C, x k x = x C operations that preserve
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 28. Suvrit Sra. (Algebra + Optimization) 02 May, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 28 (Algebra + Optimization) 02 May, 2013 Suvrit Sra Admin Poster presentation on 10th May mandatory HW, Midterm, Quiz to be reweighted Project final report
More informationMath 273a: Optimization Subgradients of convex functions
Math 273a: Optimization Subgradients of convex functions Made by: Damek Davis Edited by Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com 1 / 20 Subgradients Assumptions
More informationOptimization and Optimal Control in Banach Spaces
Optimization and Optimal Control in Banach Spaces Bernhard Schmitzer October 19, 2017 1 Convex non-smooth optimization with proximal operators Remark 1.1 (Motivation). Convex optimization: easier to solve,
More informationLocal strong convexity and local Lipschitz continuity of the gradient of convex functions
Local strong convexity and local Lipschitz continuity of the gradient of convex functions R. Goebel and R.T. Rockafellar May 23, 2007 Abstract. Given a pair of convex conjugate functions f and f, we investigate
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationCoordinate Update Algorithm Short Course Subgradients and Subgradient Methods
Coordinate Update Algorithm Short Course Subgradients and Subgradient Methods Instructor: Wotao Yin (UCLA Math) Summer 2016 1 / 30 Notation f : H R { } is a closed proper convex function domf := {x R n
More informationBASICS OF CONVEX ANALYSIS
BASICS OF CONVEX ANALYSIS MARKUS GRASMAIR 1. Main Definitions We start with providing the central definitions of convex functions and convex sets. Definition 1. A function f : R n R + } is called convex,
More informationConvex Analysis and Economic Theory AY Elementary properties of convex functions
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory AY 2018 2019 Topic 6: Convex functions I 6.1 Elementary properties of convex functions We may occasionally
More informationPrimal/Dual Decomposition Methods
Primal/Dual Decomposition Methods Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2018-19, HKUST, Hong Kong Outline of Lecture Subgradients
More informationConvex Optimization. (EE227A: UC Berkeley) Lecture 15. Suvrit Sra. (Gradient methods III) 12 March, 2013
Convex Optimization (EE227A: UC Berkeley) Lecture 15 (Gradient methods III) 12 March, 2013 Suvrit Sra Optimal gradient methods 2 / 27 Optimal gradient methods We saw following efficiency estimates for
More informationLecture: Duality of LP, SOCP and SDP
1/33 Lecture: Duality of LP, SOCP and SDP Zaiwen Wen Beijing International Center For Mathematical Research Peking University http://bicmr.pku.edu.cn/~wenzw/bigdata2017.html wenzw@pku.edu.cn Acknowledgement:
More information3. Convex functions. basic properties and examples. operations that preserve convexity. the conjugate function. quasiconvex functions
3. Convex functions Convex Optimization Boyd & Vandenberghe basic properties and examples operations that preserve convexity the conjugate function quasiconvex functions log-concave and log-convex functions
More informationThe proximal mapping
The proximal mapping http://bicmr.pku.edu.cn/~wenzw/opt-2016-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/37 1 closed function 2 Conjugate function
More informationFunctional Analysis Exercise Class
Functional Analysis Exercise Class Week 9 November 13 November Deadline to hand in the homeworks: your exercise class on week 16 November 20 November Exercises (1) Show that if T B(X, Y ) and S B(Y, Z)
More informationEE 546, Univ of Washington, Spring Proximal mapping. introduction. review of conjugate functions. proximal mapping. Proximal mapping 6 1
EE 546, Univ of Washington, Spring 2012 6. Proximal mapping introduction review of conjugate functions proximal mapping Proximal mapping 6 1 Proximal mapping the proximal mapping (prox-operator) of a convex
More informationDivision of the Humanities and Social Sciences. Supergradients. KC Border Fall 2001 v ::15.45
Division of the Humanities and Social Sciences Supergradients KC Border Fall 2001 1 The supergradient of a concave function There is a useful way to characterize the concavity of differentiable functions.
More informationDesign and Analysis of Algorithms Lecture Notes on Convex Optimization CS 6820, Fall Nov 2 Dec 2016
Design and Analysis of Algorithms Lecture Notes on Convex Optimization CS 6820, Fall 206 2 Nov 2 Dec 206 Let D be a convex subset of R n. A function f : D R is convex if it satisfies f(tx + ( t)y) tf(x)
More informationConvex envelopes, cardinality constrained optimization and LASSO. An application in supervised learning: support vector machines (SVMs)
ORF 523 Lecture 8 Princeton University Instructor: A.A. Ahmadi Scribe: G. Hall Any typos should be emailed to a a a@princeton.edu. 1 Outline Convexity-preserving operations Convex envelopes, cardinality
More informationDedicated to Michel Théra in honor of his 70th birthday
VARIATIONAL GEOMETRIC APPROACH TO GENERALIZED DIFFERENTIAL AND CONJUGATE CALCULI IN CONVEX ANALYSIS B. S. MORDUKHOVICH 1, N. M. NAM 2, R. B. RECTOR 3 and T. TRAN 4. Dedicated to Michel Théra in honor of
More informationDual Decomposition.
1/34 Dual Decomposition http://bicmr.pku.edu.cn/~wenzw/opt-2017-fall.html Acknowledgement: this slides is based on Prof. Lieven Vandenberghes lecture notes Outline 2/34 1 Conjugate function 2 introduction:
More informationGEOMETRIC APPROACH TO CONVEX SUBDIFFERENTIAL CALCULUS October 10, Dedicated to Franco Giannessi and Diethard Pallaschke with great respect
GEOMETRIC APPROACH TO CONVEX SUBDIFFERENTIAL CALCULUS October 10, 2018 BORIS S. MORDUKHOVICH 1 and NGUYEN MAU NAM 2 Dedicated to Franco Giannessi and Diethard Pallaschke with great respect Abstract. In
More informationIn Progress: Summary of Notation and Basic Results Convex Analysis C&O 663, Fall 2009
In Progress: Summary of Notation and Basic Results Convex Analysis C&O 663, Fall 2009 Intructor: Henry Wolkowicz November 26, 2009 University of Waterloo Department of Combinatorics & Optimization Abstract
More informationConvex analysis and profit/cost/support functions
Division of the Humanities and Social Sciences Convex analysis and profit/cost/support functions KC Border October 2004 Revised January 2009 Let A be a subset of R m Convex analysts may give one of two
More informationContinuity of convex functions in normed spaces
Continuity of convex functions in normed spaces In this chapter, we consider continuity properties of real-valued convex functions defined on open convex sets in normed spaces. Recall that every infinitedimensional
More informationLecture 2: Convex functions
Lecture 2: Convex functions f : R n R is convex if dom f is convex and for all x, y dom f, θ [0, 1] f is concave if f is convex f(θx + (1 θ)y) θf(x) + (1 θ)f(y) x x convex concave neither x examples (on
More informationHelly's Theorem and its Equivalences via Convex Analysis
Portland State University PDXScholar University Honors Theses University Honors College 2014 Helly's Theorem and its Equivalences via Convex Analysis Adam Robinson Portland State University Let us know
More informationIE 521 Convex Optimization Homework #1 Solution
IE 521 Convex Optimization Homework #1 Solution your NAME here your NetID here February 13, 2019 Instructions. Homework is due Wednesday, February 6, at 1:00pm; no late homework accepted. Please use the
More informationminimize x x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x 2 u 2, 5x 1 +76x 2 1,
4 Duality 4.1 Numerical perturbation analysis example. Consider the quadratic program with variables x 1, x 2, and parameters u 1, u 2. minimize x 2 1 +2x2 2 x 1x 2 x 1 subject to x 1 +2x 2 u 1 x 1 4x
More informationHence, (f(x) f(x 0 )) 2 + (g(x) g(x 0 )) 2 < ɛ
Matthew Straughn Math 402 Homework 5 Homework 5 (p. 429) 13.3.5, 13.3.6 (p. 432) 13.4.1, 13.4.2, 13.4.7*, 13.4.9 (p. 448-449) 14.2.1, 14.2.2 Exercise 13.3.5. Let (X, d X ) be a metric space, and let f
More informationLecture 3. Optimization Problems and Iterative Algorithms
Lecture 3 Optimization Problems and Iterative Algorithms January 13, 2016 This material was jointly developed with Angelia Nedić at UIUC for IE 598ns Outline Special Functions: Linear, Quadratic, Convex
More informationConvex Optimization Theory. Chapter 4 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 4 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More informationConvex Functions. Daniel P. Palomar. Hong Kong University of Science and Technology (HKUST)
Convex Functions Daniel P. Palomar Hong Kong University of Science and Technology (HKUST) ELEC5470 - Convex Optimization Fall 2017-18, HKUST, Hong Kong Outline of Lecture Definition convex function Examples
More informationOn the interior of the simplex, we have the Hessian of d(x), Hd(x) is diagonal with ith. µd(w) + w T c. minimize. subject to w T 1 = 1,
Math 30 Winter 05 Solution to Homework 3. Recognizing the convexity of g(x) := x log x, from Jensen s inequality we get d(x) n x + + x n n log x + + x n n where the equality is attained only at x = (/n,...,
More informationMath 361: Homework 1 Solutions
January 3, 4 Math 36: Homework Solutions. We say that two norms and on a vector space V are equivalent or comparable if the topology they define on V are the same, i.e., for any sequence of vectors {x
More informationLagrangian Duality and Convex Optimization
Lagrangian Duality and Convex Optimization David Rosenberg New York University February 11, 2015 David Rosenberg (New York University) DS-GA 1003 February 11, 2015 1 / 24 Introduction Why Convex Optimization?
More informationFALL 2018 MATH 4211/6211 Optimization Homework 1
FALL 2018 MATH 4211/6211 Optimization Homework 1 This homework assignment is open to textbook, reference books, slides, and online resources, excluding any direct solution to the problem (such as solution
More informationSubdifferential representation of convex functions: refinements and applications
Subdifferential representation of convex functions: refinements and applications Joël Benoist & Aris Daniilidis Abstract Every lower semicontinuous convex function can be represented through its subdifferential
More informationFunctions of Several Variables
Jim Lambers MAT 419/519 Summer Session 2011-12 Lecture 2 Notes These notes correspond to Section 1.2 in the text. Functions of Several Variables We now generalize the results from the previous section,
More informationConvex Optimization Conjugate, Subdifferential, Proximation
1 Lecture Notes, HCI, 3.11.211 Chapter 6 Convex Optimization Conjugate, Subdifferential, Proximation Bastian Goldlücke Computer Vision Group Technical University of Munich 2 Bastian Goldlücke Overview
More informationLECTURE 4 LECTURE OUTLINE
LECTURE 4 LECTURE OUTLINE Relative interior and closure Algebra of relative interiors and closures Continuity of convex functions Closures of functions Reading: Section 1.3 All figures are courtesy of
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 Minimum volume ellipsoid around a set Löwner-John ellipsoid
More informationMath 273a: Optimization Subgradient Methods
Math 273a: Optimization Subgradient Methods Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Nonsmooth convex function Recall: For ˉx R n, f(ˉx) := {g R
More informationLecture 8. Strong Duality Results. September 22, 2008
Strong Duality Results September 22, 2008 Outline Lecture 8 Slater Condition and its Variations Convex Objective with Linear Inequality Constraints Quadratic Objective over Quadratic Constraints Representation
More informationConvex Optimization Lecture 6: KKT Conditions, and applications
Convex Optimization Lecture 6: KKT Conditions, and applications Dr. Michel Baes, IFOR / ETH Zürich Quick recall of last week s lecture Various aspects of convexity: The set of minimizers is convex. Convex
More informationThe sum of two maximal monotone operator is of type FPV
CJMS. 5(1)(2016), 17-21 Caspian Journal of Mathematical Sciences (CJMS) University of Mazandaran, Iran http://cjms.journals.umz.ac.ir ISSN: 1735-0611 The sum of two maximal monotone operator is of type
More informationDual and primal-dual methods
ELE 538B: Large-Scale Optimization for Data Science Dual and primal-dual methods Yuxin Chen Princeton University, Spring 2018 Outline Dual proximal gradient method Primal-dual proximal gradient method
More informationMidterm 1 Review. Distance = (x 1 x 0 ) 2 + (y 1 y 0 ) 2.
Midterm 1 Review Comments about the midterm The midterm will consist of five questions and will test on material from the first seven lectures the material given below. No calculus either single variable
More informationCHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS. W. Erwin Diewert January 31, 2008.
1 ECONOMICS 594: LECTURE NOTES CHAPTER 2: CONVEX SETS AND CONCAVE FUNCTIONS W. Erwin Diewert January 31, 2008. 1. Introduction Many economic problems have the following structure: (i) a linear function
More informationMAXIMALITY OF SUMS OF TWO MAXIMAL MONOTONE OPERATORS
MAXIMALITY OF SUMS OF TWO MAXIMAL MONOTONE OPERATORS JONATHAN M. BORWEIN, FRSC Abstract. We use methods from convex analysis convex, relying on an ingenious function of Simon Fitzpatrick, to prove maximality
More informationEE Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko, JPL Third Term
EE 150 - Applications of Convex Optimization in Signal Processing and Communications Dr. Andre Tkacenko JPL Third Term 2011-2012 Due on Thursday May 3 in class. Homework Set #4 1. (10 points) (Adapted
More informationA SET OF LECTURE NOTES ON CONVEX OPTIMIZATION WITH SOME APPLICATIONS TO PROBABILITY THEORY INCOMPLETE DRAFT. MAY 06
A SET OF LECTURE NOTES ON CONVEX OPTIMIZATION WITH SOME APPLICATIONS TO PROBABILITY THEORY INCOMPLETE DRAFT. MAY 06 CHRISTIAN LÉONARD Contents Preliminaries 1 1. Convexity without topology 1 2. Convexity
More information5. Subgradient method
L. Vandenberghe EE236C (Spring 2016) 5. Subgradient method subgradient method convergence analysis optimal step size when f is known alternating projections optimality 5-1 Subgradient method to minimize
More informationChapter 2 Convex Analysis
Chapter 2 Convex Analysis The theory of nonsmooth analysis is based on convex analysis. Thus, we start this chapter by giving basic concepts and results of convexity (for further readings see also [202,
More informationConditional Gradient (Frank-Wolfe) Method
Conditional Gradient (Frank-Wolfe) Method Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 1 Outline Today: Conditional gradient method Convergence analysis Properties
More informationLecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem
Lecture 3: Lagrangian duality and algorithms for the Lagrangian dual problem Michael Patriksson 0-0 The Relaxation Theorem 1 Problem: find f := infimum f(x), x subject to x S, (1a) (1b) where f : R n R
More informationConvex optimization COMS 4771
Convex optimization COMS 4771 1. Recap: learning via optimization Soft-margin SVMs Soft-margin SVM optimization problem defined by training data: w R d λ 2 w 2 2 + 1 n n [ ] 1 y ix T i w. + 1 / 15 Soft-margin
More informationConvex Functions. Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK)
Convex Functions Wing-Kin (Ken) Ma The Chinese University of Hong Kong (CUHK) Course on Convex Optimization for Wireless Comm. and Signal Proc. Jointly taught by Daniel P. Palomar and Wing-Kin (Ken) Ma
More information9. Geometric problems
9. Geometric problems EE/AA 578, Univ of Washington, Fall 2016 projection on a set extremal volume ellipsoids centering classification 9 1 Projection on convex set projection of point x on set C defined
More informationLinear and non-linear programming
Linear and non-linear programming Benjamin Recht March 11, 2005 The Gameplan Constrained Optimization Convexity Duality Applications/Taxonomy 1 Constrained Optimization minimize f(x) subject to g j (x)
More informationALGORITHMS FOR MINIMIZING DIFFERENCES OF CONVEX FUNCTIONS AND APPLICATIONS
ALGORITHMS FOR MINIMIZING DIFFERENCES OF CONVEX FUNCTIONS AND APPLICATIONS Mau Nam Nguyen (joint work with D. Giles and R. B. Rector) Fariborz Maseeh Department of Mathematics and Statistics Portland State
More informationConvex Optimization Theory. Chapter 5 Exercises and Solutions: Extended Version
Convex Optimization Theory Chapter 5 Exercises and Solutions: Extended Version Dimitri P. Bertsekas Massachusetts Institute of Technology Athena Scientific, Belmont, Massachusetts http://www.athenasc.com
More information1 Sparsity and l 1 relaxation
6.883 Learning with Combinatorial Structure Note for Lecture 2 Author: Chiyuan Zhang Sparsity and l relaxation Last time we talked about sparsity and characterized when an l relaxation could recover the
More informationLECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE
LECTURE 25: REVIEW/EPILOGUE LECTURE OUTLINE CONVEX ANALYSIS AND DUALITY Basic concepts of convex analysis Basic concepts of convex optimization Geometric duality framework - MC/MC Constrained optimization
More informationAssignment 1: From the Definition of Convexity to Helley Theorem
Assignment 1: From the Definition of Convexity to Helley Theorem Exercise 1 Mark in the following list the sets which are convex: 1. {x R 2 : x 1 + i 2 x 2 1, i = 1,..., 10} 2. {x R 2 : x 2 1 + 2ix 1x
More informationConvex Analysis and Economic Theory Fall 2018 Winter 2019
Division of the Humanities and Social Sciences Ec 181 KC Border Convex Analysis and Economic Theory Fall 2018 Winter 2019 Topic 18: Differentiability 18.1 Differentiable functions In this section I want
More informationFinal exam.
EE364b Convex Optimization II June 018 Prof. John C. Duchi Final exam By now, you know how it works, so we won t repeat it here. (If not, see the instructions for the EE364a final exam.) Since you have
More informationMath 273a: Optimization Convex Conjugacy
Math 273a: Optimization Convex Conjugacy Instructor: Wotao Yin Department of Mathematics, UCLA Fall 2015 online discussions on piazza.com Convex conjugate (the Legendre transform) Let f be a closed proper
More informationconverges as well if x < 1. 1 x n x n 1 1 = 2 a nx n
Solve the following 6 problems. 1. Prove that if series n=1 a nx n converges for all x such that x < 1, then the series n=1 a n xn 1 x converges as well if x < 1. n For x < 1, x n 0 as n, so there exists
More informationg 2 (x) (1/3)M 1 = (1/3)(2/3)M.
COMPACTNESS If C R n is closed and bounded, then by B-W it is sequentially compact: any sequence of points in C has a subsequence converging to a point in C Conversely, any sequentially compact C R n is
More informationDuality and dynamics in Hamilton-Jacobi theory for fully convex problems of control
Duality and dynamics in Hamilton-Jacobi theory for fully convex problems of control RTyrrell Rockafellar and Peter R Wolenski Abstract This paper describes some recent results in Hamilton- Jacobi theory
More information8. Geometric problems
8. Geometric problems Convex Optimization Boyd & Vandenberghe extremal volume ellipsoids centering classification placement and facility location 8 1 Minimum volume ellipsoid around a set Löwner-John ellipsoid
More informationHomework 2. Convex Optimization /36-725
Homework 2 Convex Optimization 0-725/36-725 Due Monday October 3 at 5:30pm submitted to Christoph Dann in Gates 803 Remember to a submit separate writeup for each problem, with your name at the top) Total:
More informationDefinition of convex function in a vector space
Convex functions Nonlinear optimization Instituto Superior Técnico and Carnegie Mellon University PhD course João Xavier TAs: Brian Swenson, Shanghang Zhang, Lucas Balthazar Convex functions Nonconvex
More informationMath 328 Course Notes
Math 328 Course Notes Ian Robertson March 3, 2006 3 Properties of C[0, 1]: Sup-norm and Completeness In this chapter we are going to examine the vector space of all continuous functions defined on the
More information4 Linear operators and linear functionals
4 Linear operators and linear functionals The next section is devoted to studying linear operators between normed spaces. Definition 4.1. Let V and W be normed spaces over a field F. We say that T : V
More informationWhat can be expressed via Conic Quadratic and Semidefinite Programming?
What can be expressed via Conic Quadratic and Semidefinite Programming? A. Nemirovski Faculty of Industrial Engineering and Management Technion Israel Institute of Technology Abstract Tremendous recent
More informationMATHEMATICAL ECONOMICS: OPTIMIZATION. Contents
MATHEMATICAL ECONOMICS: OPTIMIZATION JOÃO LOPES DIAS Contents 1. Introduction 2 1.1. Preliminaries 2 1.2. Optimal points and values 2 1.3. The optimization problems 3 1.4. Existence of optimal points 4
More informationSOME REMARKS ON THE SPACE OF DIFFERENCES OF SUBLINEAR FUNCTIONS
APPLICATIONES MATHEMATICAE 22,3 (1994), pp. 419 426 S. G. BARTELS and D. PALLASCHKE (Karlsruhe) SOME REMARKS ON THE SPACE OF DIFFERENCES OF SUBLINEAR FUNCTIONS Abstract. Two properties concerning the space
More informationDuality Theory of Constrained Optimization
Duality Theory of Constrained Optimization Robert M. Freund April, 2014 c 2014 Massachusetts Institute of Technology. All rights reserved. 1 2 1 The Practical Importance of Duality Duality is pervasive
More informationWEAK CONVERGENCE OF RESOLVENTS OF MAXIMAL MONOTONE OPERATORS AND MOSCO CONVERGENCE
Fixed Point Theory, Volume 6, No. 1, 2005, 59-69 http://www.math.ubbcluj.ro/ nodeacj/sfptcj.htm WEAK CONVERGENCE OF RESOLVENTS OF MAXIMAL MONOTONE OPERATORS AND MOSCO CONVERGENCE YASUNORI KIMURA Department
More informationOnline Convex Optimization
Advanced Course in Machine Learning Spring 2010 Online Convex Optimization Handouts are jointly prepared by Shie Mannor and Shai Shalev-Shwartz A convex repeated game is a two players game that is performed
More informationh(x) lim H(x) = lim Since h is nondecreasing then h(x) 0 for all x, and if h is discontinuous at a point x then H(x) > 0. Denote
Real Variables, Fall 4 Problem set 4 Solution suggestions Exercise. Let f be of bounded variation on [a, b]. Show that for each c (a, b), lim x c f(x) and lim x c f(x) exist. Prove that a monotone function
More informationConvex Optimization. Dani Yogatama. School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA. February 12, 2014
Convex Optimization Dani Yogatama School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA February 12, 2014 Dani Yogatama (Carnegie Mellon University) Convex Optimization February 12,
More informationMATH 680 Fall November 27, Homework 3
MATH 680 Fall 208 November 27, 208 Homework 3 This homework is due on December 9 at :59pm. Provide both pdf, R files. Make an individual R file with proper comments for each sub-problem. Subgradients and
More informationLecture 7: September 17
10-725: Optimization Fall 2013 Lecture 7: September 17 Lecturer: Ryan Tibshirani Scribes: Serim Park,Yiming Gu 7.1 Recap. The drawbacks of Gradient Methods are: (1) requires f is differentiable; (2) relatively
More information