10-801: Advanced Optimization and Randomized Methods Lecture 2: Convex functions (Jan 15, 2014)

Transcription

1 0-80: Advanced Optmzaton and Randomzed Methods Lecture : Convex functons (Jan 5, 04) Lecturer: Suvrt Sra Addr: Carnege Mellon Unversty, Sprng 04 Scrbes: Avnava Dubey, Ahmed Hefny Dsclamer: These notes have not been subjected to the usual scrutny reserved for formal publcatons. They may be dstrbuted outsde ths class only wth the permsson of the Instructor.. Revew The lecture began wth a revew of convexty n metrc spaces, a topc that we could not cover fully n Lecture. Defnton.. A metrc space (X, d) s called complete f any Cauchy sequence wthn the space converges to a pont n the space. Defnton.. A metrc space s called locally compact f every pont n the space has a compact neghborhood. Note: Sets that have empty nterors do not have ths property (snce havng empty nterors, they cannot be neghborhoods). Defnton.3. A geodesc s a contnuous path between two ponts n (X, d) denoted by: s.t. γ() = y, γ(0) = x satsfyng: γ(t) = [x, y] t, t [0, ], d(γ(t ), γ(t )) = t t d(x, y) t, t [0, ] The latter condton mples that a geodesc s a shortest path. Theorem.4. Suppose (X, d) s a complete, locally compact metrc space. Then, the followng are equvalent: () (X, d) s Menger-convex () (X, d) has mdponts,.e., x, y X, m X s.t. d(x, m) = d(y, m) = d(x, y), () (X, d) s a geodesc space,.e., x, y X, there exsts a geodesc [x, y] t.. Convex Functons.. Notatons and Conventons: Extended Reals Before we start the topc of convex functons, we need to ntroduce notaton that wll be useful when we dscuss functons whose values are nfnte. We defne the set of extended reals as R := R {, }. By conventon, the value of nfnty has the followng propertes: x + =, < x 0 = 0 = 0 ( ) = + nf φ = +, sup φ = -

2 LECTURE Convex functons Jan 5, 04 Ths conventon allows us to talk about convex functons on R wthout always havng to worry about ther domans. For example, { f(x) = x for x > 0 for x 0. The doman of f, denoted dom f s defned to be the (convex) set on whch f assumes values smaller than +... Convexty and Mdpont Convexty Defnton.5. A functon f s mdpont convex f ( ) x + y f Defnton.6. A functon f s convex f Ths condton s called Jensen s nequalty. f(x) + f(y) x, y dom(f) f(( α)x + αy) ( α)f(x) + αf(y) x, y dom(f), α (0, ) Defnton.7. A functon f defned on a metrc space s convex f f(( α)x αy) ( α)f(x) + αf(y) x, y dom(f), α (0, ), where ( α)x αy := γ(α), γ represents the geodesc as defned n.3. Theorem.8. (Jensen, 905) If f s a contnuous mdpont convex functon then f s convex. Proof. By contradcton: Suppose f s a contnuous mdpont convex functon that does not satsfy Jensen s nequalty at some choce of x, y. Defne: Then, by our assumpton Then g(α) := f(( α)x + αy) ( α)f(x) αf(y) α (0, ) s.t. g(α) > 0. max g(α) = M > 0. α (0,) Let α 0 be the smallest value of α (0, ) satsfyng g(α 0 ) = M. Also, let δ > 0 be small enough such that (α 0 δ, α 0 + δ) (0, ). Defne: x := ( α 0 δ)x + (α 0 + δ)y ȳ := ( α 0 + δ)x + (α 0 δ)y Then the mdpont convexty assumpton mples that ( ) x + ȳ f Note that Therefore, f( x) + f(ȳ) g(α 0 + δ) + g(α 0 δ) = f( x) + f(ȳ) [( α 0 )f(x) + α 0 f(y)] whch contradcts our premse that g(α 0 ) = M. = f( x) + f(ȳ) + [g(α 0 ) f(( α 0 )x + α 0 y)] ( ) x + ȳ = g(α 0 ) + f( x) + f(ȳ) f g(α 0 ) g(α 0 ) g(α 0 + δ) + g(α 0 δ) < M + M < M -

3 LECTURE Convex functons Jan 5, 04 Theorem.8 shows that, n oder to prove the convexty of a contnuous functon, t s suffcent to prove mdpont convexty. Ths smplfcaton can be helpful, as can be seen n the followng example. Example.9. The functon f(x) := log det(x) (X P n +) s concave. Note: P n + s the set of symmetrc postve defnte matrces. Proof. Snce the functon s contnuous, t s suffcent to prove mdpont concavty, that s we need to show that X + Y X Y By dvdng each sde by X, we obtan I + X Y X Y ( + λ (X ) Y ) λ (X Y ), where λ (X) s the th egenvalue of X. Note: the th factor n the LHS s the arthmetc mean of λ and, whle the th factor n the RHS s ther geometrc mean. Therefore t suffces to prove that λ 0. X s symmetrc postve defnte, therefore so are ts nverse and square root. Hence snce X / s symmetrc and Y 0, whereby λ (X Y ) = λ (X / X Y X / ) = λ (X / Y X / ) 0, z T X / Y X / z = z T Y z 0 z 0. Exercse.. [CHALLENGE] Let x, x,..., x n > 0 be a sequence of real varables. Defne Prove that h n s convex. h (x ) := x h (x, x ) := + x x x + x h 3 (x, x, x 3 ) := x + x + x 3. x + x Defnton.0. The epgraph of a functon f s defned as x + x 3 ep(f) := {(x, t) R n R f(x) t} A convex functon f s called closed f ts epgraph ep(f) s closed and convex. An example of an epgraph s shown n fgure....3 Important Convex Functons Example.. Let C R n be a non-empty set. Defne { 0 f x C δ C (x) := f x / C If C s closed and convex then δ s closed and convex. x + x 3 + x + x + x 3-3

4 LECTURE Convex functons Jan 5, 04 Fgure.: A functon of a sngle varable wth the epgraph ndcated by the shaded regon. Example.. Let C R n be a non-empty closed set. The support functon of C, defned as s a convex functon. σ C (z) := sup z, x, Example.3. Let h(x, y) be a famly of functons ndexed by y Y. If h(x, y) s convex n x for each y then s also convex. f(x) := sup h(x, y) y Y Defnton.4. Let f : R n R. The Fenchel conjugate of functon f s defned as f (z) := sup x R n z, x f(x) Note that f s a specal case of example.3 and hence s convex regardless of the convexty of f..3 Norms Defnton.5. A functon f : R n R s a norm on R n f t satsfes the followng condtons:. f(x) 0, f(x) = 0 ff x = 0. f(λx) = λ f(x), λ R 3. f(x + y) f(x) + f(y) If f s a norm then f s convex. Example.6. The l p norm on R n for p < s defned as: For p = the norm s defned as x p := ( x p ) p x := max n x Example.7. Let x = ( x, x,..., x k ) R n+n+ +n k. The l p,q norm s defned as ( x p,q := x p q ) /p -4

5 LECTURE Convex functons Jan 5, Matrx Norms There are dfferent ways to defne matrx norms dependng on how we look at a matrx. We can vew a matrx as a lnear operator, whch leads to the operator norm. Defnton.8. The operator norm of a matrx A s defned as Ax u A u := sup x 0 x u Exercse.. Show that the followng matrx functons are norms: A = σ (A), the largest sngular value of A (also known as the spectral norm) A, the largest absolute column sum A, the largest absolute row sum It s worth notng that, for a generc p, A p s NP-hard to compute. Alternatvely, we can treat the elements of an m n matrx as a vector and use any vector norm. Usng the l norm on the matrx vewed as a vector results n the so-called Frobenus norm. Defnton.9. The Frobenus norm of a matrx A C m n s defned as A F := tr(a A) = a j Defnton.0. The Schatten p-norm of a matrx A C m n s defned as A p = σ(a) p, where σ(a) s a vector of sngular values. Settng p to yelds the trace norm A, also know as the nuclear norm. Note: Whle the largest sngular value s a vald matrx norm, the largest egenvalue s not. The largest egenvalue s, however, a vald norm over postve defnte matrces..3. Dual Norm and Hölder s Inequalty Defnton.. For a norm over R n, the dual norm s defned as: Theorem. (Hölders s nequalty). For any x, y R n, u := sup{u T x x } x x T y x y Exercse.3. Let x be a norm. Show that f (z) = δ. (z). [Hnt: Consder separately the cases where z > and z. You mght also need to use Hölder s nequalty]. j -5