ELEC-E5420 Conve functions 2{1 Lecture 2 Conve functions conve functions, epigraph simple eamples, elementary properties more eamples, more properties Jensen's inequality quasiconve, quasiconcave functions log-conve and log-concave functions K-conveity
Conve functions 2{2 Conve functions f : R n! R is conve if dom f is conve and y 2 dom f 2 [0 1] + f ( + (1 ; )y) f () + (1 ; )f (y) (1) f is concave if ;f is conve conve concave neither `Modern' denition: f : R n! R [ f+1g (but not identically +1) f is conve if (1) holds as an inequality in R [ f+1g
Conve functions 2{3 Epigraph & sublevel sets The epigraph of the function f is epi f = f( t) j 2 dom f f () t g : f () epi f f conve function, epi f conve set The (-)sublevel set of f is C() = f 2 dom f j f () g : f conve ) sublevel sets are conve (converse false)
Conve functions 2{4 Dierentiable conve functions f dierentiable and conve () 8 0 : f () f ( 0 ) + rf ( 0 ) T ( ; 0 ) f () 0 f ( 0 )+rf ( 0 ) T ( ; 0 ) Interpretation 1st order Taylor appr. is a global lower bound on f supporting hyperplane to epi f: ( t) 2 epi f =) f () 2 6 rf ( 0) 4 ;1 3 7 5 T 2 6 64 ; 0 t ; f ( 0 ) 3 7 5 0 2 4 rf ( 0) ;1 3 5 f twice dierentiable and conve () r 2 f () 0
Conve functions 2{5 Simple eamples linear and ane functions: f () = a T + b conve quadratic functions: f () = T P+ 2q T + r with P = P T 0 any norm Eamples on R is conve on R + for 1, 0 concave for 0 1 log is concave, log is conve on R + e is conve jj, ma(0 ), ma(0 ;) are conve Z ;1 e;t2 dt is concave log
Conve functions 2{6 Elementary properties a function is conve i it is conve on all lines: f conve () f ( 0 + th) conve in t for all 0 h positive multiple of conve function is conve: f conve 0 =) f conve sum of conve functions is conve: f 1 f 2 conve =) f 1 + f 2 conve etends to innite sums, integrals: g( y) conve in =) Z g( y)dy conve
Conve functions 2{7 pointwise maimum: f 1 f 2 conve =) maff 1 () f 2 ()g conve (corresponds to intersection of epigraphs) f 2 () epi maff 1 f 2 g f 1 () pointwise supremum: f conve =) sup f conve 2A ane transformation of domain f conve ) f (A + b) conve
Conve functions 2{8 More eamples piecewise-linear functions: f () = mafa T i i + b ig is conve in (epi f is polyhedron) ma distance to any set, sup s2s k ; sk, is conve in f () = [1] + [2] + [3] is conve on R n ( [i] is the ith largest j ) f () = 0 B @ Y i 1 C i A 1=n is concave on R n + f () = m X i=1 log(b i ; a T i );1 is conve on P = f j a T i < b i i = 1 ::: mg least-squares cost as functions of weights, is concave in w f (w) = inf X i w i (a T i ; b i) 2
Conve functions 2{9 Conve functions of matrices Tr X is linear in X more generally, Tr A T X = X i j A ij X ij = vec(a) T vec(x) log det X ;1 is conve on X = X T 0 Proof: let i be the eigenvalues of X ;1=2 0 HX ;1=2 0 f (t) = log det(x 0 + th) ;1 = log det X ;1 0 + log det(i + tx ;1=2 0 HX ;1=2 0 ) ;1 = log det X ;1 0 ; X i is a conve function of t log(1 + t i ) (det X) 1=n is concave on X = X T 0, X 2 R nn ma (X) is conve on X = X T Proof: ma (X) = sup y T Xy kyk=1 kxk = ma (X T 1=2 X) is conve on R nm Proof: kxk = sup u T Xv kuk=1 kvk=1
Conve functions 2 {10 Minimizing over some variables If h( y) is conve in and y, then is conve in f () = inf y h( y) corresponds to projection of epigraph, ( y t)! ( t) h( y) f () y
Conve functions 2 {11 Eample. If S R n is conve then (min) distance to S, is conve in dist( S) = inf k ; sk s2s Eample. If g() is conve, then is conve in y. f (y) = inffg() j A = yg Proof: nd B, C s.t. f j A = yg = fby + Cz j z 2 R k g so f (y) = inf z g(by + Cz) `Modern' proof: f (y) = inf z g() + h(a ; y) where is conve h(z) = 8 >< >: 0 if z = 0 +1 otherwise
Conve functions 2 {12 Composition one-dimensional case f () = h(g()) is conve if g conve h conve, nondecreasing g concave h conve, nonincreasing Eamples f () = ep g() is conve if g is conve f () = 1=g() is conve if g is concave, positive f () = g() p, p 1, is conve if g() conve, positive f 1,..., f n conve, then f () = ; X i conve on f j f i () < 0 i = 1 ::: ng log(;f i ()) is Proof: (dierentiable functions, 2 R) f 00 = h 00 (g 0 ) 2 + g 00 h 0
Conve functions 2 {13 Composition k-dimensional case f () = h(g 1 () ::: g k ()) with h : R k! R, g i : R n! R is conve if h conve, nondecreasing in each arg. g i conve h conve, nonincreasing in each arg. g i concave etc. Eamples f () = ma i g i () is conve if each g i is f () = log X i ep g i () is conve if each g i is Proof: (dierentiable functions, n = 1) f 00 = rh T 2 6 4 g 00 1. g 00 k 3 2 g 0 3 1 7 + 5 6. 4 g 0 7 5 k T r 2 h 2 6 4 g 0 3 1. g 0 7 5 k
Conve functions 2 {14 Jensen's inequality f : R n! R conve two points 2 [0 1] + f ( 1 + (1 ; ) 2 ) f ( 1 ) + (1 ; )f ( 2 ) more than two points f ( X i i 0 X i = 1 + i i i ) X i i f ( i ) continuous version most general form: p() 0 Z p()d = 1 + f (Z p()d) Z f ()p()d f (E ) E f () Interpretation: (zero mean) randomization, dithering increases average value of a conve function
Conve functions 2 {15 Applications Many (some people claim most) inequalities can be derived from Jensen's inequality Eample. Arithmetic-geometric mean inequality a b 0 ) p ab (a + b)=2 Proof. f () = log is concave on R + : 0 1 B (log a + log b) log a @ + b 1 C A 2 2
Conve functions 2 {16 Quasiconve functions f : C! R, C a conve set, is quasiconve if every sublevel set S = f j f () g is conve. y S can have `locally at' regions f () f is quasiconcave if ;f is quasiconve, i.e., superlevel sets f j f () g are conve. A function which is both quasiconve and quasiconcave is called quasilinear. f conve (concave) ) f quasiconve (quasiconcave)
Conve functions 2 {17 Eamples f () = r jj is quasiconve on R f () = log is quasilinear on R + linear fractional function, f () = at + b c T + d is quasilinear on the halfspace c T + d > 0 k ; ak k ; bk f j k ; ak k ; bkg f () = is quasiconve on the halfspace f (a) = degree(a 0 + a 1 t + + a k t k ) on R k+1
Conve functions 2 {18 Properties f is quasiconve if and only if it is quasiconve on lines, i.e., f ( 0 + th) quasiconve in t for all 0 h. modied Jensen's inequality: f : C! R quasiconve if and only if y 2 C 2 [0 1] + f ( + (1 ; )y) maff () f(y)g f () y
Conve functions 2 {19 for f dierentiable, f quasiconve if and only if for all, y f (y) f () ) (y ; ) T rf () 0 S 1 rf () S 2 S 3 1 < 2 < 3 positive multiples f quasiconve 0 =) f quasiconve pointwise maimum f 1 f 2 quasiconve =) maff 1 f 2 g quasiconve (etends to supremum over arbitrary set) ane transformation of domain f quasiconve =) f (A + b) quasiconve
Conve functions 2 {20 projective transformation of domain 0 B f quasiconve =) f A @ + b c T + d on c T + d > 0 1 C A quasiconve composition with monotone increasing function f quasiconve g monotone increasing =) g(f ()) quasiconve sums of quasiconve functions are not quasiconve in general f quasiconve in, y =) g() = inf y quasiconve in f ( y)
Conve functions 2 {21 Nested sets characterization f quasiconve ) sublevel sets S conve, nested, i.e., 1 2 ) S 1 S 2 converse: if T is a nested family of conve sets, then is quasiconve. f () = inff j 2 T g Engineering interpretation: T are specs, tighter for smaller
Conve functions 2 {25 Log-concave functions f : R n! R + is log-concave (log-conve) if log f is concave (conve) Log-conve ) conve concave ) log-concave `Modern' denition allows log-concave f to take on value zero, so log f takes on value ;1 Eamples normal density, f () = e ;(1=2)(; 0) T ;1 (; 0 ) erfc, f () = p 2 Z 1 e;t2 dt indicator function of conve set C: I C () = 8 >< >: 1 2 C 0 62 C
Conve functions 2 {26 Properties sum of log-concave functions not always log-concave (but sum of log-conve functions is log-conve) products (immediate) f g log-concave =) fg log-concave integrals f ( y) log-concave in y =) Z f ( y)dy log-concave convolutions f g log-concave =) Z f ( ; y)g(y)dy log-concave (immediate from the properties above)
Conve functions 2 {27 Log-concave probability densities Many common probability density functions are log-concave. Eamples normal ( 0) f () = 1 r(2) n det e;1 2 (;) T ;1 (;) eponential( i > 0) on R n + f () = 0 B @ n Y i=1 i 1 C A e ;( 1 1 ++ n n ) uniform distribution on conve (bounded) set C f () = 8 >< >: 1= 2 C 0 62 C where is Lebesgue measure of C (i.e., length, area, volume...)
Conve functions 2 {31 K-conveity conve cone K R m induces generalized inequality K f : R n! R m is K-conve if 0 1 =) f ( + (1 ; )y) K f () + (1 ; )f (y) Eample. K is PSD cone (called matri conveity) let's show that f (X) = X 2 is K-conve on fxjx = X T g, i.e., for 2 [0 1], (X + (1 ; )Y ) 2 X 2 + (1 ; )Y 2 (1) for any u 2 R m, u T X 2 u = kxuk 2 is a (quadratic) conve fct of X, so u T (X + (1 ; )Y ) 2 u u T X 2 u + (1 ; )u T Y 2 u which implies (1)