9-1 Mahemaical mehods in communicaion January 10h, 2012 Lecure 9 Lecurer: Haim Permuer Scribe: Moi Teiel I. INRODUCTION Lemma 1 f is convex iff g( = f (x+v is convex in for all v. Exercise: Prove Lemma 1. For example, we examine logdex, where X S N ++. Assume ha V S n. Then: g( = log X+V (a 1/2 X 1/2 +V (1 1/2 ( I+X 1/2 VX 1/2 X 1/2 1/2 ( I+UΛU T X 1/2 1/2 U(I+ΛU T X 1/2 1/2 U I+Λ U T X 1/2 (b = log X 1/2 I+Λ X 1/2 1/2 +log I+Λ +log X 1/2 = log X +log I+Λ (c = log X + log(1+λ i which is concave in, Therefore logdex is concave in X. The equaliy marked by (a is due o he fac ha X S++ n, so we can wrie X = X 1/2 X 1/2. Equaliy (b holds since X,V S n so X 1/2 VX 1/2 S n as well, and is herefore diagonalizable. Equaliy (c follows from he fac ha U is uniary, so deu = 1. Equaliy (d follows from he fac ha Λ is diagonal.
9-2 II. OPERATIONS THAT PRESERVE CONVEXITY 1 Non-negaive weighed sum: if {f i } N i=1 are convex, and {w i} N i=1 are non-negaive scalars, hen i w if i is convex. This is also rue for counably infinie suns, and for inegrals: If for all y A we have f (x,y convex in x, and w(y 0 for all y A, hen f (x,yw(ydy (2 A is convex in x. 2 poinwise maximum: if f 1,f 2 are convex, hen max(f 1,f 2 is also convex. f ( θx+ θy = max ( ( ( f 1 θx+ θy,f2 θx+ θy (3 max ( θf 1 (x+ θf 1 (y,θf 2 (x+ θf 2 (y max(θf 1 (x,θf 2 (x+max ( θf1 (y, θf 2 (y = θf (x+ θf (y Using his propery, we may conclude ha he inersecion of convex ses is convex. In order o show his, recall ha a funcion is convex iff i s epigraph is a convex se. Le f 1 and f 2 be he wo funcions whose epigraphs are he convex ses A and B respecivally. Since he inersecion of he wo convex ses A,B is he epigraph of max(f 1,f 2, which is convex, i is a convex se. 3 composiion wih an affine mapping: suppose P : R n R, A R n+m,b R n. Define g : R m R by g(x = f (Ax+b. The domain of g is domg = {x : Ax+b domf}. If f is convex hen g is convex for all A,b. g ( θx+ θy = f ( A ( θx+ θy +b = f ( θax+ θay+b (4 = f ( θ(ax+b+ θ(ay+b θf (Ax+b+ θf (Ay+b = θg(x+ θg(y
9-3 4 Scalar composiion (can be exended o a vecor space: Given g : R R,h : R R, define f = h(g(x. a If h is a convex funcion, non-decreasing, and g is convex, hen f is convex. b If h is a convex funcion, non-increasing, and g is concave, hen f is convex. c If h is a concave funcion, non-increasing, and g is convex, hen f is concave. d If h is a concave funcion, non-decreasing, and g is concave, hen f is concave. For example, if g is convex, e g(x is convex. If g is concave and posiive, logg(x is concave. We will prove (a: Since h is non-decreasing, we have h 0. Since h is convex, we have h 0. Since g is convex, we have g 0. So, [h(g] = h (gg (5 [h(g] = [h (gg ] = h (g(g 2 +h (gg 0 The res of he properies are proven similarly. 5 Vecor composiion: f (x = h(g(x = h(g 1 (x,g 2 (x,...,g k (x, where h : R k R,g : R n R k,g i : R n R i = 1,2,...,k. a If h is a convex funcion, non-decreasing, and g i are convex for all i, hen f is convex. b If h is a convex funcion, non-increasing, and g i are concave for all i, hen f is convex. c If h is a concave funcion, non-increasing, and g i are convex for all i, hen f is concave. d If h is a concave funcion, non-decreasing, and g i are concave for all i, hen f is concave. If h is convex, non-decreasing in each elemen, and g i are concave for all i, hen f is convex.
9-4 For n = 1, for example, we have f (x = h T (g(xg (x, g (x = (g 1 (x,g 2 (x,...,g k (x (6 f (x = (g (x T 2 h(g(xg (x+ h T (g(xg (x So if he condiions hold we have ha f 0. The oher combinaions follow similarly. 6 If f (x,y is a convex funcion in (x,y C, where C is a convex se, hen is a convex funcion. g(x = inf f (x,y (7 y:(x,y C From he definiion of he infimum we have ǫ > 0, x 1,x 2 y 1,y 2 : f (x i,y i g(x i +ǫ i = 1,2. (8 We may hen wrie, for all ǫ > 0, g ( θx 1 + θx 2 = inf f ( θx 1 + θx 2,y (a f ( θx 1 + θx 2,θy 1 + θy 2 y θf (x 1,y 1 + θf (x 2,y 2 (b θg(x 1 + θg(x 2 +2ǫ (9 where (a holds as inf y mus achieve a lower (or equal value ha any specific y, in paricular han for θy 1 + θy 2, and (b follows from Equaion 8. 7 Perspecive ransformaions: If f (x is a convex funcion, hen g(x, = f ( f is also convex in (x, for 0. g ( θ(x 1, 1 + θ(x 2, 2 = ( θ 1 + θ 2 f ( θx1 + θx 2 θ 1 + θ 2 = ( ( θ 1 + θ x θ1 1 2 f 1 + θ x 2 2 2 θ 1 + θ 2 ( θ 1 + θ [ ( θ 1 x1 2 θ 1 + θ f 2 1 ( ( x1 = θ 1 f + θ x2 2 f 1 2 + θ 2 θ 1 + θ 2 f ( x2 2 (10 ] = θg(x 1, 1 + θg(x 2, 2
9-5 For example, define P = [p 1,p 2,...,p n ] and Q = [q 1,q 2,...,q n ] o be probabiliy disribuions, and examine D(P Q = i p ilog p i q i = p i log q i Since f (x = logx is convex, we have g(x, = f ( f p i. = log x convex as well. Since he sum of convex funcions is convex, we may define D(X,T = i g(x i, i = i ilog x i i convex. By subsiuing X = Q,T = P, we have ha D(P Q = p i log q i p i is convex.