10-1 Multi-User Information Theory Jan 17, 2012 Lecture 10 Lecturer:Dr. Haim Permuter Scribe: Wasim Huleihel I. CONJUGATE FUNCTION In the previous lectures, we discussed about conve set, conve functions (properties and eamples), and we present some operations that preserve conveity. This lecture we will continue the discussion about conve functions; definition and eamples of conjugate functions will be given, and we will present some general conve optimization problems. Definition 1 (Conjugate function) Let f : R n R. The function f : R n R defined as is called the conjugate function of f. This definition is illustrated in Fig. 1. ( f (y) sup y T f () ), (1) domf Remark 1 The domain of the conjugate function is given by { ( domf = y R n : sup y T f () ) } <. (2) domf Remark 2 The conjugate function, f, is a conve function. This can be easily verified using that fact that the supremum of a set of conve functions (in our case, for a fied, the difference y T f () is a linear function of y and hence the difference is conve) is conve function. Note that this is true whether or not f is conve. Remark 3 y T stands for inner product in finite space dimensions. In infinite dimensional vector spaces, the conjugate function is defined via an appropriate inner product. This operation play an important role in many applications such as duality. In the following, we present some eamples of conjugate functions.
10-2 y f () (0, f (y)) Fig. 1. The conjugate function f (y) is the maimum gap between the linear function y and the function f (). Eample 1 (Affine function) f () = a + b. By definition, the conjugate function is given by f (y) = sup (y a b). As a function of, the difference is bounded iff y a = 0. The conjugate function is given f (y) = b, and the domain is domf = {a}. Eample 2 (Negative logarithm) f () = log, with domf = R ++. By definition, the conjugate function is given by f (y) = sup (y+log). As a function of, the difference is bounded iff y < 0, and reaches its maimum at = 1. The conjugate y function is given f (y) = log( y) 1, and the domain is domf = R ++. Eample 3 (Eponential) f () = e. By definition, the conjugate function is given by f (y) = sup (y e ). As a function of, the difference is bounded iff y > 0, and reaches its maimum at = lny. The conjugate function is given f (y) = ylny y, and the domain is domf = R ++. Eample 4 (Entropy) f () = log, with domf = R ++. By definition, the conjugate function is given by f (y) = sup (y log). As a function of, the difference is bounded for all y, and reaches its maimum at = e (y 1). The conjugate function is given f (y) = e (y 1), and the domain is domf = R. So the conjugate of entropy is
10-3 related to the eponential family. Eample 5 (Log-sum-ep) f () = log n i=1 e i, with domf = R n. By definition, the conjugate function is given by f (y) = sup (y log n i=1 e i ). By setting the gradient with respect to to zero, the critical points are given by y i = e i n i=1 e i i = 1,...,n. (3) The domain of f is domf = {y R n : 0 y i 1, n i=1 y i = 1}. The conjugate function is given f (y) = n i=1 y ilogy i. Net, we present two interesting properties of conjugate function. Properties of conjugate function f (y)+f () T y. Conjugate of the conjugate: if f is conve function, and its epigraph is closed set, then f = f. II. CONVEX OPTIMIZATION PROBLEMS The general optimization problem is given by f 0 () s.t. f i () 0; i = 1,...,m h j () = 0; j = 1,...,p, (4) where f 0 (),{f i ()} m i=1 are conve functions, and {h j()} p j=1 are affine functions. The optimization problem (4) describe the problem of finding an that imizes the objective f 0 subject to (s.t.) the constraintsf i () 0; i = 1,...,m, and h j () = 0; j = 1,...,p. In the following, we present some classes of optimization problems that can be efficiently solved using Matlab. Linear Programg: linear programg is a technique for the optimization of a linear objective function, subject to linear equality and linear inequality constraints.
10-4 Its feasible region (the space of all possible solutions), is a conve polyhedron. Mathematically, the optimization problem in this case can be in the following form c T +d s.t. G h A = b. (5) Linear Fraction: linear fraction optimization problem can be written as c T +d e T +f s.t. G h A = b e T +f > 0. (6) Let us transform this optimization problem to a linear one (5). This can be easily done by introducing auiliary variables defined as and z = y = 1 e T +f, e T +f. Using these new variables the objective function becomec T y dz, and the constraints become G h Gy hz A = b Ay = bz e T +f > 0 z 0 e T y +fz = 1 New constraint. Therefore the optimization problem is given by y,z ct y dz
10-5 s.t. Gy hz Ay = bz z 0, e T y +fz = 1. (7) Quadratic Optimization Problem: the quadratic optimization problem is given by where P S n +. 1 2 T P+q T +r s.t. G h A = b. (8) Second Order Cone Programg: the second order cone optimization problem is given by f T s.t. A i +b 2 c T i +b i; i = 1,...,n F = g. (9) Geometric Programg: this class of problems is not conve, but we will transform it to a conve one. Definition 2 The function f () = c q 1 1 q 2 2... qn, where c 0, is called monomial. Definition 3 The functionf () = K k=1 c k q 1,k 1 q 2,k 2... q n,k n, is called posynomial. The Geometric optimization problem is given by f 0 () s.t. f i () 0; i = 1,...,m n h j () = 0; j = 1,...,p, (10) where f 0 (),{f i ()} m i=1 are posynomial functions, and {h j()} p j=1 functions. are monomial
10-6 Eample 6 Consider the following optimization problem,y,z y is a Geometric optimization problem. s.t. 2 3 2 + 3y z y y = z2. Geometric programs are not (in general) conve optimization problems, but they can be transformed to conve problems by a change of variables and a transformation of the objective and constraint functions. We will use the variables defined as i = e y i. Using this transformation, the Geometric optimization problem given in (10), can be rewritten as y log s.t. log ( )] ep a 0,k,m y m +b 0,k [ K0 k=1 [ Ki m ( )] ep a i,k,m y m +b i,k 0; i = 1,...,m k=1 m c j,m y m +d j = 0; j = 1,...,p. (11) m Since the objective function, and the first constraint are conve (log-sum eponential function), and the second constraint is affine, this problem is conve optimization one. Semi-Definite Programg (SDP) the optimization problem is given by e T s.t. 1 F 1 +...+ n F n 0 A = b, (12) REFERENCES [1] S. Boyd, Conve Optimization. Cambridge University Press, 2004.