Convex Duality and Financial Mathematics

Transcription

1 Peter Carr and Qiji Jim Zhu Convex Duality and Financial Mathematics September 14, 2016

2 Contents 1 Convex Duality Convex Functions and Sets Subdifferential and Lagrange multiplier Fenchel Conjugate Convex duality theory Generalized conjugate and duality Exercises Financial models in one period economy Portfolio Utility functions Fundamental Theorem of Asset Pricing Risk measures Exercises Finite period financial models The model Arbitrage and admissible trading strategies Fundamental theorem of asset pricing Hedging and Super Hedging Conic Finance Continuous financial models Continuous stochastic processes Bachelier and Black-Scholes formulae Duality and Delta Hedging Generalized Duality and Hedging with Contingent Claims References

3

4 Preface Convex duality plays essential role in financial problems involving maximizing concave utility functions and minimizing convex risk measures. Recently convex and generalized convex duality are also shown to be crucial in the process of dynamic hedging of contingent claims. The goal of this book is to provide a concise introduction of this growing field to graduate students and researchers in related areas. We start with a quick introduction of convex duality and related tools emphasizing its relationship with Lagrange multiplier rule for constrained optimization problems. Then we discuss the intrinsic duality relationship in diverse financial problems. Topics include Markowitz portfolio theory, growth optimal portfolio theory, fundamental theorem of asset pricing emphasizing the duality between utility optimization and pricing by martingale measures, risk measures and its dual representation, hedging and super-hedging and its relationship with linear programming duality and the duality relationship in dynamic hedging of contingent claims. The material grows out of lecture notes of our joint doctoral seminar in Courant Institute at NYU in the fall of Parts of the material have also been used previously for graduate topic courses on optimization and modelling at Western Michigan University. We thank our colleagues at NYU and WMU for providing us supporting environments. We are indebted to the participants of these seminar and courses for stimulating discussions. In particular we thank Monty Essid, Mathew Forman, Sanjay Karanth, Mehdi Vazifadan, Guolin Yu whose detailed comments on various parts of our lecture notes have been incorporated into the text. Place(s), month year Peter Carr Qiji Zhu

5

6 1 Convex Duality Summary. We present a concise description of the convex duality theory in this chapter. The goal is to lay a foundation for later application in various financial problems rather than comprehensive. 1.1 Convex Functions and Sets Definitions Definition (Convex sets and functions) Let X be a Banach space. We say that a subset C of X is convex if, for any x, y C and any λ [0, 1], λx+(1 λ)y C. We say an extended-valued function f : X R {+ } is convex if its domain is convex and for any x, y dom f and any λ [0, 1], one has f(λx + (1 λ)y) λf(x) + (1 λ)f(y). We call a function f : X [, + ) concave if f is convex. In some sense convex functions are the simplest functions next to linear functions. Convex functions and convex sets are intrinsically related. For example, it is easy to verify that C is a convex set if and only if ι C, its indicator function, is a convex function. On the other hand if f is a convex function then epi f and f 1 ((, a]), a R are convex sets. In fact, we can check that the convexity of epi f characterizes that of f. This geometric characteization is very useful in many situations. For example, it is easy to see that the intersection of a class of convex sets is convex. Now let f α be a class of convex functions we can see that epi sup f α = α epi f α α and, thus, sup α f α is convex. In particular, the support function of a set C X defined on the dual space X by σ C (x ) = σ(c; x ) := sup{ x, x x C}

7 4 1 Convex Duality is always convex. Note that allowing the extended value + in the definition of convex function is important in establishing those relations. An important property of convex functions related to applications in economics and finance is the Jensen inequality. Proposition (Jensen s inequality) Let f be a convex function. Then, for any random variable X on a finite probability space, f(e(x)) E(f(X)). When X has a finite image this result directly follows from the definition. The general result can be proven by approximation. A special kind of convex set convex cone is very useful. Definition Let X be a finite dimnsional Banach space. We say K X is a convex cone if for any x, y K and any α, β 0, αx + βy K. Moreover, we say K is pointed if K ( K) = {0}. A pointed convex cone K induces a partial order K by defining x K y iff y x K). We can easily check that K is reflexive (x K x), antisymmetric (x K y and y K x implies x = y) and transitive (x K y and y K z implies that x K z). The definition of convexity can easily be extended to mappings whose image space has such a partial order. Definition (Convex mappings) Let X and Y be two Banach spaces. Assume that Y has a partial order K generated by the pointed convex cone K Y. We say that a mapping f : X Y is K-convex provided that, for any x, y dom f and any λ [0, 1], one has f(λx + (1 λ)y) K λf(x) + (1 λ)f(y) Convex programming We will often encounter special forms of the general convex programming problem below in financial applications. Let X, Y and Z be finite dimensional Banach spaces. Assume that Y has a partial order K generated by the pointed convex cone K. We denote the polar cone of K by K + := {y Y : y, y 0 for all y K}. Consider the following class of constrained optimization problems P (y, z) Minimize f(x) (1.1.1) Subject to g(x) K y, h(x) = z, x C,

8 1.1 Convex Functions and Sets 5 where C is a closed set, f : X R is lower semicontinuous, g : X Y is lower semicontinuous with respect to K, and h : X Z is continuous. We will use v(y, z) to represent the optimal value function v(y, z) := inf{f(x): g(x) K y, h(x) = z, x C}, which may take values ± (in infeasible or unbounded below cases), and S(y, z) the (possibly empty) solution set of problem P (y, z). A concrete example is P (y, z) Minimize f(x) (1.1.2) Subject to g i (x) y m, m = 1, 2,..., M, h k (x) = z k, k = 1, 2,..., K x C R N, where C is a closed subset, f, g m : R N R are lower semicontinuous and h k : R N R are continuous. Defining vector valued function g = (g 1, g 2,..., g M ) and h = (h 1, h 2,..., h k ) problem (1.1.2) becomes problem (1.1.1) with K = R M+. Beside euclidean spaces, for applications in this class we will often need to consider the Banach space of random variables. It turn out that the optimal value function of a convex programming problem is convex. Proposition (Convexity of optimal value function) Suppose that in the constriained optimization problem (1.1.1), function f is convex, mapping g is K convex, and mapping h is affine and set C is convex. Then the optimal value function v is convex. Proof. Consider (y i, z i ), i = 1, 2 in the domain of v and an arbitrary ε > 0. We can find x i ε feasible to the constraint of problem P (y i, z i ) such that Now for any λ [0, 1], we have f(x i ε) < v(y i, z i ) + ε, i = 1, 2. (1.1.3) f(λx 1 ε + (1 λ)x 2 ε) λf(x 1 ε) + (1 λ)f(x 2 ε) (1.1.4) < λv(y 1, z 1 ) + (1 λ)v(y 2, z 2 ) + ε. It is easy to check that λx 1 ε + (1 λ)x 2 ε is feasible for problem P (λ(y 1, z 1 ) + (1 λ)(y 2, z 2 )). Thus, v(λ(y 1, z 1 ) + (1 λ)(y 2, z 2 )) f(λx 1 ε + (1 λ)x 2 ε). Combining with inequality (1.1.4) and letting ε 0 we arrive at v(λ(y 1, z 1 ) + (1 λ)(y 2, z 2 )) λv(y 1, z 1 ) + (1 λ)v(y 2, z 2 ), that is to say v is convex. This is a very potent result that can help us to recognize the convexity of many other functions. For example, let C be a convex set then the distance function to C defined by d C (z) = inf[ z c : c C] is a convex function because we can rewrite it as the optimal value of the following special case of problem (1.1.1)

9 6 1 Convex Duality d C (z) = inf[ x : x + c = z, c C]. Similarly, the inf-converlusion function defined below is convex f g(z) := inf[f(z x) + g(x)] y = inf{f(u) + r : g(x) r 0, u + x = z} While the value function of a convex programming problem is always convex it is not necessarily smooth even if all the data involved are smooth. The following is an example { v(y) = inf[x : x 2 y y 0 y] = + y < Subdifferential and Lagrange multiplier Many naturally arising nonsmooth convex functions lead to the definition of subdifferential as a replacement for the nonexisting derivative Definition Definition (Subdifferential) Let X be a finite dimensional Banach space. The subdifferential of a lower semicontinuous function ϕ : X R {+ } at x dom ϕ is defined by ϕ(x) = {x X : ϕ(y) ϕ(x) x, y x for all y X}. We define the domain of the subdifferential of ϕ by dom ϕ = {x X ϕ(x) }. An element of ϕ(x) is called a subgradient of ϕ at x. Definition (Normal cone) For a closed convex set C X, we define the normal cone of C at x C by N(C; x) = ι C ( x). Sometimes we will also use the notation N C ( x) = N(C; x). A useful characterization of the normal cone is x N(C; x) if and only if, for all y C, x, y x 0. It is easy to verify that if f has a continuous derivative at x then f(x) = {f (x)}. At a nondifferentiable point a convex function s subdifferential is usually a set. Here are a few examples. Example We can easily verify that (0) = B 1 (0), the closed ball centered at 0 with radius 1, in particular, (0) = [ 1, 1]. ( ) + (0) = [0, + ). ( ) (0) = (, 0].

10 1.2.2 Nonemptyness of subdifferential 1.2 Subdifferential and Lagrange multiplier 7 A natural and important question is when can we ensure the subdifferential is nonempty. The following Fenchel - Rockafellar theorem provides a basic form of sufficient conditions. Theorem (Fenchel - Rockafellar Theorem on Nonemptiness of Subdifferential) Let f : X R {+ } be a convex function. Suppose that x int(dom f). Then the subdifferential f( x) is nonempty. Proof. We observe that ( x, f( x)) is a boundary point of the closed set epi f which has a nonempty interior. Thus by the Hahn-Banach extension theorem there exists a supporting hyperplane of epi f at ( x, f( x)) whose normal vector is (0, 0) (x, r) X R. Now, for any x dom f and u f(x), we have r(u f( x)) + x, x x 0. (1.2.5) Since u f(x) is arbitrary, r 0. Moreover, if r = 0, then x int dom f would imply also x = 0, a contradiction. Thus, r > 0. Letting u = f(x) in (1.2.5) we see that x /r f( x). The Fenchel Rockafellar Theorem is a fundamental result that we will use often in the sequel. Condition x int(dom f) is a sufficient condition that can be improved. Notice that we don t need to worry about points at which f =. Thus, we need only check the condition of Theorem on span(dom f). Thus, condition x int(dom f) can be revised to x ri(dom f) and f is lsc, where ri signifies the relative interior, i.e. interior points on span(dom f). Remark Recall that a set is polydedral if it is the intersection of finitely many closed half-spaces. A function is polydedral if it s epigraph is a polydedral set. For a polyhedral function its subdifferential is nonempty in any point of its domain (see e.g. [6]). This sufficient condition is very useful in dealing with linear programming problems. The conclusion f( x) can be stated alternatively as there exists a linear functional x such that f x attains its minimum at x. This is a very useful perspective on the use of variational arguments deriving results by observing a certain auxiliary function attains a minimum or maximum Calculus For more complicated convex functions we need the help of a convenient calculus for calculating or estimating its subdifferential. It turns out the key for developing such a calculus is to combine a decoupling mechanism with the existence of subgradient. We summarize this idea in the following lemma.

11 8 1 Convex Duality Lemma (Decoupling Lemma) Let the functions f : X R and g : Y R be convex and let A: X Y be a linear transform. Suppose that f, g and A satisfy the condition 0 ri[dom g A dom f]. (1.2.6) Then there is a y Y such that for any x X and y Y, where p = inf X{f(x) + g(ax)}. p [f(x) y, Ax ] + [g(y) + y, y ], (1.2.7) Proof. Define an optimal value function v : Y [, + ] by v(u) = inf {f(x) + g(ax + u)} x X = inf {f(x) + g(y) : y Ax = u.} (1.2.8) x X Proposition implies that v is convex. Moreover, it is easy to check that dom v = dom g A dom f so that the constraint qualification condition (1.2.6) ensures that v(0). Let y v(0). By definition we have v(0) = p v(y Ax) + y, y Ax f(x) + g(y) + y, y Ax. (1.2.9) We apply the decoupling lemma of Lemma to establish a sandwich theorem. Theorem (Sandwich Theorem) Let f : X R {+ } and g : Y R {+ } be convex functions and let A: X Y be a linear map. Suppose that f g A and f, g and A satisfy condition (1.2.6). Then there is an affine function α: X R of the form α(x) = A y, x + r satisfying f α g A. Moreover, for any x satisfying f( x) = g A( x), we have y g(a x). Proof. By Lemma there exists y Y such that for any x X and y Y, 0 p [f(x) y, Ax ] + [g(y) + y, y ]. (1.2.10) For any z X setting y = Az in (1.2.10) we have Thus, f(x) A y, x g(az) A y, z. (1.2.11) a := inf [f(x) x X A y, x ] b := sup[ g(az) A y, z ]. Picking any r [a, b], α(x) := A y, x +r is an affine function that separates f and g A. Finally, when f( x) = g A( x), it follows from (1.2.10) that y g(a x). We now use the tools established above to deduce calculus rules for the convex functions. We start with a sum rule playing a role similar to the sum rule for derivatives in calculus. z X

12 1.2 Subdifferential and Lagrange multiplier 9 Theorem (Convex Subdifferential Sum Rule) Let f : X R {+ } and g : Y R {+ } be convex functions and let A: X Y be a linear map. Then at any point x in X, we have the sum rule with equality if condition (1.2.6) holds. (f + g A)(x) f(x) + A g(ax), (1.2.12) Proof. Inclusion (1.2.12) is easy and left as an exercise. We prove the reverse inclusion under condition (1.2.6). Suppose x (f + g A)( x). Since shifting by a constant does not change the subdifferential of a convex function, we may assume without loss of generality that x f(x) + g(ax) x, x attains its minimum 0 at x = x. By the sandwich theorem there exists an affine function α(x) := A y, x + r with y g(a x) such that f(x) x, x α(x) g(ax). Clearly equality is attained at x = x. It is now an easy matter to check that x + A y f( x). Note that when A is the identity mapping and both f and g are differentiable Theorem recovers sum rules in calculus. The geometrical interpretation of this is that one can find a hyperplane in X R that separates the epigraph of f and hypograph of g i.e. {(x, r) : g(x) r}. By applying the subdifferential sum rule to the indicator functions of two convex sets we have parallel results for the normal cones to the intersection of convex sets. Theorem (Normals to an Intersection) Let C 1 and C 2 be two convex subsets of X and let x C 1 C 2. Suppose that C 1 int C 2. Then N(C 1 C 2 ; x) = N(C 1 ; x) + N(C 2 ; x). Proof. C 2. Applying the subdifferential sum rule to the indicator functions of C 1 and The condition (1.2.6) is often referred to as a constraint qualification. Without it the equality in the convex subdifferential sum rule may not hold (Exercise??) Role in convex programming Subdifferential plays important roles in convex programming. First for unconstrained convex minimization problem we have Fermat s rule: Proposition (Subdifferential at Optimality) Let X be a Banach space and let f : X R {+ } be a proper convex function. Then the point x X is a (global) minimizer of f if and only if the condition 0 f( x) holds.

13 10 1 Convex Duality Proof. We only need to observe thatt x X is a minimizer of f iff f(x) f( x) 0 = 0, x x which by definition is equivalent to 0 f( x). Alternatively put, minimizers of f correspond exactly to zeroes of f. Consider the constrained convex optimization problem of CP minimize f(x) subject to x C X, (1.2.13) where C is a closed convex subset of X and f : X R {+ } is a convex lsc function. Combining the Fermat s rule with the subdifferential sum rule we derive a characterization for solutions to CP. Theorem (Pshenichnii Rockafellar Conditions) Let C be a closed convex subset of R N and let f : X R {+ } be a convex function. Suppose that C cont f and f is bounded below on C. Then x is a solution of CP if and only if it satisfies 0 f( x) + N(C; x). Proof. Apply the convex subdifferential sum rule of Theorem to f + ι C at x. Finally we turn to the relationship between subdifferential of optimal value functions in convex programming and Lagrange multipliers. We shall see from the two versions of Lagrange multiplier rules given below, the subdifferential of the optimal value function completely characterizes the set of Lagrange multipliers (denoted λ in these theorems). Theorem (Lagrange multiplier without existence of optimal solution) Let v(y, z) be the optimal value function of the constrained optimization problem P (y, z). Then λ v(0, 0) if and only if (i) (nonnegativity) λ K + Z ; and (ii) (unconstrained optimum) for any x C, f(x) + λ, (g(x), h(x)) v(0, 0). Proof. (a) The only if part. Suppose that λ v(0, 0). It is easy to see that v(y, 0) is non-increasing with respect to the partial order K. Thus, for any y K, 0 v(y, 0) v(0, 0) λ, (y, 0) so that λ K + Z. Conclusion (ii) follows from, the fact that for all x C, f(x) + λ, (g(x), h(x)) v(g(x), h(x)) + λ, (g(x), h(x)) v(0, 0). (1.2.14)

14 1.2 Subdifferential and Lagrange multiplier 11 (b) The if part. Suppose λ satisfies conditions (i) and (ii). Then we have, for any x C, g(x) K y and h(x) = z, f(x) + λ, (y, z) f(x) + λ, (g(x), h(x)) v(0, 0). (1.2.15) Taking the infimum of the leftmost term under the constraints x C, g(x) K y and h(x) = z, we arrive at Therefore, λ v(0, 0). v(y, z) + λ, (y, z) v(0, 0). (1.2.16) If we denote by Λ(y, z) the multipliers satisfying (i) and (ii) of Theorem then we may write the useful set equality Λ(0, 0) = v(0, 0). The next corollary is now immediate. It is often a useful variant since h may well be affine. Corollary (Lagrange multiplier without existence of optimal solution) Let v(y, z) be the optimal value function of the constrained optimization problem P (y, z). Then λ v(0, 0) if and only if (i) (nonnegativity) λ K + Z ; (ii) (unconstrained optimum) for any x C, satisfying g(x) K y and h(x) = z, f(x) + λ, (y, z) v(0, 0). When an optimal solution for the problem P (0, 0) exists, we can also derive a so called complementary slackness condition. Theorem (Lagrange multiplier when optimal solution exists) Let v(y, z) be the optimal value function of the constrained optimization problem P (y, z). Then the pair ( x, λ) satisfies λ v(0, 0) and x S(0, 0) if and only if (i) (nonnegativity) λ K + R K ; (ii) (unconstrained optimum) the function x f(x) + λ, (g(x), h(x)) attains its minimum over C at x; (iii)(complementary slackness) λ, (g( x), h( x)) = 0. Proof. (a) The only if part. Suppose that x S(0, 0) and λ v(0, 0). As in the proof of Theorem we can show that λ K + Z. By the definition of the subdifferential and the fact that v(g( x), h( x)) = v(0, 0), we have 0 = v(g( x), h( x)) v(0, 0) λ, (g( x), h( x)) 0, so that the complementary slackness condition λ, (g( x), h( x)) = 0 holds.

15 12 1 Convex Duality Observing that v(0, 0) = f( x)+ λ, (g( x), h( x)), the strengthened unconstrained optimum condition follows directly from that of Theorem (b) The if part. Let λ, x satisfy conditions (i), (ii) and (iii). Then, for any x C satisfying g(x) K 0 and h(x) = 0, f(x) f(x) + λ, (g(x), h(x)) (1.2.17) f( x) + λ, (g( x), h( x)) = f( x). That is to say x S(0, 0). Moreover, for any g(x) K y, h(x) = z, f(x)+ λ, (y, z) f(x)+ λ, (g(x), h(x)). Since v(0, 0) = f( x), by (1.2.17) we have f(x) + λ, (y, z) f( x) = v(0, 0). (1.2.18) Taking the infimum on the left hand side of (1.2.18) yields which is to say, λ v(0, 0). v(y, z) + λ, (y, z) v(0, 0), We can deduce from Theorems and that v(0, 0) completely characterizes the set of Lagrange multipliers. 1.3 Fenchel Conjugate Obtaining Lagrange multipliers by using convex subdifferentials is closely related to convex duality theory based on the concept of conjugate functions introduced by Fenchel The Fenchel Conjugate The Fenchel conjugate of a function (not necessarily conveex) f : X [, + ] is the function f : X [, + ] defined by f (x ) = sup{ x, x f(x)}. x X The function f is convex and if the domain of f is nonempty then f never takes the value. Clearly the conjugacy operation is order-reversing : for functions f, g : X [, + ], the inequality f g implies f g. We can consider the conjugate of f called the biconjugate of f and denoted f. This is a function on X. When X is a reflexive Banach space, i.e. X = X it follows from the Fenchel Young inequality (1.3.2) that f f. The function f is the largest among all the convex function dominated by f and is called the convex hull of f. Many important convex functions f on X = R N equals to their biconjugate f. Such functions thus occur as natural pairs, f and f. Table 1.1 shows some elegant examples on R and Table 1.2 describes some simple transformations of these examples. Check the calculation in Table 1.1 and verify the formulas in Table 1.2 are good exercises to get familiar with concept of conjugate functions. Note that the first four functions in Table 1.1 are special cases of indicator functions on R. A more ganeral result is:

16 1.3 Fenchel Conjugate 13 Example Let C be a closed convex set in the reflexive Banach spacce X. Then ι C = σ C and σ C = ι C. f(x) = g (x) dom f g(y) = f (y) dom g 0 R 0 {0} 0 R + 0 R + 0 [ 1, 1] y R 0 [0, 1] y + R x p /p, p > 1 R y q /q ( = 1) p q R x p /p, p > 1 R + y + q /q ( = 1) p q R x p /p, 0<p<1 R + ( y) q /q ( 1 p + 1 q = 1) int R + log x int R + 1 log( y) int R + e x R { y log y y (y > 0) 0 (y = 0) R + Table 1.1. Conjugate pairs of convex functions on R. f = g f(x) h(ax) (a 0) h(x + b) ah(x) (a > 0) g = f g(y) h (y/a) h (y) by ah (y/a) Table 1.2. Transformed conjugates.

17 14 1 Convex Duality The Fenchel Young Inequality This is an elementary but important result that relates conjugate operation with the subgradient. Proposition (Fenchel Young Inequality) Let f : X R {+ } be a convex function. Suppose that x X and x dom f. Then Equality holds if and only if x f(x). f(x) + f (x ) x, x. (1.3.1) Proof. equality The inequality (1.3.1) follows directly from the definition. We have the f(x) + f (x ) = x, x, if and only if, for any y X, f(x) + x, y f(y) x, x. That is or x f(x). f(y) f(x) x, y x, Combining Fenchel inequality and the sandwich theorem we can show that f = f for convex lsc function f. Theorem (Biconjugate) Let X be a finite dimensional Banach space. Then f f in dom f and equality holds at point x int dom f. Proof. It is easy to check f f and we leave it as an exercise. For any x int dom f, f( x). Let x f( x). By the Fenchel inequality we have f( x) = x, x f (x ) sup y [ y, x f (y )] = f ( x) f( x). Remark The order reversing property and the self-involution property described in Theorem 1.3.3, in fact, characterizes the Fenchel -Legendre transform up to a linear transform. This is a deep result derived in []. To relate Fenchel duality and convex programming with linear constraints, we let g be the indicator function of a point, which gives the following particularly elegant and useful corollary.

18 1.3 Fenchel Conjugate 15 Corollary (Fenchel Duality for Linear Constraints) Given any function f : X R {+ }, any bounded linear map A: X Y, and any element b of Y, the weak duality inequality inf {f(x) Ax = b} sup { b, x f (A x )} x X x Y holds. If f is lsc and convex and b ri(a dom f) then equality holds, and the supremum is attained when finite Graphic illustration and generalizations For increasing function ϕ, ϕ(0) = 0, f(x) = x 0 ϕ(s)ds is convex and f (x ) = x 0 ϕ 1 (t)dt. Graphs Fig and Fig illustrates the Fenchel-Young inequality graphically. The additional areas enclosed by the graph of ϕ 1, s = x and t = x or that of ϕ, s = x and t = x beyond the area of the rectangle [0, x] [0, x ] generates the additional area that leads to a strict inequality. We also see that equality holds when x = ϕ(x) = f (x) and x = ϕ 1 (x ) = (f ) (x ) in Fig x t ϕ 1 ϕ O x s Fig Fenchel-Young inequality t x ϕ 1 ϕ O x s Fig Fenchel-Young inequality

19 16 1 Convex Duality t x ϕ 1 ϕ O x s Fig Fenchel-Young equality 1.4 Convex duality theory Using the Fenchel-Young inequality for each constrained optimization problem we can write its companion dual problem. There are several different but equivalent perspectives Rockafellar duality We start with the Rockafellar formulation of bi-conjugate. It is very general and as we shall see other perspectives can easily be written as special cases. Consider a two variable function F (x, y) on X Y where X, Y are Banach spaces. Treating y as a parameter, consider the parameterized optimization problem Our associated primal optimization problem 1 is and the dual problem is v(y) = inf F (x, y). (1.4.1) x p = v(0) = inf F (x, 0) (1.4.2) x X d = v (0) = sup F (0, y ). (1.4.3) y Y Since v dominates v as the Fenchel-Young inequality establishes, we have v(0) = p d = v (0). This is called weak duality and the non-negative number p d = v(0) v (0) is called the duality gap which we aspire to be small or zero. 1 The use of the term primal is much more recent than the term dual and was suggested by George Dantzig s father Tobias when linear programming was being developed in the 1940 s.

20 1.4 Convex duality theory 17 Let F (x, (y, z)) := f(x) + ι epi(g) (x, y) + ι graph(h) (x, z). Then problem P (y, z) becomes problem (1.4.1) with parameters (y, z). On the other hand, we can rewrite (1.4.1) as v(y) = inf{f (x, u) : u = y} x which is problem P (0, y) with x = (x, u), C = X Y, f(x, u) = F (x, u), h(x, u) = u and g(x, u) = 0. So where we start is a metter of taste and predisposition. Theorem (Duality and Lagrange multipliers) The following are equivalent: (i) the primal problem has a Lagrange multiplier λ. (ii) there is no duality gap, i.e. d = p is finite and the dual problem has solution λ. Proof. If the primal problem has a Lagrange multiplier λ then λ v(0). By the Fenchel-Young equality Direct calculation yields Since v(0) + v ( λ) = λ, 0 = 0. v ( λ) = sup{ λ, y v(y)} y = sup{ λ, y F (x, y)} = F (0, λ). y,x F (0, λ) v (0) v(0) = v ( λ) = F (0, λ), (1.4.4) λ is a solution to the dual problem and p = v(0) = v (0) = d. On the other hand, if v (0) = v(0) and λ is a solution to the dual problem then all the quantities in (1.4.4) are equal. In particular, v(0) + v ( λ) = 0. This implies that λ v(0) so that λ is a Lagrange multiplier of the primal problem. Example (Finite duality gap) Consider v(y) = inf{ x 2 1 : x x2 2 x 1 y}. We can easily calculate 0 y > 0 v(y) = 1 y = 0 + y < 0, and v (0) = 0, i.e. there is a finite duality gap v(0) v (0) = 1. In this example neither the primal nor the dual problem has a Lagrange multiplier yet both have solutions. Hence, even in two dimensions, existence of a Lagrange multiplier is only a sufficient condition for the dual to attain a solution and is far from necessary.

21 18 1 Convex Duality Fenchel duality Let us specify F (x, y) := f(x) + g(ax + y), where A : X Y is a linear operator. We thence get the Fenchel formulation of duality. Now the primal problem is To derive the dual problem we calculate Letting u = Ax + y we have Thus, the dual problem is p = v(0) = inf[f(x) + g(ax)]. (1.4.5) x F (0, y ) = sup[ y, y f(x) g(ax + y)]. x,y F (0, y ) = sup[ y, u Ax f(x) g(u)] x,u = sup[ y, Ax f(x)] + sup[ y, u g(u)] x u = f (A y ) + g ( y ). d = v (0) = sup y [ f (A y ) g ( y )]. (1.4.6) If both f and g are convex functions it is easy to see that so is v(y) = inf[f(x) + g(ax + y)]. x We have already checked that dom v = dom g A dom f. Thus, a sufficient condition for the existence of Lagrange multipliers for the primal problem, i.e., v(0), is 0 ri dom v = ri[dom g A dom f]. (1.4.7) Figure 1.1 illustrate the Fenchel duality theorem for f(x) := x 2 /2+1 and g(x) = (x 1) 2 /2 + 1/2. The upper function is f and the lower one is g. The minimum gap occurs at 1/2 and is 7/4. In infinite dimensions more care is necessary but in Banach space when the functions are lower semicontinuous and the operator is closed 0 core dom v = core[dom g A dom f]. (1.4.8) suffices. Here, for a given set S, its core is defined as core(s) := {s S : λ 0 λ(s s) = X}. A condition of the form (3.3.12) or (1.4.8) is often referred to as a constraint qualification or a transversality condition. Enforcing such constraint qualification conditions we can write Theorem in the following form: Theorem (Duality and constraint qualification) If the convex functions f, g and the linear operator A satisfy the constraint qualification conditions (3.3.12) or (1.4.8) then there is a zero duality gap between the primal and dual problems, (1.4.5) and (1.4.6), and the dual problem has a solution.

22 1.4 Convex duality theory Fig The Fenchel duality sandwich A really illustrative example is the application to entropy optimization. Example (Entropy optimization problem) Entropy maximization refers to minimize f(x) subject to Ax = b R N, (1.4.9) with the lower semicontinuous convex function f defined on a Banach space of signals, emulating the negative of an entropy and A emulating a finite number of continuous linear constraints representing conditions on some given moments. A wide variety of applications can be covered by this model due to its physical relevance. Applying Theorem with g = ι {b} we have if b ri(a dom f) then inf {f(x) Ax = b} = max x X ϕ R N{ ϕ, b f (A ϕ)}. (1.4.10) When N < dim X (often infinite) the dual problem is typically much easier to solve then the primary. Example (Boltzmann Shannon entropy in Euclidean space) Let where N f(x) := p(x n), (1.4.11) n=1 t ln t t if t > 0, p(t) := 0 if t = 0, + if t < 0.

23 20 1 Convex Duality The functions p and f defined above are (negatives of) Boltzmann Shannon entropy functions on R and R N, respectively. For c R N, b R M and linear mapping A : R N R M consider the entropy optimization problem minimize {f(x) + c, x : Ax = b}. (1.4.12) Example can help us conveniently derive an explicit formula for solutions of (1.4.12) in terms of the solution to its dual problem. First we note that the sublevel sets of the objective function are compact, thus ensuring the existence of solutions to problem (1.4.12). We can also see by direct calculation that the directional derivative of the cost function is on any boundary point x of dom f = R N +, the domain of the cost function, in the direction of z x. Thus, any solution of (1.4.12) must be in the interior of R N +. Since the cost function is strictly convex on int (R N + ), the solution is unique. Let us denote this unique solution of (1.4.12) by x. Then the duality result in Example implies that f( x) + c, x = inf + c, x : Ax = b} x RN{f(x) = max ϕ R M{ ϕ, b (f + c) (A ϕ)}. Now let ϕ be a solution to the dual problem, i.e,, a Lagrange multiplier for the constrained minimization problem (1.4.12). We have f( x) + c, x + (f + c) (A ϕ) = ϕ, b = ϕ, A x = A ϕ, x. It follows from the Fenchel-Young equality that A ϕ (f + c)( x). Since x int (R N + ) where f is differentiable, we have A ϕ = f ( x) + c. Explicit computation shows x = ( x 1,..., x N ) is determined by x n = exp(a ϕ c)n, n = 1,..., N. (1.4.13) Indeed, we can use the existence of the dual solution to prove that the primal problem has the given solution without direct appeal to compactness we deduce the existence of the primal from the duality theory. Remark We say a function f is polyhedral if its epigraph is polyhedral, i.e. finite intersection of closed half-spaces. When both f and g are polyhedral functions the constraint qualification condition (3.3.12) simplifies to dom g A dom f. (1.4.14) This is very useful in dealing with polyhedral cone programming and, in particular, linear programming problems. One can also similarly handle a subset of polyhedral constraints [6, 7].

24 1.4 Convex duality theory Lagrange duality For problem (1.1.1) define the Lagrangian Then L(λ, x; (y, z)) = f(x) + λ, (g(x) y, h(x) z). sup L(λ, x; (y, z)) = λ K + Z Then problem (1.1.1) can be written as We can calculate { f(x) if g(x) K y, h(x) = z. + otherwise. p = v(0) = inf sup L(λ, x; 0). (1.4.15) x C λ K + Z v ( λ) = sup[ λ, (y, z) v(y, z)] y,z = sup[ λ, (y, z) inf {f(x) : g(x) K y, h(x) = z}] y,z x C = sup { λ, (y, z) f(x) : g(x) K y, h(x) = z}. x C,y,z Letting ξ = y g(x) K we can rewrite the expression above as v ( λ) = Thus, the dual problem is sup [ λ, (g(x), h(x)) f(x) + λ, (ξ, 0) ] x C,ξ K = inf L(x, λ, 0) λ, (ξ, 0) x C,ξ K { inf x L(x, λ, 0) if λ K + Z = + otherwise. d = v (0) = sup v ( λ) = λ sup λ K + Z x C inf L(λ, x; 0). (1.4.16) We can see that the weak duality inequality v(0) v (0) is simply the familiar fact that inf sup sup inf. Example (Classical linear programming duality) Consider a linear programming problem max c, x (1.4.17) subject to Ax b, x 0 where x R N, b R M, A is a M N matrix and = R M +. Then by the Lagrange duality the dual problem is min b, λ (1.4.18) subject to A λ c, λ 0.

25 22 1 Convex Duality In fact, deal with the minimizing problem min[ c, x : Ax b, x 0] = max[ c, x : Ax b, x 0] We write the Lagrangian L(λ, x) = c, x + λ, Ax b Then the primal problem is The dual problem is inf sup x 0 λ 0 L(λ, x). sup inf L(λ, x). λ 0 x 0 We can see that So we have inf L(λ, x) = inf c + x 0 x 0 A λ, x λ, b = { λ, b if A λ c + otherwise. max[ c, x : Ax b, x 0] = max λ 0 [ λ, b : A λ c] = min[ λ, b : A λ c, λ 0]. Clearly all the functions involved here are polyhedral. Applying the constraint qualification condition for polyhedral functions we can conclude that if either the primary problem or the dual problem is feasible then there is no duality gap. Moreover, when the common optimal value is finite then both problems have optimal solutions. The hard work in Example was hidden in establishing that the constraint qualification (1.4.14) is sufficient, but unlike many applied developments we have rigorously recaptured linear programming duality within our framework. Note that the primal Lagrange multiplier λ is the dual solution and vice versa we have the following table in helping formulate the dual problem: Generalized Fenchel-Young inequality Reexamining the graphic representation of the Fenchel -Young inequality we also realize that the underlying inequality relationship remains valid when the area is weighted by a positive density function K(s, t). Thus, we have Theorem (Weighted Fenchel-Young inequality) Let K(x, y) be a continuous positive function and let ϕ be a continuous increasing function with ϕ(0) = 0. Then x x K(s, t)dtds x ϕ(s) K(s, t)dtds + x ϕ 1 (t) and equality holds when x = ϕ(x) and x = ϕ 1 (x ). K(s, t)dsdt

26 1.4 Convex duality theory 23 Primal constraint Dual variable P.variable D.constraint Ax b λ 0 x 0 A λ c Ax = b λ free x free A λ = c Ax b λ 0 x 0 A λ c Table 1.3. Transformed conjugates. Proof. If ϕ(x) x we have x ϕ(s) 0 0 x x 0 0 x x 0 0 x K(s, t)dtds + K(s, t)dsdt + K(s, t)dtds. Otherwise, ϕ(x) < x and we have x ϕ(s) 0 0 x x 0 0 x x x ϕ 1 (x ) x K(s, t)dtds + K(s, t)dsdt + K(s, t)dtds. Clearly equality holds if and only if ϕ(x) = x. ϕ 1 (t) 0 ϕ(s) x ϕ 1 (t) 0 0 ϕ 1 (x ) x x ϕ(s) K(s, t)dsdt (1.4.19) K(s, t)dtds K(s, t)dsdt (1.4.20) K(s, t)dtds The condition ϕ(0) = 0 merely conveniently locate the lower left conner of the graph to the coordinate origin and is clearly not essential. In general we can always shift this corner to any point (a, ϕ(a)). More substantively, the requirment that ϕ being a continuous increasing function can be relaxed to nondecresing as long as ϕ 1 is replaced appropriately by ϕ 1 inf (t) = inf{s, ϕ(s) t}. Now we can state a more general Fenchel- Young inequality whose proof is an easy exercise. Theorem (Weighted Fenchel-Young inequality) Let K(x, y) be a bounded essentially positive measurable function and let ϕ be a nondecreasing function. Then x x a ϕ(a) K(s, t)dsdt x ϕ(s) a ϕ(a) x ϕ 1 K(s, t)dtds + ϕ(a) a inf (t) K(s, t)dsdt

27 24 1 Convex Duality With equality attiained when x [ϕ(x ), ϕ(x+)], x [ϕ 1 inf (x ), ϕ 1 inf (x +)]. The above idea can be further pushed in two different directions in the next two sections. n-dimensional Fenchel Young inequality It is easier to understand and to formulate n-dimensional Fenchel Young inequality starting by re-examining the graphs presented above with a parameterization (ϕ 1, ϕ 2 ) of the graph of ϕ in Fig ϕ 2 ϕ 2(b 2) (ϕ 1(t), ϕ 2(t)) (ϕ 1(a), ϕ 2(a)) ϕ 1(b 1) ϕ 1 Fig Fenchel-Young inequality b 2 > b 1 ϕ 2 ϕ 2(b 2) (ϕ 1(t), ϕ 2(t)) (ϕ 1(a), ϕ 2(a)) ϕ 1(b 1) ϕ 1 Fig Fenchel-Young inequality b 2 < b 1

28 1.4 Convex duality theory 25 ϕ 2 ϕ 2(b 1) (ϕ 1(t), ϕ 2(t)) (ϕ 1(a), ϕ 2(a)) ϕ 1(b 1) ϕ 1 Fig Fenchel-Young equality b 2 = b 1 Let K(s 1, s 2 ) be a nonnegative function and let ϕ 1, ϕ 2 be increasing functions. To avoid technical complication we assume ϕ 1, ϕ 2 are invertible. Then we can rewrite the Fenchel-Young inequality as ϕ2 (b 2 ) ϕ1 (b 1 ) ϕ 2 (a) ϕ 1 (a) ϕ1 (b 1 ) ϕ2 (ϕ 1 1 (s 1)) ϕ 1 (a) ϕ 2 (a) K(s 1, s 2 )ds 1 ds 2 (1.4.21) K(s 1, s 2 )ds 2 ds 1 + ϕ2 (b 2 ) ϕ1 (ϕ 1 2 (s 2)) ϕ 2 (a) ϕ 1 (a) K(s 1, s 2 )ds 1 ds 2 with equality attained when b 1 = b 2. This form of the Fenchel-Young inequality can easily be generalized to N- dimention with an induction argument. We will use the following vector notation: s N = (s 1,..., s N ), 1 N = (1, 1,..., 1) and s N n = (s 1,..., s n 1, s n+1,..., s N ). When ϕ N = (ϕ 1,..., ϕ N ) is a vector valued function we define Similarly, ϕ N (b N ) ϕ N (a N ) K(s N )ds N = ϕ N (s N ) = (ϕ 1 (s 1 ),..., ϕ N (s N )). ϕn (b 1 ) ϕ 1 (a 1 )... ϕn (b N ) ϕ N (a N ) K(s 1,..., s N )ds N... ds 1. Theorem (N-dimensional generalized Fenchel-Young inequality) Let K : R N R be a nonnegative function and let ϕ N be vector function with all the components increasing and invertible. We have ϕ N (b N ) ϕ N (a 1 N ) K(s N )ds N N n=1 ϕn (b n ) ϕ n (a) with equality attained when b 1 = b 2 =... = b N. ϕ N n (ϕ 1 n (s n) 1 N 1 ) ϕ N n (a 1N 1 ) K(s N )ds N n ds n

29 26 1 Convex Duality Proof. We prove by induction. The case N = 2 has already been established. We focus on the induction step. By separating the integration wirh respect to ds N+1, we can write the left hand side of the inequality as LHS = = ϕ N (b N+1 ) K(s N+1 )ds N+1 ϕ N+1 (a 1 N+1 ) ϕn+1 (b N+1 ) ϕ N (b N ) ϕ N+1 (a) ϕ N (a 1 N ) K(s N+1 )ds N ds N+1 Applying the induction hypothesis to the inner layer of the integration we have LHS = ϕn+1 (b N+1 ) N ϕn (b n ) ϕ N+1 (a) n=1 ϕ n(a) ϕn+1 (b N+1 ) ϕn (b n ) N n=1 ϕ N+1 (a) ϕ n (a) ϕ N n (ϕ 1 n (s n) 1 N 1 ) ϕ N n (a 1N 1 ) ( ϕ N n (ϕ 1 n (s n) 1 N 1 ) ϕ N n (a 1N 1 ) K(s N+1 )ds N n ds nds N+1 K(s N+1 )ds N n ) ds n ds N+1. The last equality groups the two out layers of the integration together. Now Applying the Fenchel-Young inequality with N = 2 to get LHS + N ϕn(b n) ϕn+1 (ϕ 1 n (sn)) ϕ N n (ϕ 1 n (sn) 1N 1 ) n=1 ϕ n(a) ϕ N+1 (a) ϕ N n (a 1N 1 ) ϕn+1 (b N+1 ) N ϕn (ϕ 1 N+1 (s N+1)) ϕ N n (ϕ 1 n (s n) 1 N 1 ) ϕ N+1 (a) n=1 ϕ n (a) ϕ N n (a 1N 1 ) K(s N+1 )ds N n ds N+1ds n K(s N+1 )ds N n ds n ds N+1 Combining the inner layersof the integration in the first sum and applying the equality part of the induction hypothesis for the second sum we arrive at LHS + = N ϕn(b n) ϕ N+1 n (ϕ 1 n (sn) 1N ) n=1 ϕ n(a) ϕ N+1 n (a 1 N ) ϕn+1 (b N+1 ) ϕ N (ϕ 1 N+1 (s N+1) 1 N ) ϕ N+1 (a) N+1 ϕn(bn) n=1 ϕ n(a) ϕ N n (a 1N ) ϕ N+1 n (ϕ 1 n (sn) 1N ) ϕ N+1 n (a 1 N ) K(s N+1 )ds N+1 n ds n K(s N+1 )ds N ds N+1 K(s N+1 )ds N+1 n ds n = RHS. Remark We also have the following alternative form of estimations by changing the way of integration. Let K(s 1, s 2) be a nonnegative function and let ϕ 1, ϕ 2 be non-decreasing functions. ϕ2 (b 2 ) ϕ1 (b 1 ) ϕ 2 (a) ϕ 1 (a) ϕ1 (b 2 ) ϕ2 (b 2 ) ϕ 1 (a) ϕ 2 (ϕ 1 1 (s 1)) K(s 1, s 2 )ds 1 ds 2 K(s 1, s 2 )ds 2 ds 1 + with equality attained when b 1 = b 2. ϕ2 (b 1 ) ϕ1 (b 1 ) ϕ 2 (a) ϕ 1 (ϕ 1 2 (s 2)) K(s 1, s 2 )ds 1 ds 2

30 1.5 Generalized conjugate and duality 1.5 Generalized conjugate and duality 27 Note that the graphic illustrations in section only works when x, x R. When in general (x, x ) X X we can immitate the general defintion of the Fenchel conjugate. In such a generalization a nonlinear function c(x, x ) replace the role of x, x just as in Theorem x x K(s, t)dsdt replacing the product x x. In 0 0 fact, x does not even have to be in X. This is a more significant generalization. For this to work one needs to first revise the concept of convexity. Definition (Generalized convexity) Let Φ be a set of extended real valued functions. We say f is Φ convex if f(x) = sup{ϕ(x) : ϕ Φ, f ϕ}. It is easy to verify that Φ convex functions are closed under supremum. Thus, every function has a largest Φ convex minorant called its Φ convex hull. Moreover, if f is Φ convex then it is coincide with its Φ convex hull. By setting Φ to be the class of affine functions we get the usual convexity with in the class of lsc functions. Similar to Fenchel conjugate we define: Definition (Generalized Fenchel conjugate) Let c be a function on X Y. We define f c(1) (y) = sup x [c(x, y) f(x)] and g c(2) (x) = sup[c(x, y) g(y)]. y They are generalizations of Fenchel conjugate. When the function c is not symmetric with respect to its two variables, the c(1) and c(2) conjugate are different. It is easy to see that the generalized Fenchel conjugate also has the order reversing property. Define Φ c(1) = {c(, y) b : y Y, b R} and Φ c(2) = {c(x, ) b : x X, b R}. Then f c(1) is Φ c(2) convex and g c(2) is Φ c(1) convex. Next we discuss some basic properties of generalized Fenchel conjugate. Theorem (Fenchel inequality and duality) Let f : X R {+ } and g : Y R {+ }. Then (i) (Fenchel inequality) f c(1) (y) c(x, y) f(x), g c(2) (x) c(x, y) g(y), (ii) (Convex hull) The Φ c(1)(c(2)) convex hull of f(g) is f c(1)c(2) (g c(2)c(1) ), (iii) (Duality) f c(1) = f c(1)c(2)c(1), g c(2) = g c(2)c(1)c(2). Proof. (i) follows directly from the definitions. To prove (ii) we observe that by (i) f(x) c(x, y) f c(1) (y). Taking sup over y we get f f c(1)c(2). On the other hand if for some y, b, f(x) c(x, y) b for all x, then b c(x, y) f(x). Taking sup over x we have b f c(1) (y). Thus, f(x) f c(1)c(2) (x) c(x, y) f c(1) (y) c(x, y) b

31 28 1 Convex Duality establishing f c(1)c(2) as the largest Φ c(1) convex function dominated by f. The proof that g c(2)c(1) is the Φ c(2) convex hull of g is similar. (iii) follows from (ii) since f c(1) is Φ c(2) convex and g c(2) is Φ c(1) convex. Remark We see from the discussion about generalized Fenchel conjugate that what is essential in dealing with conjugate operation is the closedness with respect to the sup operation. For simple convexity the key link is that a convex function is the sup of all the affine functions it dominates. A fact based on the fundamental convex separation theorem. The generalized convexity can characterize many class of functions. The following are a few examples show case the potent of this concept. Some additional examples can be found in exercises. Example Let, be the dual pairing between X and X. Define c(x, x ) = ln x, x, with ln t = for t 0. Then a function f : X R {+ } is Φ c(1) convex if and only if e f (with the convention e = 0) is sublinear. Example Let X = Y = [0, + ] and define c(x, y) = xy, with the convention a(+ ) = +. Then a function f : X R {+ } is Φ c(1) convex if and only if it is convex and nondecreasing. Example Let X be a Hilbert space and Y = R + X. Define c(x, (ρ, y)) = ρ x y 2. Then f : X R {+ } is Φ c(1) convex if and only if it is lsc and has a finite minorant ϕ Φ c(1). The concept of subdifferential and its relationship with Fenchel conjugate can also be generalized. Definition (Generalized subdifferential) Let c be a function on X Y. We say y 0 (x 0 ) is a c(1)(c(2)) subdifferential of f(g) at x 0 (y 0 ) if attains minimum at x 0 (y 0 ). f(x) c(x, y 0 )(g(y) c(x 0, y)) Notation y 0 c(1) f(x 0 )(x 0 c(2) g(y 0 )). Theorem (Fenchel equality) (i) (Fenchel equality) y 0 c(1) f(x 0) iff f(x 0) + f c(1) (y 0) = c(x 0, y 0). (ii) (Symmetry) y 0 c(1) f c(1)c(2) (x 0 ) iff x 0 c(2) f c(1) (y 0 ).

32 1.5 Generalized conjugate and duality 29 (iii)(φ convexity) c(1) f(x 0 ) implies that f is Φ c(1) convex at x 0. On the other hand f is Φ c(1) convex at x 0 implies that c(1) f(x 0 ) = c(1) f c(1)c(2) (x 0 ). Proof. The argument for proving Fenchel equality applies to (i) with y 0, x 0 replaced by c(x 0, y 0 ). The rest follows from this generalized Fenchel equality. Details are left as an exercise. Similar to the usual subdifferential we have Theorem (Cyclical monotonicity) Subdifferential c(1) f is c(1) cyclically monotone that is for any m pairs of points y i c(1) f(x i ) we have (c(x 1, y 0 ) c(x 0, y 0 )) + (c(x 2, y 1 ) c(x 1, y 1 )) (c(x 0, y m) c(x m, y m)) 0. Proof. Adding the following inequalities: f(x 1 ) f(x 0 ) c(x 1, y 0 ) c(x 0, y 0 ) f(x 2 ) f(x 1 ) c(x 2, y 1 ) c(x 1, y 1 ) f(x 0) f(x m) c(x 0, y m) c(x m, y m). and noticing all the terms on the left hand side are cancelled. Next we look at an axiomatic approash to the c conjugate. Theorem (Characterization of c conjugate) Define an operator that maps an extended valued function f on X to an extended valued function f on Y. Then is a c conjugate iff (i) (Duality) inf α f α = sup α f α (ii) (Shift reversing) (f + d) = (f) d, for all d R where c(x, y) = (ι {x} )(y). Proof. The if part: The two properties can be derived from direct computation. For property (i) For property (ii) (inf f α ) c(1) (y) = sup x = sup x = sup α [c(x, y) inf f α(x)] α sup[c(x, y) f α (x)] α sup x [c(x, y) f α (x)] = sup α fα c(1) (y).

33 30 1 Convex Duality (f + d) c(1) (y) = sup[c(x, y) (f(x) d)] x = sup[c(x, y) f(x)] + d = f c(1) (y) + d. x The only if part: The key is the representation f( ) = inf x [ι {x}( ) + f(x)]. Applying the operator to the above representation we have ( ) ( f)(y) = inf [ι {x} + f(x)] (y) x ( = sup [ι{x} + f(x)] ) (y) x where = sup[ (ι {x} )(y) f(x)] = f c(1) (y) x c(x, y) = (ι {x} )(y). Rockafellar duality Consider the bi-conjugate setting again. The primal problem is p = v(0) = inf F (x, 0) (1.5.22) x X as one of the family v(y) = inf x F (x, y) on the perturbation space Y. Let Z be the dual parameter space and let c(y, z) be a coupling function. Define the dual problem as d = v c(1)c(2) (0) = sup{c(0, z) v c(1) (z)}. (1.5.23) z Z This definition is the same as the Rockafellar duality. However, since now c(0, z) is not necesarily 0 the problem is more involved. Theorem (Dual solution set) If d = v c(1)c(2) (0) < then the optimal solution set to the dual problem is cv c(1)c(2) (0). Proof. Follows directly from definition and is left as an exercise. Also similar to the Rockafellar duality we have Theorem (Weak and Strong Duality) We always have the weak duality d = v c(1)c(2) (0) v(0) = p. Equality holds iff v is Φ c(1) convex at 0. In this case if d = p is finite then the optimal solution set to the dual problem is c(1) v(0). Proof. As before the weak duality follows easily from the Fenchel -Young inequality. To prove strong duality notice that v is Φ c(1) convex at 0 implies that c(1) v(0). Then we can check each element of c(1) v(0) is a solution to the dual problem.

34 1.5 Generalized conjugate and duality 31 Lagrangian duality Define Lagrangian for the primal problem as L(x, z) = c(0, z) F c x(z) where F x (y) := F (x, y). Then we have the Lagrange form of the primal: If F x (y) is Φ c convex for all x X at y = 0 then sup z L(x, z) = sup z Thus, the primal problem becomes {c(0, z) Fx c(1) (z)} = Fx c(1)c(2) (0) = F x (0) = F (x, 0). inf x sup L(x, z). z Next we consider the Lagrange form of the dual If c < + we have inf x L(x, z) = inf x = inf x c(1) {c(0, z) Fx (z)} {c(0, z) sup(c(y, z) F x (y))} y = c(0, z) sup y Therefore, the dual problem becomes {c(y, z) inf F (x, y)} x = c(0, z) sup{c(y, z) v(y)} = c(0, z) v c(1) (z). y sup z inf L(x, z). x We see that the primal and dual value equal iff inf x sup z L(x, z) = sup z inf L(x, z). x

35 1.6 Exercises Exercise Let C be a convex subset. Show that (1) d C is convex, (2) σ C is convex. Exercise Let f be a convex function. Show that for any a R, f 1 ((, a]) is a convex set. Exercise Show that the intersection of a family of arbitrary convex sets is convex. Conclude that f(x) := sup{f α(x) : α A} is convex (and lsc) when {f α} α A is a collection of convex (and lsc) functions. Exercise (Jensen s inequality) Prove the Jensen s inequality in Proposition Exercise (AM-GM inequality) Using Jensen s inequality to prove the AM- GM inequality: for nonnegative numbers x n, n = 1, 2,... N, Hint: Let f = ln. (x 1x 2... x N ) 1/N x1 + x xn. N Exercise Define h(u) = inf x X {f(x) + g(ax + u)}. Prove that dom h = dom g A dom f. Exercise Apply the Pshenichnii Rockafellar conditions (Theorem ) to the following two cases: (i) C a single point {x 0 } R N, (ii) C a polyhedron {x Ax b}, where b R M. Exercise Show that if ϕ is C 1 at x then ϕ(x) {ϕ (x)} and use example to illustrate that ϕ(x) maybe empty. Exercise Find the maximum and minimum of the function f(x, y, z) = x + y+z+3 on the feasible region defined by constraints x 2 +y 2 +z 2 1 and x+2y+z = 0. Exercise Representing the payoff of portfolios by random variable ξ, an investor with risk aversion factor a will try to maximize E[ξ] avar(ξ). Suppose the portfolio consists three assets that are independent with returns r 1, r 2, r 3 and variance σ 2 1, σ 2 2, σ 2 3. Solve the portfolio optimization problem for such an investor: max w 1 r 1 + w 2 r 2 + w 3 r 3 a(w 2 1σ w 2 2σ w 2 3σ 2 3) (1.6.24) subject to w 1 + w 2 + w 3 = 1, w 1, w 2, w 3 0, where w 1, w 2, w 3 are weights of the corresponding assets in the portfolio.