Fenchel-Moreau Conjugates of Inf-Transforms and Application to Stochastic Bellman Equation

Transcription

1 Fenchel-Moreau Conjugates of Inf-Transforms and Application to Stochastic Bellman Equation Jean-Philippe Chancelier and Michel De Lara CERMICS, École des Ponts ParisTech First Conference on Discrete Optimization and Machine Learning RIKEN Center for Advanced Intelligence Project July, 2018, Tokyo, Japan

2 Examples of inf-transforms in optimization Perturbation of constraints Y Yx gives and value function inf gy inf y Y f x = inf y gy y Yx δ Yx y }{{} Kx,y Two-stage linear stochastic programming f s x = inf y gy c s, x p s, y δ {y 0, Asx+bs+y 0}

3 Examples of inf-transforms in optimization continued Product from the left by a linear operator L δ Ly=x gy Lgx = inf y Moreau-Yosida approximation of g f x = inf y }{{} Kx,y 1 x y 2 } α {{} Kx,y Inf-convolution of g 1 and g 2 f x = inf g 1 x y y } {{ } Kx,y gy g 2 y

4 Examples of inf-transforms in optimization continued Lasso problem f x = inf y Supervised learning and sparsity f x = inf y 1 2 x Ay 2 2 λ y 1 }{{} sparsity, regularization lx, Ay }{{} loss function λ y 0 }{{} l0 pseudo-norm Bregman distance f x = inf Hx Hy Hx, x y y } {{ } Bregman distance Kx,y gy

5 Examples of inf-transforms in optimization continued Upper envelope representations [Rockafellar and Wolenski, 2001] V τ, ξ = inf Eτ, ξ, ξ + gξ ξ and Hamilton-Jacobi equation Question: what about their Fenchel conjugate f x = sup x, x x X + f x? hence what about dual problems?

6 Main result Two couplings Φ and Ψ, and an inf-operation with kernel K X Φ X fx inf y Y Kx, y gy K K Φ +Ψ f Φ x inf y Y K Φ +Ψ x, y g Ψ y Y Ψ Y

7 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution Conclusion

8 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution Conclusion

9 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel conjugates of Bellman functions Application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution The duality equality case Design of dual problems Fenchel-Moreau conjugate of generalized inf-convolution Conclusion

10 Basic spaces We introduce a first couple of spaces in bilinear duality X = R n X and X = R n X and a second couple of spaces in bilinear duality Y = L p Ω, F, P, R n X and Y = L q Ω, F, P, R n X p and q-integrable random variables with values in R n X, where Ω, F, P is a probability space 1 p < + and q are such that 1/p + 1/q = 1 Elements of Y = L p Ω, F, P, R n X will be denoted by bold letters like X and elements of Y = L q Ω, F, P, R n X by X All Fenchel conjugates are denoted by g, g

11 Ingredients for a stochastic optimal control problem Let time t = 0, 1,..., T be discrete, with T N Consider a stochastic optimal control problem with state space X = R n X control space U = R n U white noise process {Wt } t=1,...,t taking values in uncertainty space W = R n W and defined over the probability space Ω, F, P For each time t = 0, 1,..., T 1, we have dynamics Ft : X U W X instantaneous costs Lt : X U W [0, + ] final cost K : X [0, + ]

12 We introduce the Bellman functions We define Bellman functions by, for all x X and t = T 1,..., 0, V T x = Kx V t x = inf X,U E[ T 1 L s X s, U s, W s+1 KX T ] s=t where X t = x X, X s+1 = F s X s, U s, W s+1 and σu s σx s, for s = t,..., T 1 If the Bellman functions are measurable, they satisfy the backward Bellman inequation, for t = T 1,..., 0 Bellman function {}}{ V t x inf E[ L t x, u, W t+1 u U }{{} instantaneous cost ] V t+1 Ft x, u, W t+1 }{{} dynamics

13 Fenchel conjugates of the Bellman functions Proposition The Bellman functions satisfy the backward inequalities X V t x inf inf Hx, u, E [ V t+1 X ] X u U for t = T 1,..., 0, where the Hamiltonian H is defined by Hx, u, X = E [ L t x, u, W t+1 F t x, u, W t+1, X ] Moreover, letting { Vt } be the Fenchel conjugates of the t=0,1,...,t Bellman functions, we have, for all x X and t = T 1,..., 0, V t x inf X sup u U H, u, X x E [ Vt+1X ]

14 [Rockafellar and Wolenski, 2001] [Pennanen and Perkkiö, 2018]

15 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel conjugates of Bellman functions Application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution The duality equality case Design of dual problems Fenchel-Moreau conjugate of generalized inf-convolution Conclusion

16 Obtaining upper and lower estimates in approximations of Bellman functions I Suppose that the Bellman functions {V t } t=0,1,...,t satisfy the Bellman equation and are convex l.s.c. and proper, that is, V t = V t, t = 0, 1,..., T This is the case in Stochastic Dual Dynamic Programming SDDP, when the dynamics F t are jointly linear in state and control the instantaneous costs Lt are jointly convex in state and control the final cost K is convex together with technical assumptions

17 Obtaining upper and lower estimates in approximations of Bellman functions II The Fenchel conjugates {V t } t=0,1,...,t of the Bellman functions are convex l.s.c. and proper, by construction Suppose that they satisfy a Bellman like equation Vt x = inf H, u, X x E [ Vt+1X ] X sup u U for t = T 1,..., 0

18 Obtaining upper and lower estimates in approximations of Bellman functions III With the Bellman operators deduced from the Bellman equation and Bellman like equation, one can produce by an adequate algorithm like the SDDP algorithm lower bound functions { V k N t,k V t,k+1 V t Ṽ t,k Ṽ t,k+1 Vt that are piecewise affine Since the Bellman functions {V t } t=0,1,...,t are convex l.s.c. and proper, we deduce that V t,k V t,k+1 V t = V t Ṽ t,k+1 Ṽ t,k Thus, we can control the evolution of the SDDP algorithm

19 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution Conclusion

20 Main result Two couplings Φ and Ψ, and an inf-operation with kernel K X Φ X fx inf y Y Kx, y gy K K Φ +Ψ f Φ x inf y Y K Φ +Ψ x, y g Ψ y Y Ψ Y

21 Background on couplings and Fenchel-Moreau conjugacy Let be given two sets X primal and X dual Consider a coupling function Φ : X X R We also use the notation X Φ X for a coupling, so that X Φ X Φ : X X R Definition The Φ-Fenchel-Moreau conjugate of a function f : X R, with respect to the coupling Φ, is the function f Φ : X R defined by f Φ x = sup Φx, x f x, x x X + X

22 Examples of couplings in generalized convexity Vector space X with algebraic dual X linear forms and the bilinear coupling Φx, x = x, x X = X = {r 1,..., r n R n r 1 > 0,..., r n > 0} and Φ x 1,..., x n, x 1,..., x n = min{x 1 x 1,..., x nxn} X = X is equipped with a distance d and Φx, x = αdx, x

23 A present for Kazuo Murota Vector space R n coupled with R n R n by localization of the bilinear coupling, w.r.t. neighbours N x Z n Φ x, x, x = x, x δ N x x The local convex extension of f : R n R { f x = x, x + α sup x X,α R appears to be a partial conjugate f x = sup x X and we deduce the relation x, x + α f x, x N x { x, x f Φ x, x sup f x = f ΦΦ x f x x N 1 x } }

24 We introduce a sum coupling Let be given two primal sets X, Y and two dual sets X, Y, together with two coupling functions Φ : X X R, Ψ : Y Y R We define the sum coupling Φ + Ψ coupling the primal product set X Y with the dual product set X Y by Φ + Ψ : X Y X Y R, x, y, x, y Φx, x + Ψy, y

25 A kernel K, two couplings Φ and Ψ and a new kernel K Φ +Ψ X Φ X K K Φ +Ψ Y Ψ Y

26 We introduce the conjugate of a kernel bivariate function w.r.t. a sum coupling With any kernel bivariate function K : X Y R, defined on the primal product set X Y, we associate the conjugate, with respect to the coupling Φ Ψ, + defined on the dual product set X Y, by K Φ +Ψ x, y = sup x X,y Y Φx, x + Ψy, y + Kx, y x, y X Y

27 Main result: Fenchel-Moreau conjugation inequalities with three couplings Theorem For any bivariate function K : X Y R and univariate functions f : X R and g : Y R, all defined on the primal sets, we have that f x inf Kx, y gy, x X y Y f Φ x inf K Φ +Ψ x, y g Ψ y, x X y Y

28 Main result Two couplings Φ and Ψ, and an inf-operation with kernel K f x inf Kx, y gy y Y f Φ x }{{} Φ Fenchel- Moreau conjugate of f inf K Φ +Ψ x, y y Y }{{} Φ Moreau +Ψ Fenchelconjugate of K g Ψ y }{{} Ψ Fenchel- Moreau conjugate of g

29 Main result Two couplings Φ and Ψ, and an inf-operation with kernel K f x inf Kx, y gy y Y f Φ x }{{} Φ Fenchel- Moreau conjugate of f inf K Φ +Ψ x, y y Y }{{} Φ Moreau +Ψ Fenchelconjugate of K g Ψ y }{{} Ψ Fenchel- Moreau conjugate of g The left hand side assumption is a primal inequality, which is rather weak upper bound for an infimum whereas the right hand side conclusion is a dual inequality, which is rather strong lower bound for an infimum

30 Main result second formulation Three couplings Φ, Ψ and K f g K }{{} K Fenchel- Moreau conjugate of g 0 f Φ }{{} Φ Fenchel- Moreau conjugate of f + g Ψ KΦ +Ψ }{{} K Φ +Ψ Fenchel- Moreau conjugate of g Ψ 0

31 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution Conclusion

32 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel conjugates of Bellman functions Application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution The duality equality case Design of dual problems Fenchel-Moreau conjugate of generalized inf-convolution Conclusion

33 The duality equality case The duality equality case is the property that f x = inf Kx, y gy, x X y Y f Φ x = inf K Φ +Ψ x, y g Ψ y, x X y Y

34 Sufficient conditions for the duality equality case Corollary Consider any bivariate function K : X Y R and univariate functions f : X R and g : Y R, all defined on the primal sets. The equality case holds true when 1. g Ψ Ψ = g 2. the following function has a saddle point or no duality gap x, y, y X Y Y Φx, x Kx, y + + Ψy, y g Ψ y 3. the two coupling functions Φ : X X R and Ψ : Y Y R, and the kernel K : X Y R all take finite values

35 Sufficient conditions for the duality equality case Corollary Consider any bivariate function K : X Y R and univariate functions f : X R and g : Y R, all defined on the primal sets. We define K x y = K, y Φ x = inf x X The equality case holds true when Kx y + gy = inf y Y sup y Y Φx, x Kx, y K Ψ x y g Ψ y

36 Sufficient conditions for the duality equality case Corollary Consider any bivariate function K : X Y R and univariate functions f : X R and g : Y ], + ], all defined on the primal sets. The equality case holds true when 1. the coupling Ψ : Y Y R is the duality bilinear form, between Y and its algebraic dual Y 2. the function g is a proper convex function the function g never takes the value and is not identically equal to +, 3. for any x X, the function K x is a proper convex function 4. for any x X, the function g is continuous at some point where K x is finite [Rockafellar, 1966]

37 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel conjugates of Bellman functions Application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution The duality equality case Design of dual problems Fenchel-Moreau conjugate of generalized inf-convolution Conclusion

38 Perturbation + Fenchel-Moreau duality To design a dual problem to the original problem hy gy inf y Y take a bivariate function K : X Y R, where X is a perturbation set, such that K0, y = hy then define f x = inf y Y Kx, y gy, x X then take two couplings Φ and Ψ, and obtain f Φ x inf K Φ +Ψ x, y g Ψ y, x X y Y and finally obtain the dual problem inf hy gy = f 0 y Y f ΦΦ 0 sup y Y K Φ +Ψ, y Φ 0 g Ψ y } {{ } dual problem

39 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel conjugates of Bellman functions Application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution The duality equality case Design of dual problems Fenchel-Moreau conjugate of generalized inf-convolution Conclusion

40 Fenchel conjugate of inf-convolution The classic inf-convolution satisfies g1 g 2 x = inf y 1 +y 2 =x g1 g 2 = g 1 + g 2 g 1 y 1 g 2 y 2

41 Definition of generalized inf-convolution Definition Let be given three sets X, Y 1 and Y 2 For any trivariate convoluting function I : Y 1 X Y 2 R, we define the I-inf-convolution of two functions g 1 : Y 1 R and g 2 : Y 2 R by I g1 g2 x = inf g 1 y 1 Iy 1, x, y 2 y 1 Y 1,y 2 Y 2 } {{ } convoluting function g 2 y 2

42 Definition of generalized inf-convolution Definition Let be given three sets X, Y 1 and Y 2 For any trivariate convoluting function I : Y 1 X Y 2 R, we define the I-inf-convolution of two functions g 1 : Y 1 R and g 2 : Y 2 R by I g1 g2 x = inf g 1 y 1 Iy 1, x, y 2 y 1 Y 1,y 2 Y 2 } {{ } convoluting function g 2 y 2 The classic inf-convolution corresponds to Iy 1, x, y 2 = δ x y 1 + y 2 : g1 g 2 x = infy1 +y 2 =x g 1 y 1 g 2 y 2

43 Fenchel-Moreau conjugate of generalized inf-convolution Proposition Let be given three primal sets X, Y 1, Y 2 and three dual sets X, Y 1, Y 2, together with three coupling functions X Φ X Ψ, Y 1 1 Y 1, Y Ψ 2 2 Y 2 For any univariate functions f : X R, g 1 : Y 1 R and g 2 : Y 2 R, all defined on the primal sets, we have that f x g 1 I g2 x, x X f Φ x g Ψ I 1 1 g Ψ 2 2 x, x X, where the convoluting function I on the dual sets is given by I = I Φ +Ψ1 +Ψ 2

44 The I-inf-convolution is minus the Fenchel-Moreau conjugate of a sum Proposition The I-inf-convolution is given by g 1 I g2 = g 1 g 2 I

45 Fenchel-Moreau conjugate of generalized inf-convolution Proposition If there exist two coupling functions Γ 1 : X Y 1 R, Γ 2 : X Y 2 R, such that the partial Φ-Fenchel-Moreau conjugate of the convoluting function I splits as Iy 1,, y 2 Φ x = Γ 1 x, y 1 + Γ 2 x, y 2, then the Φ-Fenchel-Moreau conjugate of the inf-convolution g 1 I g2 is given by a sum as I Φ g1 g2 = g Γ g Γ 2 2

46 Fenchel-Moreau conjugate of generalized inf-convolution Proposition If there exist two coupling functions Γ 1 : X Y 1 R, Γ 2 : X Y 2 R, such that the partial Φ-Fenchel-Moreau conjugate of the convoluting function I splits as Iy 1,, y 2 Φ x = Γ 1 x, y 1 + Γ 2 x, y 2, then the Φ-Fenchel-Moreau conjugate of the inf-convolution g 1 I g2 is given by a sum as I Φ g1 g2 = g Γ g Γ 2 2 This generalizes g 1 g 2 = g 1 + g 2

47 Outline of the presentation Fenchel conjugates of Bellman functions and application to SDDP Fenchel-Moreau conjugation inequality with three couplings Complements: equality case, Fenchel s duality theorem, design of dual problems, inf-convolution Conclusion

48 Conclusion We have proven a new Fenchel-Moreau conjugation inequality with three couplings and given sufficient conditions for the equality case We have provided a general method to design dual problems by means of one kernel and two couplings We have introduced a generalized inf-convolution, and have provided formulas for Fenchel-Moreau conjugates We have shown that Fenchel conjugates of Bellman functions satisfy a Bellman like inequation, and we have sketched an application to the SDDP algorithm

49 Research agenda Look for couplings Φ adapted to the structure of the original optimization problem For example, are there couplings Φ for which the l0 pseudo-norm 0 has a nice Φ-Fenchel-Moreau conjugate Φ 0? What can be said of the conjugacy with respect to the Bregman distance coupling Φx, y = Hx Hy Hx, x y? Investigate possible couplings adapted to discrete convex analysis Investigate the algebra behind Fenchel-Moreau conjugacies and dualities Make connections with probabilistic notions [Akian, Quadrat, and Viot, 1998]

50 The algebra behind conjugacies and dualities The set R equipped with the operations = sup and = + is the max-plus algebra R max = R, sup, + Fenchel-Moreau conjugacies are morphisms between modules f Φ x = sup Φx, x f x x X + = Φx, x f x 1 = Φf 1 x }{{} x X matrix Fenchel-Moreau conjugacies are special dualities, that is, mappings D : R X R X such that, for any family f i i I of functions f i R X, Dinf f i = sup Df i i I i I Dualities are also residuated mappings, functional Galois connections

51 We define a new idempotent integral An idempotent measure M on Ω is a mapping M : 2 Ω R, sup, + with the property that M i I A i = sup i I MA i, for any family {A i } i I of subsets A i of Ω For example, MA = sup ω A Mω, where M : Ω R Define a new idempotent integral with respect to the idempotent measure M : 2 X R R, sup, by + f dm = M {x, r f x < t}, f R X }{{} hypograph This area below the curve integral has the property that inf f i dm = sup f i dm i I i I

52 Dualities are also idempotent conditional integrals Let be given two sets X primal and X dual and an idempotent measure M on X R X M : 2 X R X R, sup, + The following idempotent conditional integrals are dualities Df x = f dm x = M {x, r f x < t} {x }, f R X and dualities are the idempotent conditional integrals Fenchel-Moreau conjugacies are the special dualities for which Mdxdx dr = coupling {}} Φ { Φx, x }{{} idempotent density dxdx }{{} uniform idempotent measure dr 1 }{{} uniform idempotent measure

53 Thank you:- X Φ X fx inf y Y Kx, y gy K K Φ +Ψ f Φ x inf y Y K Φ +Ψ x, y g Ψ y Y Ψ Y

54 M. Akian, J.-P. Quadrat, and M. Viot. Duality between probability and optimization. In J. Gunawardena, editor, Idempotency. Cambridge University Press, Teemu Pennanen and Ari-Pekka Perkkiö. Shadow price of information in discrete time stochastic optimization. Mathematical Programming, 1681: , Mar R. T. Rockafellar. Extension of Fenchel s duality theorem for convex functions. Duke Math. J., 33:81 89, ISSN R. T. Rockafellar and P. R. Wolenski. Envelope representations in hamilton-jacobi theory for fully convex problems of control. In Proceedings of the 40th IEEE Conference on Decision and Control Cat. No.01CH37228, volume 3, pages , 2001.