Multistage stochastic programs: Time-consistency, time-inconsistency and martingale bounds

Transcription

1 Multistage stochastic programs: Time-consistency, time-inconsistency and martingale bounds Georg Ch. Pflug, joint work with Raimund Kovacevic and Alois Pichler April 2016

2 The time-consistency problem Many concrete multistage stochastic optimization problems we typically consider (hydrostorage and electricity production management, thermal plant optimization, asset-liability management, insurance management) contain risk (utility) functionals in their objective or constraints. If we plan for a sequence of decision times, do we have to reconsider the optimal decisions at later stages, when more information comes in?

3 Conditional utility functionals We consider a probability space (Ω,F,P). Let F 1 be a σ-field contained in F. A mapping U( F 1 ) : L p (F) L p (F 1 ) is called conditional utility mapping (with observable information F 1 ) if the following conditions are satisfied for all Y, λ [0,1]: (CA1) U(Y +Y 1 F 1 ) = U(Y F 1 )+Y 1, if Y 1 F 1 (predictable translation-equivariance), (CA2) U(λY +(1 λ)ỹ F 1 ) λu(y F 1 )+(1 λ)u(ỹ F 1 ) (concavity), (CA) Y Ỹ implies U(Y F 1) U(Ỹ F) (monotonicity). Let F 0 = (Ω, ) be the trivial σ-algebra. Then U( F 0 ) is an unconditional utility functional. Risk is just negative utility.

4 The extension of the classical Fenchel-Moreau Theorem to L p -valued concave functionals leads to a representation of the form U(Y F 1 ) = inf{e(y Z F 1 ) U + F 1 (Z) : Z S F1 }, If U is positively homogeneous (CA4) U(λY F 1 ) = λu(y F 1 )) for λ > 0 then U(Y F 1 ) = inf{e(y Z F 1 ) : Z S F1 }. The relevant Z s are called supergradients and S F1 is called the supergradient set.

5 Conditional functionals have arguments Concavity of Y U P (Y F 1 ) Convexity of P U P (Y F 1 ) Monotonicity of F 1 U P (Y F 1 )

6 Example: The entropic functional Primal form Dual form U(Y) = 1 γ loge[exp( γy)]. U(Y) = inf{e(y Z)+ 1 E(Z logz) : E(Z) = 1,Z 0}. γ Conditional form Dual conditional form U(Y F 1 ) = 1 γ loge[exp( γy) F 1]. U(Y F 1 ) = inf{e(y Z F 1 )+ 1 γ E(Z logz F 1) : E(Z F 1 ) = 1,Z 0}

7 Example: The average value-at-risk Primal form. α (Y) = 1 α Dual form α 0 G 1 Y (p) dp AV@R 0 (Y) = ess inf(y);av@r 1 (Y) = E(Y). AV@R α (Y) = inf{e(y Z) : E(Z) = 1,0 Z 1/α}. Conditional form AV@R α (Y F 1 ) = sup{x 1 α E([Y X] ) : X F 1 }. Dual conditional form AV@R α (Y F 1 ) = inf{e(y Z F 1 ) : E(Z F 1 ) = 1,0 Z 1/α}. Other names for this functional: conditional value-at-risk (Rockefellar and Uryasev (2002)), expected shortfall (Acerbi and Tasche (2002)) and tail value-at-risk (Artzner et al. (1999)). The name average value-at-risk is due to Föllmer and Schied (2004).

8 Time consistency for final processes Let U( F 1 ) : L p (Ω,F,P) L p (Ω,F 1,P) be a conditional utility-type mapping and let U( ) be its unconditional counterpart. Definition. (see also Artzner at al. 2007). The functional U( F 1 ) is called time consistent, if for all Y,Ỹ L p (Ω,F,P) the implication holds. U(Y F 1 ) U(Ỹ F 1)a.s. = U(Y) U(Ỹ)

9 is not time-consistent. Y Ỹ AV@R 0.1 (Y F 1 ) = (4;0) (;0) = AV@R 0.1 (Ỹ F 1) while AV@R 0.1 (Y) = 0.9 < 1.8 = AV@R 0.1 (Ỹ).

10 Kusuoka representation. Theorem. (Kusuoka) Any law invariant, positively homogeneous acceptability functional U on L has the representation U (Y) = inf µ M 1 0 AV@R α (Y)µ(dα), (1) where M is a set of probability measures on [0,1].

11 Positively homogeneous utility functionals are typically not time consistent Theorem. Suppose that the positively homogeneous functional U has a Kusuoka representation If U(Y) = inf{ for some ǫ > 0 and 1 0 AV@R α (Y)dµ(α) : µ M}. inf{µ([ǫ,1 ǫ]) : µ M} > 0 sup{µ([0,γ]) : µ M} 0 for γ 0, then U is not time-consistent as such, but has to be randomly decomposed for ensuring time-consistency (see later). The only exceptions are the expectation the essential infimum the essential supremum

12 Definition. A functional U( F 1 ) is called acceptance consistent, if for all Y L p (Ω,F,µ) the implication essinfu(y F 1 ) U(Y) holds. It is called rejection consistent, if (adapted from Weber, 2006). esssupu(y F 1 ) U(Y).

13 Definition. The functional U( F 1 ) is called (i) compound convex, if for all Y domu U(Y) E(U(Y F 1 )). (ii) compound concave, if for all Y domu U(Y) E(U(Y F 1 )).

14 Theorem. (i) compound convexity implies rejection consistency. (ii) compound concavity implies acceptance consistency. Remark. Let F = (F 0,...,F T ) be a filtration. (i) U( ) is compound convex, iff Y t = U(Y F t ) is a submartingale. (ii) U( ) is compound concave, iff Y t = U(Y F t ) is a supermartingale.

15 The is compound convex, hence rejection consistent. is not acceptance consistent. 1/ 1/ 1/2 1/ 1/ 1/2 1/ 1/ AV@R 2/ (Y F 1 ) = 0, AV@R 2/ (Y) = 2.

16 Definition. (Artzner (2008), Kupper (2008), Jobert (2000)) A pair of functionals U 0 and U( F 1 ) is called recursive, if for all Y L p (Ω,F,µ) the equation U 0 (Y) = U 0 (U 1 (Y F 1 )) holds. Of special interest are version-independent conditional functionals, which are auto-recursive (i.e. for which U 0 ( ) = U 1 ( F 0 )). Examples. EC-functionals (i.e. functionals of the form E[U(Y F 1 )]) are recursive. The entropic functional is auto-recursive. The AV@R is not auto-recursive.

17 The relation between time consistency and recursivity Theorem. (Artzner et. al., 2007) A pair U 0 ( ), U 1 ( F 1 ) with translation equivariant U 1 ( F 1 ), the centered-at-zero property U 1 (0 F 1 ) = 0 and monotonic U 0 ( ) is time consistent if and only if it is recursive.

18 Enforcing time consistency (recursivity) by composition Let, for each t = 1,...,T, conditional acceptability mappings U t 1 := U( F t 1 ) be given. Introduce a multi-period probability functional U by compositions of the conditional acceptability mappings U t 1, t = 1,...,T, namely, U(Y;F) := U 0 [Y 1 + +U T 2 [Y T 1 +U T 1 (Y T )] ] T = U 0 U 1 U T 1 ( Y t ) for every Y t Y t. (Ruszczynski and Shapiro, 2006). Notice that these functionals are recursive in a trivial way. t=1

19 The nested Example. Consider the conditional Average Value-at-Risk (of level α (0, 1]) as conditional acceptability mapping U t 1 (Y t ) := AV@R α ( F t 1 ) for every t = 1,...,T. Then the multi-period probability functional nav@r α (Y;F)=AV@R α ( F 0 ) AV@R α ( F T 1 )( T t=1 Y t) is a concave multiperiod acceptability functional. It is called the nested Average Value-at-Risk.

20 The entropic functional The nested entropic acceptability functional is U 0 U 1 U T 1 (Y) with U t (Y) = 1 γ loge[exp( γy) F t], for Y nonnegative and nonvanishing. The nested entropic functional collapses to the unconditional entropic functional.

21 The nested Example. The nested has the following dual representation: α (Y;F) = inf{e[(y 1 + +Y T )M T ] : 0 M t 1 α M t 1, E(M t F t 1 ) = M t 1,M 0 = 1,t = 1,...,T}. The nested average value-at-risk nav@r is given by a linear stochastic optimization problem containing functional constraints.

22 Time consistent decisions Let a stochastic multistage decision problem be given, for simplicity assume that it is defined on the basis of a stochastic tree. Its solution is called time-consistent, if the solutions of every conditional problem (that is when the process is conditioned to a given node at some stage t) can be chosen among the solutions of the original problem (when the decisions at times 1,...,t 1 are kept fixed). Proposition. If the objective is a nested acceptability functional (and no other constraints are present), then the decision problem leads to time-consistent decisions.

23 Non-nested functionals make the trouble Time inconsistency appears in a natural way in optimality problems. We want to find max{e(y) : AV@R 0.05 (Y) 2} or maxe(y)+av@r 0.05 (Y) double line = optimal decision

24 The conditional problem given the first node:

25 Nested functionals make the trouble Time consistency contradicts information monotonicity F 0 F (1) 1 F F 0 F (2) In both examples, the final income Y is the same, but in the right example, the filtration is finer. One calculates AV@R 0.1 [AV@R 0.1 (Y F (1) 1 )] = 0.9 > 0 = AV@R 0.1[AV@R 0.1 (Y F (2) 1 )]. Notice that E[AV@R 0.1 (Y F (1) 1 )] = E[AV@R 0.1(Y F (2) 1 )] = 0.9. F

26 Information monotonicity of pos. homogeneous compositions Proposition. Let U (Y;F) be a composition of information monotone, positively homogeneous acceptability mappings U t with supergradient sets S t ( ). The composition is information monotone if and only if the following nesting condition holds: if F F. S t 1 (F ) S t (F) S t (F )

27 A composition of conditional positively homogeneous functionals U 0 U t 1 U t U t+1 U T is information monotone only if U 0 = = U t 1 = E = AV@R 1 A good example is U t+1 = U T = essinf = AV@R 0 U(Y 1,...,Y T ) = T w t E[AV@R αt (Y t F t 1 )] t=2 AV@R 0 and AVaR 1 are also the only functionals, which are time-consistent.

28 Example: The nested entropic functional Nested entropic functionals are defined as compositions U 0 U 1 U T 1, where U t (Y F t ) = 1 γ t loge[exp( γ t Y) F t ], and γ t 0 for Y Y (Now γ may depend on t!). The correct dual pairing for the entropic functional is given by the Zygmund spaces L exp and Llog + L. Nested entropic functionals are information monotone, iff γ 0 γ 1 γ T 1.

29 Changing functionals as more information gets available may ensure time consistency Decomposing the final using random level s Let α F t be a random variable with values in [0,1]. Define the AV@R with random level α as AV@R α (Y F t ) = inf{e(yz F t ) : E(Y F t ) = 1,0 Z;αZ 1}. It has an alternate characterization for α > 0 by AV@R α (Y F t ) = sup{q 1 α E([Q Y] + F t ) : Q F t }. The AV@R with random level obeys all properties like the usual AV@R, i.e. translation-equivariance, concavity, monotonicity, and positive homogeneity. Moreover, α AV@R α is convex.

30 Illustration: Artzner s Example Y Z Z 1 = E(Z F 1 ) AV@R 2 1 = 5 1 AV@R 2 = 1 1 AV@R 2 1 = 20 The total AV@R2 is 1, while AV@R2(Y F 1 ) 1.

31 Theorem. Nested decomposition of the Let Y L 1 (F T ), F t F τ F T. 1. For α [0,1] the Average Value-at-Risk obeys the decomposition AV@R α (Y) = infe[z t AV@R α Zt (Y F t )], (2) where the infimum is among all densities Z t F t with 0 Z t, αz t 1l and EZ t = 1. For α > 0 the infimum in (2) is attained. 2. Moreover if Z is the optimal dual density for the AV@R, that is AV@R α (Y) = EYZ with Z 0, αz 1l and EZ = 1, then Z t = E[Z F t ] is the best choice in (2).. The conditional Average Value-at-Risk at random level α F t (0 α 1l) has the recursive (nested) representation AV@R α (Y F t ) = infe[z τ AV@R α Zτ (Y F τ ) F t ], () where the infimum is among all densities Z τ F τ with 0 Z τ, αz τ 1l and E[Z τ F t ] = 1l.

32 The simple special cases α = 0 α = 1 essinf Y = essinf (essinf (Y F 1 )) E(Y) = E[E(Y F 1 )]

33 Illustration Y Z Z 1 = E(Z F 1 ) AV@R 2 1 = 5 1 AV@R 2 = 1 1 AV@R 2 1 = 20 The total AV@R is AV@R α (Y) = E[Z 1 AV@R αz1 (Y F 1 )] = 1, while AV@R2(Y F 1 ) 1. Notice that for t < τ AV@R α (Y F t ) E[AV@R α (Y F τ ) F t )] E(Y F t )

34 A typical multistage decision problem Let H(x 0,ξ 1,...,x T 1,ξ T ) be some (concave in x) profit function depending on the random scenario process ξ = (ξ 1,...,ξ T ) and the decisions x = (x 0,...,x T 1 ) The multistage decision problem is maximize EH(x,ξ)+γ AV@R[H(x,ξ)] s.t. x F x X, (4) where H(x,ξ) is a short notation for H(x 0,ξ 1,...,x T 1,ξ T ).

35 As typical for Markov decision processes, we define the value function V t (x 0:t 1,α,γ) := esssup xt:t E[H(x 0:T ) F t ]+γ AV@R α (H(x 0:T ) F t ). The value function depends on the decisions up to time t 1, x 0:t 1, where x t:t is chosen such that (x 0:T ) = (x 0:t 1, x t,t ) X, the random model parameters α F t and γ F t and the current status of the system due to the filtration F t. Evaluated at initial time t = 0 and assuming the sigma-algebra F 0 trivial the value function relates to the initial problem as supeh(x 0:T )+γ AV@R α (H(x 0:T )) = x 0:T = esssup x0:t E[H(x 0:T ) F 0 ]+γ AV@R α (H(x 0:T ) F 0 ) = V 0 ([],α,γ).

36 Theorem. Dynamic Programming Principle. Assume that H is upper semi-continuous with respect to x and ξ valued in some convex, compact subset of R n. 1. The value function evaluates to V T (x 0:T 1,α,γ) = (1+γ)esssup xt H(x 0:T ) at terminal time T. 2. For any t < τ, (t,τ T) the recursive relation V t (x 0:t 1,α,γ) = esssup xt:τ 1 essinf Zt:τ E[V τ (x 0:τ 1,α Z t:τ,γ Z t:τ ) F t ], where Z t:τ F τ, 0 Z t:τ, αz t:τ 1l and E[Z t:τ F t ] = 1l, holds true.

37 Definition. Let α [0,1] be a fixed level. 1. Z = (Z t ) t T is a feasible (for the nonanticipativity constraints) process of densities if 1.1 Z t is a martingale with respect to the filtration F t and Z t, αz t 1l and EZ t = 1 (t T). 2. The intermediate densities are Z t:τ := Zτ Z t 1 (0 < t < τ) and Z 0:τ := Z τ.

38 Verification Theorem. Let x and Z be feasible for the multistage decision problem. 1. Suppose that W satisfies W T (x 0:T 1,αZ 0:T,γZ 0:T ) (1+γZ 0:T )H(x 0:T (ξ 0:T )) and W t (x 0:t 1,αZ 0:t,γZ 0:t ) esssup xt E[W t+1 (x 0:t,α Z 0:t+1,γ Z 0:t+1 ) F t ], then the process W t (x 0:t 1,αZ 0:t,γZ 0:t ) (t T) is a super-martingale dominating V(x 0:t 1,αZ 0:t,γZ 0:t ), i.e. V W. 2. Let Y satisfy Y T (x 0:T 1,αZ 0:T,γZ 0:T ) (1+γZ 0:T )H(x 0:T (ξ 0:T )) and Y t (x 0:t 1,αZ 0:t,γZ 0:t ) essinf Zt+1 E[Y t+1 (x 0:t,α Z 0:t+1,γ Z 0:t+1 ) F t ], then the process Y t (x 0:t 1,αZ 0:t,γZ 0:t ) is a sub-martingale dominated by V (x 0:t 1,αZ 0:t,γZ 0:t ), i.e. Y V.

39 The Algorithm Step 0 Let x0:t 0 be any feasible, initial solution of the problem (4). Set k 0. Set Y(x 0 0:T ) = EH(x0 0:T )+γav@r α(h(x 0 0:T )) Step 1 Find Z k, such that 0 Z k 1 α, EZk = 1 and define ) Zt k := E (Z k F t. (5) A good initial choice is often Z k satisfying ( ) ( EZ k H x0:t k = AV@R α H ( x k 0:T )). (6) Step 2 (check for local improvement). Choose [ ( argmax xt F t E H x k+1 t +γz k t AV@R αz k t (H x k 0:T ( x k 0:T ) Ft ] ) Ft ) (7) (8) at any arbitrary stage t and a node specified by F t.

40 Step (Verification). Accept x k+1 Y ( ) x0:t k EH x k+1 0:T ( 0:t if x k+1 0:T x k+1 0:T )+γav@r α ( H ( x k+1 0:T x k+1 0:T )), else try another feasible Z k (for example Z k 1 ( 2 1l+Z k ), Z k (1+α)1l αz k or Z k = 1l B (P(B) α)) and repeat Step 2. If no direction Z k can be found providing an improvement, then x 0:T is already optimal. Set ( ) ( ( ( )) Y := EH )+γav@r α H, (9) increase k k +1 and continue with Step 1 until ( ) ( ) Y Y x0:t k < ε, x k+1 0:T where ε > 0 is the desired improvement in each cycle k.

41 The martingale citerion Let Z be the familily of feasible processes (Z t ), i.e. they are martingales w.r.t. F t and 0 αz t 1l, E(Z t ) = 1 for all t. Theorem. If (x) and (Z) are optimal, then M t (x,z) := V t (x 0:t 1,αZ o,t,γz 0,t ) is a martingale w.r.t. F t. Conversely, if M t (x,z) is a martingale and the argmax sets (for x) and the argmin sets (for Z) are nonempty, then x and Z are optimal.

42 A numerical example: The flowergirl problem The flower girl problem is the multistage extension of the single stage newsboy problem and an example for multistage inventory control The flowergirl may buy in the morning of each day t a certain amount x t of flowers. If the random demand ξ t is not met, a penalty has to be paid; if there are left-overs, the flowers may be sold the next day, but some proportion fades and cannot be sold any more. The problem is to find the optimal ordering policy, even for non-markovian demand processes. The overall profit is given by a function H(ξ,x (ξ)) = H(x 0,ξ 1,x 1 (ξ 1 ),ξ 2,x 2 (ξ 1,ξ 2 ),...,ξ T )

43 The flower-girl uses the objective EH(ξ,x α (H(ξ,x (ξ))) for a suitable choice of α and γ and considers the problem maximize EH(x)+γ AV@R α (H(x)) subject to x F, x 0. (10)

44 The time complexity of the decomposed algorithm stages leaves straight forward , ,90 1,20 5,21 26,157 Our algorithm ,20 4,697 Run-times (CPU seconds) to solve the flowergirl problem for exemplary tree processes ξ with stages and leaves as indicated. (Our algorithm in its sequential, non-parallel implementation)

45 Extension for general distortion functionals Decomposition Theorem. Let U h (Y) = 1 distortion functional. 1. U h obeys the decomposition 0 G 1 Y (u)h(u) du be a U h (Y) = infe [ Z U Z (Y F t ) ], (11) where the infimum is among all feasible, positive random variables Z F t satisfying EZ = 1 and h(u) SSD Z for U Uniform[0,1]. 2. Let F t F τ. The utility functional obeys the nested decomposition [ U (Y F t ) = essinf E Z τ U Zτ (Y F τ ) F t ], the essential infimum being among all feasible random variables Z τ F τ.

46 Y Z E(Z F) Z 0.7 Z 0.7 Z 0.4 Z 0.4 R Z (Y F) = 5 AV@R AV@R 0.4 = 0 AV@R AV@R = 21 1 AV@R AV@R 0 Nested decomposition of R = 5 UAV@R 0.7(Y)+ 2 5 UAV@R 0.4(Y). We get R(Y) = E[Z R Z (Y F t )] = = 5.74

47 Conclusions Compositions of risk functionals are time consistent (but not interpretable) and not information monotone Final risk functionals are typically information monotone but not time consistent Exceptions are only the expectation and the (essential) infimum resp. supremum When using time-inconsistent functionals one one has to decide: either to accept time-inconsistent decisions in a rolling horizon setup or to accept path dependent utility functionals, i.e. decision criteria which depend on the actual path the scenario process takes.

48 References Georg Ch. Pflug, Alois Pichler. Multistage Stocahstic Optimization. Springer, 2014 Raimund M. Kovacevic, Georg Ch. Pflug. Are time consistent valuations information-monotone? International Journal of Theoretical and Applied Finance, 17(1), Georg Ch. Pflug, Alois Pichler. Time-inconsistent multistage stochastic programs: Martingale bounds. European J of OR 249(1), , Georg Ch. Pflug, Alois Pichler. Time-consistent decisions and temporal decomposition of coherent risk functionals. to appear in: Mathematics of OR 2016.