Risk-Averse Dynamic Optimization. Andrzej Ruszczyński. Research supported by the NSF award CMMI

Transcription

1 Research supported by the NSF award CMMI

2 Outline 1 Risk-Averse Preferences 2 Coherent Risk Measures 3 Dynamic Risk Measurement 4 Markov Risk Measures 5 Risk-Averse Control Problems 6 Solution Methods

3 Uncertain Outcomes and Risk Why Probabilistic Models? Wealth of results of probability theory Connection to real data via statistics Universal language (engineering, economics, medicine,... ) Probability space (Ω, F, P) Decision space X Random outcome (e.g., cost) Z x (ω), Z : X Ω R Expected Value Model min x E[Z x ] = Ω Z x (ω) P(dω) It optimizes the outcome on average (Law of Large Numbers?) What is Risk? Existence of unlikely and undesirable outcomes - high Z x (ω) for some ω

4 Classical Utility Models Expected Utility Models (von Neumann and Morgenstern, 1944) ( E [u(z x)] = u ( Z x (ω) ) ) dp(ω) min x X u : R R is a nondecreasing disutility function Rank Dependent Utility (Distortion) Models (Quiggin, 1982; Yaari, 1987) min x X 1 0 F 1 Z x (p) dw(p) Ω F 1 Z x ( ) - quantile function w : [0, 1] R is a nondecreasing rank dependent utility function Existence of utility functions is derived from systems of axioms, but in practice they are difficult to elicit

5 Mean Risk Models Two Objectives Minimize the expected outcome, the mean E[Z x ] Minimize a scalar measure of uncertainty of Z x, the risk r[z x ] r 1 [Z ] = Var[Z ] r 2 [Z ] = ( E[(Z EZ ) s + ]) 1/s [ ( p )] r 3 [Z ] = min E max η Z, η 1 p (Z η) (Markowitz model) (semideviation) (deviation from quantile) Mean Risk Optimization min ρ(z x) = E[Z x ] + κ r[z x ], 0 κ κ max x X Interesting application of parametric optimization r[z x ] is nonlinear w.r.t. probability and possibly nonconvex in x

6 Coherent Risk Measures Space of uncertain outcomes Z = L p (Ω, F, P), p [1, ] A functional ρ : Z R is a coherent risk measure if it satisfies the following axioms Convexity: ρ(λz + (1 λ)w ) λρ(z ) + (1 λ)ρ(w ) λ (0, 1), Z, W Z Monotonicity: If Z W then ρ(z ) ρ(w ), Translation Equivariance: ρ(z + a) = ρ(z ) + a, Z, W Z Positive Homogeneity: ρ(τz) = τρ(z ), Z Z, τ 0 Z Z, a R Kijima-Ohnishi (1993) no monotonicity Artzner-Delbaen-Eber-Heath (1999 ) - space L R.-Shapiro (2005) spaces L p,.

7 Coherent Risk Measures Space of uncertain outcomes Z = L p (Ω, F, P), p [1, ] A functional ρ : Z R is a coherent risk measure if it satisfies the following axioms Convexity: ρ(λz + (1 λ)w ) λρ(z ) + (1 λ)ρ(w ) λ (0, 1), Z, W Z Monotonicity: If Z W then ρ(z ) ρ(w ), Translation Equivariance: ρ(z + a) = ρ(z ) + a, Z, W Z Positive Homogeneity: ρ(τz) = τρ(z ), Z Z, τ 0 Z Z, a R Kijima-Ohnishi (1993) no monotonicity Artzner-Delbaen-Eber-Heath (1999 ) - space L R.-Shapiro (2005) spaces L p,.

8 Coherent Risk Measures Space of uncertain outcomes Z = L p (Ω, F, P), p [1, ] A functional ρ : Z R is a coherent risk measure if it satisfies the following axioms Convexity: ρ(λz + (1 λ)w ) λρ(z ) + (1 λ)ρ(w ) λ (0, 1), Z, W Z Monotonicity: If Z W then ρ(z ) ρ(w ), Translation Equivariance: ρ(z + a) = ρ(z ) + a, Z, W Z Positive Homogeneity: ρ(τz) = τρ(z ), Z Z, τ 0 Z Z, a R Kijima-Ohnishi (1993) no monotonicity Artzner-Delbaen-Eber-Heath (1999 ) - space L R.-Shapiro (2005) spaces L p,.

9 Coherent Risk Measures Space of uncertain outcomes Z = L p (Ω, F, P), p [1, ] A functional ρ : Z R is a coherent risk measure if it satisfies the following axioms Convexity: ρ(λz + (1 λ)w ) λρ(z ) + (1 λ)ρ(w ) λ (0, 1), Z, W Z Monotonicity: If Z W then ρ(z ) ρ(w ), Translation Equivariance: ρ(z + a) = ρ(z ) + a, Z, W Z Positive Homogeneity: ρ(τz) = τρ(z ), Z Z, τ 0 Z Z, a R Kijima-Ohnishi (1993) no monotonicity Artzner-Delbaen-Eber-Heath (1999 ) - space L R.-Shapiro (2005) spaces L p,.

10 Coherent Risk Measures Space of uncertain outcomes Z = L p (Ω, F, P), p [1, ] A functional ρ : Z R is a coherent risk measure if it satisfies the following axioms Convexity: ρ(λz + (1 λ)w ) λρ(z ) + (1 λ)ρ(w ) λ (0, 1), Z, W Z Monotonicity: If Z W then ρ(z ) ρ(w ), Translation Equivariance: ρ(z + a) = ρ(z ) + a, Z, W Z Positive Homogeneity: ρ(τz) = τρ(z ), Z Z, τ 0 Z Z, a R Kijima-Ohnishi (1993) no monotonicity Artzner-Delbaen-Eber-Heath (1999 ) - space L R.-Shapiro (2005) spaces L p,.

11 Coherent Risk Measures Space of uncertain outcomes Z = L p (Ω, F, P), p [1, ] A functional ρ : Z R is a coherent risk measure if it satisfies the following axioms Convexity: ρ(λz + (1 λ)w ) λρ(z ) + (1 λ)ρ(w ) λ (0, 1), Z, W Z Monotonicity: If Z W then ρ(z ) ρ(w ), Translation Equivariance: ρ(z + a) = ρ(z ) + a, Z, W Z Positive Homogeneity: ρ(τz) = τρ(z ), Z Z, τ 0 Z Z, a R Kijima-Ohnishi (1993) no monotonicity Artzner-Delbaen-Eber-Heath (1999 ) - space L R.-Shapiro (2005) spaces L p,.

12 Conjugate Duality of Risk Measures Pairing of a linear topological space Z with a linear topological space Y of regular signed measures on Ω with the bilinear form µ, Z = Eµ [Z ] = Z (ω) µ(dω) We assume standard conditions on pairing and the polarity: (Z + ) = Y Dual Representation Theorem If ρ : Z R is a lower semicontinuous coherent risk measure, then ρ(z ) = sup µ, Z, Z Z µ A with a convex A P (set of probability measures in Y). Delbaen (2001), Föllmer Schied (2002), R. Shapiro (2005), Rockafellar Uryasev Zabarankin (2006),... Lower semicontinuity is automatic if ρ is finite and Z is a Banach lattice Ω

13 Optimization of Risk Measures Minimize over x X a random outcome Z x (ω) = f (x, ω), ω Ω Composite Optimization Problem Theorem (R. Shapiro, 2005) min ρ(z x) (P) x X Let x Z x (ω) be convex and ρ( ) be coherent. Suppose that ˆx X is an optimal solution of (P) and ρ( ) is continuous at Z ˆx. Then there exists a probability measure ˆµ ρ(z ˆx ) A such that ˆx solves min x X E ˆµ[Z x ] = min x X We also have the duality relation: min x X max E µ[z x ] µ A ρ(z x) = max µ A inf x X E µ[z x ]

14 How to Measure Risk of Sequences? Probability space (Ω, F, P) with filtration F 1 F T F Adapted sequence of random variables (costs) Z 1, Z 2,..., Z T Spaces: Z t = L p (Ω, F t, P), p [1, ], and Z t,t = Z t Z T Conditional Risk Measure A mapping ρ t,t : Z t,t Z t satisfying the monotonicity condition: ρ t,t (Z ) ρ t,t (W ) for all Z, W Z t,t such that Z W Dynamic Risk Measure A sequence of conditional risk measures ρ t,t : Z t,t Z t, t = 1,..., T ρ 1,T (Z 1, Z 2, Z 3,..., Z T ) Z 1 = R ρ 2,T (Z 2, Z 3,..., Z T ) Z 2 ρ 3,T (Z 3,..., Z T ) Z 3.

15 Evaluating Risk on a Scenario Tree t = 1 t = 2 t = 3 t = 4 Z 1 4 Z 1 Z 2 1 Z 2 2 Z Z 3 1 Z 3 2 Z Z Z 2 4 Z 3 4 Z 4 4 Z 5 4 Z 6 4 Z 7 4 Z 8 4

16 Evaluating Risk on a Scenario Tree t = 1 t = 2 t = 3 t = 4 ρ 1,4 (Z 1, Z 2, Z 3, Z 4 ) Z 1 Z 2 1 Z 2 2 Z Z 3 1 Z 3 2 Z Z Z Z Z 6 Z 7 Z Z 4 2 Z Z

17 Evaluating Risk on a Scenario Tree t = 1 t = 2 t = 3 t = 4 Z 1 4 ρ 2,4 1 (Z 2, Z ρ 2 3, Z 4 ) 2,4 (Z 2, Z 3, Z 4 ) Z 2 1 Z 2 2 Z Z 3 1 Z 3 2 Z Z Z 2 4 Z 3 4 Z 4 4 Z 5 4 Z 6 4 Z 7 4 Z 8 4

18 Time Consistency of Dynamic Risk Measures A dynamic risk measure { } T ρ t,t is time-consistent if for all τ < θ t=1 Z k = W k, k = τ,..., θ 1 and ρ θ,t (Z θ,..., Z T ) ρ θ,t (W θ,..., W T ) imply that ρ τ,t (Z τ,..., Z T ) ρ τ,t (W τ,..., W T ) Define ρ τ,θ (Z τ,..., Z θ ) = ρ τ,t (Z τ,..., Z θ, 0,..., 0), 1 τ θ T Risk-Averse Equivalence Theorem Suppose { } T ρ t,t satisfies the conditions: t=1 ρ t,t (Z t, Z t+1,..., Z T ) = Z t + ρ t,t (0, Z t+1,..., Z T ) ρ t,t (0,..., 0) = 0 Then it is time-consistent if and only if for all τ θ: ( ) ( ρ τ,t Zτ,..., Z θ,..., Z T = ρτ,θ Zτ,..., Z θ 1, ρ θ,t (Z θ,..., Z T ) )

19 Collapsing Subtrees by Conditional Risk Measures t = 1 t = 2 t = 3 t = 4 Z 1 4 Z2 1 Z2 2 Z 1 Z 2 Z 3 Z 4 Z Z 2 4 Z 3 4 Z 4 4 Z 1 Z 5 4 Z 6 4 Z 7 4 Z 8 4

20 Collapsing Subtrees by Conditional Risk Measures t = 1 t = 2 t = 3 t = 4 Z 1 4 ρ 2,4 1 (Z 2, Z Z2 1 ρ 2 3, Z 4 ) Z2 2 2,4 (Z 2, Z 3, Z 4 ) Z 1 Z 2 Z 3 Z 4 Z Z 2 4 Z 3 4 Z 4 4 Z 1 Z 5 4 Z 6 4 Z 7 4 Z 8 4

21 Recursive Structure of Dynamic Risk Measures Define one-step conditional risk measures ρ t : Z t+1 Z t : ρ t (Z t+1 ) = ρ t,t (0, Z t+1, 0,..., 0) Nested Decomposition Theorem Suppose a dynamic risk measure { } T ρ t,t is time-consistent and t=1 ρ t,t (Z t, Z t+1,..., Z T ) = Z t + ρ t,t (0, Z t+1,..., Z T ) ρ t,t (0,..., 0) = 0 Then for all t we have the representation ρ t,t (Z t,..., Z T ) = ( ( = Z t + ρ t Z t+1 + ρ t+1 (Z t ρ T 2 ZT 1 + ρ T 1 (Z T ) ) ) )

22 Coherent One-Step Conditional Risk Measures Stronger assumptions about one-step measures ρ t : Z t+1 Z t : Convexity: ρ t (λz + (1 λ)w ) λρ t (Z ) + (1 λ)ρ t (W ) λ (0, 1), Z, W Z t+1 Monotonicity: If Z W then ρ t (Z ) ρ t (W ), Z, W Z t+1 Predictable Translation Equivariance: ρ t (Z + W ) = Z + ρ t (W ), Z Z t, W Z t+1 Positive Homogeneity: ρ t (τz) = τρ t (Z ), Z Z t+1, τ 0 Scandolo ( 03), Riedel ( 04), R.-Shapiro ( 06), Cheridito-Delbaen-Kupper ( 06), Föllmer-Penner ( 06), Artzner-Delbaen-Eber-Heath-Ku ( 07), Pflug-Römisch ( 07) Example: Conditional Mean Semideviation [ (Zt+1 ρ t (Z t+1 ) = E[Z t+1 F t ] + κe E[Z t+1 F t ] ) s ] 1 s Ft Here s [1, p] and κ [0, 1] may be F t -measurable +

23 Coherent One-Step Conditional Risk Measures Stronger assumptions about one-step measures ρ t : Z t+1 Z t : Convexity: ρ t (λz + (1 λ)w ) λρ t (Z ) + (1 λ)ρ t (W ) λ (0, 1), Z, W Z t+1 Monotonicity: If Z W then ρ t (Z ) ρ t (W ), Z, W Z t+1 Predictable Translation Equivariance: ρ t (Z + W ) = Z + ρ t (W ), Z Z t, W Z t+1 Positive Homogeneity: ρ t (τz) = τρ t (Z ), Z Z t+1, τ 0 Scandolo ( 03), Riedel ( 04), R.-Shapiro ( 06), Cheridito-Delbaen-Kupper ( 06), Föllmer-Penner ( 06), Artzner-Delbaen-Eber-Heath-Ku ( 07), Pflug-Römisch ( 07) Example: Conditional Mean Semideviation [ (Zt+1 ρ t (Z t+1 ) = E[Z t+1 F t ] + κe E[Z t+1 F t ] ) s ] 1 s Ft Here s [1, p] and κ [0, 1] may be F t -measurable +

24 Coherent One-Step Conditional Risk Measures Stronger assumptions about one-step measures ρ t : Z t+1 Z t : Convexity: ρ t (λz + (1 λ)w ) λρ t (Z ) + (1 λ)ρ t (W ) λ (0, 1), Z, W Z t+1 Monotonicity: If Z W then ρ t (Z ) ρ t (W ), Z, W Z t+1 Predictable Translation Equivariance: ρ t (Z + W ) = Z + ρ t (W ), Z Z t, W Z t+1 Positive Homogeneity: ρ t (τz) = τρ t (Z ), Z Z t+1, τ 0 Scandolo ( 03), Riedel ( 04), R.-Shapiro ( 06), Cheridito-Delbaen-Kupper ( 06), Föllmer-Penner ( 06), Artzner-Delbaen-Eber-Heath-Ku ( 07), Pflug-Römisch ( 07) Example: Conditional Mean Semideviation [ (Zt+1 ρ t (Z t+1 ) = E[Z t+1 F t ] + κe E[Z t+1 F t ] ) s ] 1 s Ft Here s [1, p] and κ [0, 1] may be F t -measurable +

25 Coherent One-Step Conditional Risk Measures Stronger assumptions about one-step measures ρ t : Z t+1 Z t : Convexity: ρ t (λz + (1 λ)w ) λρ t (Z ) + (1 λ)ρ t (W ) λ (0, 1), Z, W Z t+1 Monotonicity: If Z W then ρ t (Z ) ρ t (W ), Z, W Z t+1 Predictable Translation Equivariance: ρ t (Z + W ) = Z + ρ t (W ), Z Z t, W Z t+1 Positive Homogeneity: ρ t (τz) = τρ t (Z ), Z Z t+1, τ 0 Scandolo ( 03), Riedel ( 04), R.-Shapiro ( 06), Cheridito-Delbaen-Kupper ( 06), Föllmer-Penner ( 06), Artzner-Delbaen-Eber-Heath-Ku ( 07), Pflug-Römisch ( 07) Example: Conditional Mean Semideviation [ (Zt+1 ρ t (Z t+1 ) = E[Z t+1 F t ] + κe E[Z t+1 F t ] ) s ] 1 s Ft Here s [1, p] and κ [0, 1] may be F t -measurable +

26 Coherent One-Step Conditional Risk Measures Stronger assumptions about one-step measures ρ t : Z t+1 Z t : Convexity: ρ t (λz + (1 λ)w ) λρ t (Z ) + (1 λ)ρ t (W ) λ (0, 1), Z, W Z t+1 Monotonicity: If Z W then ρ t (Z ) ρ t (W ), Z, W Z t+1 Predictable Translation Equivariance: ρ t (Z + W ) = Z + ρ t (W ), Z Z t, W Z t+1 Positive Homogeneity: ρ t (τz) = τρ t (Z ), Z Z t+1, τ 0 Scandolo ( 03), Riedel ( 04), R.-Shapiro ( 06), Cheridito-Delbaen-Kupper ( 06), Föllmer-Penner ( 06), Artzner-Delbaen-Eber-Heath-Ku ( 07), Pflug-Römisch ( 07) Example: Conditional Mean Semideviation [ (Zt+1 ρ t (Z t+1 ) = E[Z t+1 F t ] + κe E[Z t+1 F t ] ) s ] 1 s Ft Here s [1, p] and κ [0, 1] may be F t -measurable +

27 Coherent One-Step Conditional Risk Measures Stronger assumptions about one-step measures ρ t : Z t+1 Z t : Convexity: ρ t (λz + (1 λ)w ) λρ t (Z ) + (1 λ)ρ t (W ) λ (0, 1), Z, W Z t+1 Monotonicity: If Z W then ρ t (Z ) ρ t (W ), Z, W Z t+1 Predictable Translation Equivariance: ρ t (Z + W ) = Z + ρ t (W ), Z Z t, W Z t+1 Positive Homogeneity: ρ t (τz) = τρ t (Z ), Z Z t+1, τ 0 Scandolo ( 03), Riedel ( 04), R.-Shapiro ( 06), Cheridito-Delbaen-Kupper ( 06), Föllmer-Penner ( 06), Artzner-Delbaen-Eber-Heath-Ku ( 07), Pflug-Römisch ( 07) Example: Conditional Mean Semideviation [ (Zt+1 ρ t (Z t+1 ) = E[Z t+1 F t ] + κe E[Z t+1 F t ] ) s ] 1 s Ft Here s [1, p] and κ [0, 1] may be F t -measurable +

28 Multistage Risk-Averse Optimization Problems Probability Space: (Ω, F, P) with filtration F 1 F T F Decision Variables: x t (ω), ω Ω, t = 1,..., T Nonanticipativity: Each x t is F t -measurable Cost per Stage: Z t (x t ) with realizations Z t (x t (ω), ω), ω Ω Objective Function: Time-consistent dynamic measure of risk Interchangeability Principle ( {Z 1 (x 1 ) + ρ 1 (Z 2 (x 2 ) + ρ 2 Z 3 (x 3 ) +... min x 1,x 2 ( ),...,x T ( ) ( + ρ T 2 ZT 1 (x T1 + ρ T 1 (Z T (x T )) ) ) )} [ [ ( = min {Z 1 (x 1 ) + ρ 1 min (Z 2 (x 2 ) + ρ 2 min Z 3 (x 3 ) +... x 1 x 2 x 3 + ρ T 2 [min x T 1 ( ZT 1 (x T1 ) + ρ T 1 (min x T Z T (x T )) ) ] )] )]}

29 Interchangeability on a Scenario Tree t = 1 t = 2 t = 3 t = 4 Z 1 4 Z 1 Z 2 1 Z 2 2 Z Z 3 1 Z 3 2 Z Z Z 2 4 Z 3 4 Z 4 4 Z 5 4 Z 6 4 Z 7 4 Z 8 4

30 Interchangeability on a Scenario Tree min ρ 1,4(Z 1, Z 2, Z 3, Z 4 ) x 1,x 2 ( ),x 3 ( ),x 4 ( ) t = 1 t = 2 t = 3 t = 4 Z 1 4 Z 1 Z 2 1 Z 2 2 Z Z 3 1 Z 3 2 Z Z Z 2 4 Z 3 4 Z 4 4 Z 5 4 Z 6 4 Z 7 4 Z 8 4

31 Interchangeability on a Scenario Tree t = 1 t = 2 t = 3 t = 4 min x 1 4 min x 1 ( Z1 + ρ 1 ( ) ) ( min Z ρ 2 ( ) ) x 2 1 min( ) min( ) min( ) min( ) min ( ) x x x x x ( ) min( ) min( ) min( ) min( ) ( ) min( ) min x x 4 x 5 x 4 2 x x min x 6 4 ( min Z ρ 2 ( ) ) x 2 2 ( )

32 Controlled Markov Models State space X (Polish with Borel σ -algebra) Control space U (Polish with Borel σ -algebra) Feasible control sets U t : X U, t = 1, 2,... Controlled transition kernels Q t : graph(u t ) P, t = 1, 2,... P - set of probability measures on X Cost functions c t : graph(u t ) R, t = 1, 2,... State history X t (up to time t = 1, 2,... ) Policy π t : X t U, t = 1, 2,... (always with values in U t (x t )) Markov policy π t : X U, t = 1, 2,... (stationary if π t = π 1 for all t) x t u t = π t (x t ) (x t, u t ) x t+1 Q t (x t, u t )

33 Two Basic Risk-Neutral Control Problems Finite horizon expected cost problem: min π 1,...,π T [ T ] E c t (x t, u t ) + c T +1 (x T +1 ) t=1 with controls u t = π t (x 1,..., x t ) Infinite horizon discounted expected cost problem: [ ] E α t 1 c t (x t, u t ) min π 1,π 2,... t=1 Both problems have optimal solutions in form of Markov policies Optimal policies can be found by dynamic programming equations Our Intention Introduce risk aversion to both problems by replacing the expected value by dynamic risk measures

34 Using Dynamic Risk Measures for Markov Decision Processes Controlled Markov process x t, t = 1,..., T, T + 1 Policy Π = {π 1, π 2,..., π T } defines u t = π t (x t ) Cost sequence c t (x t, u t ), t = 1,..., T, and c T +1 (x T +1 ) Dynamic time-consistent risk measure ( J(Π) = c 1 (x 1, u 1 ) + ρ 1 c 2 (x 2, u 2 ) + ρ 2 (c 3 (x 3, u 3 ) + ( + ρ T 1 ct (x T, u T ) + ρ T (c T +1 (x T +1 )) ) ) ) Difficulty Risk-averse optimal control problem min J(Π) Π The value of ρ t ( ) is F t -measurable and is allowed to depend on the entire history of the process. We cannot expect a Markov optimal policy if our attitude to risk depends on the whole past

35 New Construction of a Conditional Risk Measure Consider functions of the state, in fixed space V = L p (X, B, P 0 ) Additional argument: density on (X, B, P 0 )) in the set { } 1 M = m L q (X, B, P 0 ) : m(x) P 0 (dx) = 1, m 0, p + 1 q = 1 X Risk Transition Mapping Associated with a Kernel Q : graph(u) M A measurable functional σ : V X M R satisfying (i) For every x X the functional v σ ( v, x, Q(x, u(x)) ) is a coherent measure of risk on V (ii) For every v V the function x σ ( v, x, Q(x, u(x)) ) is in V Dual Representation If σ (, x, m) is lsc, then there exist convex sets A(x, m) such that σ (v, x, m) = sup v, µ µ A(x,m)

36 Markov Risk Measures Assumption: The controlled kernels Q t have values in the set M (with densities with respect to P 0 ) A one-step conditional risk measure ρ t : Z t+1 Z t is a Markov risk measure with respect to the controlled Markov process {x t }, if there exists a risk transition mapping σ t : V X M R such that for all v V and for all measurable u t U t (x t ) we have ρ t (v(x t+1 )) = σ t ( v, xt, Q t (x t, u t ) ) Duality: ρ t (v(x t+1 )) = sup v, µ µ A t (x t,q t (x t,u t )) A t (x t, Q t (x t, u t )) controlled multikernel In the risk neutral setting, when ρ t (v(x t+1 )) = E[v(x t+1 ) F t ] we have a single-valued controlled kernel A t (x t, Q t (x t, u t )) = {Q t (x t, u t )}. Risk-averse preferences Ambiguity in the transition kernel

37 Markov Risk Evaluation t 1 t t + 1 states a b c d

38 Markov Risk Evaluation t 1 t t + 1 states a b c x t x t 1 x t+1 d Q t 1 Q t

39 Markov Risk Evaluation t 1 t t + 1 states a b c d v t 1 (a) v t (a) v t+1 (a) v t 1 (b) v t (b) v t+1 (b) v t 1 (c) v t (c) v t+1 (c) v t 1 (d) v t (d) v t+1 (d) σ t 1 (v t, x t 1, Q t 1 ) σ t (v t+1, x t, Q t )

40 Finite Horizon Risk-Averse Control Problem Consider a controlled Markov process {x t } with u t = π t (x 1,..., x t ). Risk-averse optimal control problem: ( min c 1(x 1, u 1 ) + ρ 1 (c 2 (x 2, u 2 ) + ρ 2 c 3 (x 3, u 3 ) + Π ( + ρ T 1 ct (x T, u T ) + ρ T (c T +1 (x T +1 )) ) ) ) Theorem If the conditional measures ρ t are Markov (+ technical conditions), then the optimal solution is given by the dynamic programming equations: v T +1 (x) = c T +1 (x), x X { ( v t (x) = min c t (x, u) + σ t vt+1, x, Q t (x, u) ) }, t = T,..., 1 u U t (x) Optimal Markov policy ˆΠ = { ˆπ 1,..., ˆπ T } - the minimizers above

41 Finite Horizon Risk-Averse Control Problem Consider a controlled Markov process {x t } with u t = π t (x 1,..., x t ). Risk-averse optimal control problem: ( min c 1(x 1, u 1 ) + ρ 1 (c 2 (x 2, u 2 ) + ρ 2 c 3 (x 3, u 3 ) + Π ( + ρ T 1 ct (x T, u T ) + ρ T (c T +1 (x T +1 )) ) ) ) Theorem If the conditional measures ρ t are Markov (+ technical conditions), then the optimal solution is given by the dynamic programming equations: v T +1 (x) = c T +1 (x), { v t (x) = min c t (x, u) + sup u U t (x) µ A t (x,q t (x,u)) x X } v t+1, µ, t = T,..., 1 Optimal Markov policy ˆΠ = { ˆπ 1,..., ˆπ T } - the minimizers above

42 Discounted Risk Measures for Infinite Sequences {F t } - filtration on (Ω, F) Z t, t = 1, adapted sequence of random variables Z t = L p (Ω, F t, P), Z = Z 1 Z 2 ρ t : Z t+1 Z t - conditional risk mappings Fix the discount factor α (0, 1). For T = 1, 2,... define ρ α 1,T (Z 1, Z 2,..., Z T ) = ρ 1,T (Z 1, αz 2,..., α T 1 Z T ) = Z 1 + ρ 1 ( αz 2 + ρ 2 ( α 2 Z ρ T 1 (α T 1 Z T ) )) Discounted Risk Measure ϱ α (Z ) = lim T ρ α 1,T (Z 1, Z 2,..., Z T ) It is well defined, convex, monotone, and positively homogeneous, whenever max essup Z t (ω) < t

43 Discounted Infinite Horizon Problem We consider a controlled stationary Markov process {x t }, t = 1, 2,... with a discounted measure of risk (0 < α < 1): min J(Π, x 1) = ϱ α( c(x 1, u 1 ), c(x 2, u 2 ), ) Π ( ( = c(x 1, u 1 ) + ρ 1 αc(x 2, u 2 ) + ρ 2 α 2 c(x 3, u 3 ) + )) Conditional Markov risk measures ρ t are stationary, if they share the same risk transition mapping σ : X V M R Theorem If the conditional measures ρ t are Markov and stationary, then the optimal value function ˆv(x) satisfies the dynamic programming equation: v(x) = min u U(x) { c(x, u) + α σ ( v, x, Q(x, u) )}, x X Optimal stationary Markov policy ˆΠ = { ˆπ, ˆπ,... } - the minimizer above

44 Discounted Infinite Horizon Problem We consider a controlled stationary Markov process {x t }, t = 1, 2,... with a discounted measure of risk (0 < α < 1): min J(Π, x 1) = ϱ α( c(x 1, u 1 ), c(x 2, u 2 ), ) Π ( ( = c(x 1, u 1 ) + ρ 1 αc(x 2, u 2 ) + ρ 2 α 2 c(x 3, u 3 ) + )) Conditional Markov risk measures ρ t are stationary, if they share the same risk transition mapping σ : X V M R Theorem If the conditional measures ρ t are Markov and stationary, then the optimal value function ˆv(x) satisfies the dynamic programming equation: v(x) = min u U(x) { c(x, u) + α sup v, µ }, µ A(x,Q(x,u)) x X Optimal stationary Markov policy ˆΠ = { ˆπ, ˆπ,... } - the minimizer above

45 Solution Methods Value iteration v k+1 (x) = min u U(x) { c(x, u) + α σ ( v k, x, Q(x, u) )}, x X, k = 1, 2,... Policy iteration For k = 0, 1, 2,..., given a stationary Markov policy {π k, π k,... }, find the value function v k by solving (by a specialized Newton method) the nonsmooth equation v(x) = c(x, π k (x)) + ασ ( v, x, Q(x, π k (x)) ), x X Find the next policy π k+1 ( ) by one-step optimization { ( π k+1 (x) = argmin c(x, u) + ασ v k, x, Q(x, u) )}, x X u U(x)

46 Example: Service and Insurance 5 states - new, good, bad, broken uninsured, broken insured 5 controls - nothing, insure, service, service and insure, replace As the cost of insurance is greater than the expected benefit from it, no risk-neutral model can suggest insurance Results for the mean-semideviation model Value Iteration steps Policy Iteration - 2 steps (with 3 and 2 Newton steps) Optimal Policy new - service and insure good - service bad - do nothing broken uninsured and broken insured - replace