Risk-Averse Control of Partially Observable Markov Systems. Andrzej Ruszczyński. Workshop on Dynamic Multivariate Programming Vienna, March 2018
|
|
- Melvin Pierce
- 5 years ago
- Views:
Transcription
1 Risk-Averse Control of Partially Observable Markov Systems Workshop on Dynamic Multivariate Programming Vienna, March 2018
2 Partially Observable Discrete-Time Models Markov Process: fx t ; Y t g td1;:::;t on the Borel state space X Y The process fx t g is observable, while fy t g is not observable Control sets: U t W X U, t D 1; : : : ; T Transition kernel: PŒ.X tc1 ; Y tc1 / 2 C j x t ; y t ; u t D Q t.x t ; y t ; u t /.C/ Costs: Z t D c t.x t ; Y t ; U t /, t D 1; : : : ; T Two relevant filtrations X ;Y F t defined by the full state process fxt ; Y t g defined by the observed process fxt g F X t n Space of costs: Z t D Classical Problem: Z W! R ˇˇ o Z is F X ;Y t -measurable and bounded min E c 1.X 1 ; Y 1 ; U 1 / C c 2.X 2 ; Y 2 ; U 2 / C C c T.X T ; Y T ; U T /
3 Partially Observable Discrete-Time Models Markov Process: fx t ; Y t g td1;:::;t on the Borel state space X Y The process fx t g is observable, while fy t g is not observable Control sets: U t W X U, t D 1; : : : ; T Transition kernel: PŒ.X tc1 ; Y tc1 / 2 C j x t ; y t ; u t D Q t.x t ; y t ; u t /.C/ Costs: Z t D c t.x t ; Y t ; U t /, t D 1; : : : ; T Two relevant filtrations X ;Y F t defined by the full state process fxt ; Y t g defined by the observed process fxt g F X t n Space of costs: Z t D Risk-Averse Problem: Z W! R ˇˇ o Z is F X ;Y t -measurable and bounded min 1;T c1.x 1 ; Y 1 ; U 1 /; c 2.X 2 ; Y 2 ; U 2 /; : : : ; c T.X T ; Y T ; U T /
4 Dynamic Risk Measures Probability space. ; F ; P/ with filtration F 1 F T F Adapted sequence of random variables (costs) Z 1 ; Z 2 ; : : : ; Z T Spaces: Z t of F t -measurable functions and Z t;t D Z t Z T Dynamic Risk Measure A sequence of conditional risk measures t;t W Z t;t! Z t, t D 1; : : : ; T. Monotonicity condition: t;t.z/ t;t.w / for all Z; W 2 Z t;t such that Z W Local property: For all A 2 F t t;t.1 A Z/ D 1 A t;t.z/
5 Time Consistency and Nested Representation A dynamic risk measure t;t T td1 is time-consistent if for all 1 t < T Z t D W t and tc1;t.z tc1 ; : : : ; Z T / tc1;t.w tc1 ; : : : ; W T / imply that ;T.Z ; : : : ; Z T / ;T.W ; : : : ; W T / Define one-step mappings: t.z tc1 / D t;t.0; Z tc1 ; 0; : : : ; 0/ Nested Decomposition Theorem Suppose a dynamic risk measure t;t T td1 is time-consistent, and t;t.0; : : : ; 0/ D 0 t;t.z t ; Z tc1 ; : : : ; Z T / D Z t C t;t.0; Z tc1 ; : : : ; Z T / Then for all t we have the representation t;t.z t ; : : : ; Z T / D Z t C t Z tc1 C tc1 Z tc2 C C T 1.Z T /
6 Issues with General Theory in the Markov Setting Probability measure P, processes X t and Z t depend on policy We have to deal with a family of risk measures t;t./ The values of the risk measures may depend on history, and Markov policies cannot be expected The cost may not be observable Motivating Example X D f0; 1g, T D 2, and Z t D Z t.x t / (cost depends on state). Consider the risk measure 2;2.Z 2 /.x 1 ; x 2 / D Z 2.x 2 / 1;2.Z 1 ; Z 2 /.x 1 / D Z 1.x 1 / C Z 2.x 1 / (assumes that x 1 will not change) It is time-consistent and has the normalization, translation, and local properties. As 1;2 does not depend on the distribution of x 2, given x 1, it is useless for controlling Markov models. In fact, it is much worse than expectation.
7 Conditional Risk Evaluators Space of observable random variables: n t D S W! R ˇˇ o S is F X t -measurable and bounded ; t D 1; : : : ; T A mapping t;t W Z t Z T! t is a conditional risk evaluator (i) It is monotonic if Z s W s for all s D t; : : : ; T, implies that t;t.z t ; : : : ; Z T / t;t.w t ; : : : ; W T / (ii) It is normalized if t;t.0; : : : ; 0/ D 0; (iii) It is translation equivariant if 8.Z t ; : : : ; Z T / 2 t Z tc1 Z T, t;t.z t ; : : : ; Z T / D Z t C t;t.0; Z tc1 ; : : : ; Z T /; (iv) It is decomposable if a mapping t W Z t! t exists such that: t.z t / D Z t ; 8Z t 2 t ; t;t.z t ; : : : ; Z T / D t.z t / C t;t.0; Z tc1 ; : : : ; Z T /; 8 Z 2 Z t;t
8 Risk Filters and their Time Consistency A risk filter t;t td1;:::;t t;t W Z t;t! t. is a sequence of conditional risk evaluators We have index risk filters by policy, because affects the measure P History: H t D.X 1 ; X 2 ; : : : ; X t /, h t D.x 1 ; x 2 ; : : : ; x t / 2 td1;:::;t A family of risk filters t;t is stochastically conditionally time consistent if for any ; 0 2, for any 1 t < T, for all h t 2 X t, all.z t ; : : : ; Z T / 2 Z t;t and all.w t ; : : : ; W T / 2 Z t;t, the conditions tc1;t.z tc1; : : : ; Z T / j H t Z t D W t D h t st 0 tc1;t.w tc1; : : : ; W T / j Ht 0 D h t imply t;t.z t; Z tc1 ; : : : ; Z T /.h t / 0 t;t.w t; W tc1 ; : : : ; W T /.h t / The relation st is the conditional stochastic order
9 Bayes Operator Belief State: Conditional distribution of Y t given initial distribution 1 and history g t D. 1 ; x 1 ; u 1 ; x 2 ; : : : ; u t 1 ; x t / t.g t /.A/ D PŒY t 2 A j g t ; 8 A 2 B.Y/; t D 1; : : : ; T Conditional distribution of the observable part: Z P ŒX tc1 2 B j g t ; u t D Q X t.x t ; ; u t /.B/ d t.g t /; where Q X t.x t; y t ; u t / is the marginal of Q t.x t ; y t ; u t / on the space X Transition of the belief state - Bayes operator Y tc1.g tc1 / D t.x t ; t.g t /; u t ; x tc1 / Example: Y D fy 1 ; : : : ; y n g and Q t.x; y; u/ has density q t.x 0 ; y 0 jx; y; u/ t.x; ; u; x 0 / P n.fy k id1 g/ D q t.x 0 ; y k j x; y i ; u/ i P ǹ P n D1 id1 q t.x 0 ; y ` j x; y i ; u/ i
10 Markov Risk Filters Extended state history (including belief states): h t D.x 1 ; 1 ; x 2 ; 2 ; : : : ; x t ; t / 2 H t Policies D. 1 ; : : : ; T / with decision rules t.h t / 2 U t.x t / Markov Policy For all h t ; h 0 t 2 H t, if x t D x 0 t and t D 0 t, then t.h t / D t.h 0 t / D t.x t ; t / Policy value function: v t.h t/ D t;t c t.x t ; Y t ; t.h t //; : : : ; c T.X T ; Y T ; T.H T //.h t / 2 td1;:::;t A family of risk filters t;t is Markov if for all Markov policies 2, for all h t D.x 1 ; : : : ; x t / and h 0 t D.x 1 0 ; : : : ; x t 0/ in Xt such that x t D xt 0 and t D t 0, we have v t.h t/ D v t.h0 t / D v t.x t; t /
11 Structure of Markov Risk Filters (with Jingnan Fan) 2 A family of risk filters t;t is normalized, translation-invariant, td1;:::;t stochastically conditionally time consistent, decomposable, and Markov if and only if transition risk mappings exist: t W x t ; t ; Q t.h t/ W 2 ; h t 2 X t V! R; t D 1 : : : T 1; (i) t.x; ; ; / is normalized and strongly monotonic with respect to stochastic dominance (ii) for all 2, for all t D 1; : : : ; T 1, and for all h t 2 X t, v t.h t/ D r t.x t ; t ; t.h t // C t x t ; t ; Q t.h t/; v tc1.h t; / Evaluation of a Markov policy : v t.x t; t / D r t.x t ; t ; t.x t ; t // C t x t ; t ; Q t.x t; t /; x 0 7! v tc1.x 0 ; t.x t ; t ; t.x t ; t /; x 0 //
12 Examples of Transition Risk Mappings Average Value at Risk.x; ; m; v/ D min 2R where.x; / 2 Œ min ; max.0; 1. n C 1 Z.x; / X v.x 0 / o C m.dx 0 / Mean Semideviation of Order p Z Z.x; ; m; v/ D v.x 0 / m.dx 0 / C.x; / X ƒ X E m Œv where.x; / 2 Œ0; 1. v.x 0 / E m Œv 1 p C m.dx 0 p / Entropic Mapping.x; ; m; v/ D 1.x; / ln E m he.x;/ v.x0 / i ;.x; / > 0
13 Dynamic Programming Risk-averse optimal control problem: c1.x 1 ; Y 1 ; U 1 /; c 2.X 2 ; Y 2 ; U 2 /; : : : ; c T.X T ; Y T ; U T / Theorem min 1;T If the risk measure is Markovian (+ general conditions), then the optimal solution is given by the dynamic programming equations: v t vt.x; / D min.x; / D min Z t x; ; u2u t.x/ Y r T.x; ; u/; x 2 X; 2 P.X/ r t.x; ; u/ C u2u T.x/ K X t.x; y; u/.dy/; x 0 7! v tc1 x 0 ; t.x; ; u; x 0 / ; x 2 X; 2 P.Y/; t D T 1; : : : ; 1 Optimal Markov policy Ŏ D f O 1 ; : : : ; O T g - the minimizers above
14 Risk-Averse Clinical Trials (Darinka Dentcheva and Curtis McGinity) In stages t D 1; : : : ; T successive patients are given drugs (cytotoxic agents), to which severe toxic response (even death) is possible Probability of toxic response (x tc1 D 1) depends on the unknown optimal dose and the administered dose (control) u t : 1 F.u t ; / D 1 e '.u t;/ The belief state t, the conditional probability distribution of the unknown optimal dose, is the current state of the system The state evolves according to Bayes operator, depending on the response of the patient: for 2 Y (the range of doses) ( F.u t ; / t./ if toxic (x tc1 D 1) tc1./ 1 F.u t ; / t./ if not toxic (x tc1 D 0) Cost per stage: c t.; u t / D t ju t j (other forms possible) Medical ethics naturally motivates risk-averse control
15 Total Cost Models Find the best policy D. 1 ; : : : ; T / to determine doses u t D t. t / Expected Value Model TX C1 min E t ju t 2 td1 j T C1 is the weight of the final recommendation u T C1 Risk-Averse Model Two sources of risk n min 2 1;T C1 t ju t jo td1;:::;t C1 Unknown state (only belief state t available at time t) Unknown evolution of f t g due to random responses of patients
16 Dynamic Programming Equations All memory is carried by the belief state t For each t and u t, only two next states are possible, corresponding to x tc1 D 0 or 1 Simplified equation Z v t./ D min r t.; u/ C ; PŒx 0 D 1jy; u.dy/; v u tc1 t.x; ; u; / Y Examples: r t.; u/ D E ju j ; p; './ D E max '.x 0 / x 0 2f0;1g Any law invariant risk measure on the space of functions on U (for r t ) or on U f0; 1g (in the case of t ) can be used here.
17 Limited Lookahead Policies At each time t, assume that this is the last test before the final recommendation, and solve the two-stage problem Risk-Neutral h min E t t ju t j C x tc1 E response min E tc1 ju tc1 u t u tc1 Risk-Averse h min E t t ju t j C x tc1 max min E tc1 ju tc1 u t response u tc1 i j i j x tc1 D TX C1 DtC1 (weight of the future)
18 Simulation Results for Expected Value and Risk-Averse Policies Comparison of Policies: Risk-Averse vs. Expected Value Lookahead 0.4 Distribution of Dosage 0.3 T = 20 η = 2.6 Risk-AverseLookahead ExpectedValueLookahead η T+1 * 0 AndrzejMcGinity, Ruszczyński Dentcheva, Ruszczyński Risk-Averse Risk-Averse ControlOptimal of Partially Learning Observable for ClinicalMarkov Trial DoseSystems Escalation
19 Machine Deterioration (with Jingnan Fan) We consider the problem of minimizing costs of a machine in T periods. Unobserved state: y t 2 f1; 2g, with 1 being the good and 2 the bad state Observed state: x t - cost incurred in the previous period Control: u t 2 f0; 1g, with 0 meaning continue, and 1 meaning replace The dynamics of Y is Markovian, with the transition matrices K Œu : K Œ0 1 p p D K Œ1 1 p p D p p Distribution of costs: P x tc1 C ˇˇ yt D i; u t D 0 D P x tc1 C ˇˇ yt D i; u t D 1 D Z C 1 Z C 1 f i.x/ dx; i D 1; 2 f 1.x/ dx; i D 1; 2
20 Value and Policy Monotonicity Belief state: i 2 Œ0; 1 - conditional probability of the good state The optimal value functions: vt.x; / D x C w t./, t D 1; : : : ; T C 1 Dynamic programming equations w t./ D min nr C f 1 ; x 0 7! x 0 C w tc1.1 p/ I with the final stage value wt C1./ D 0. f 1 C.1 /f 2 ; x 0 7! x 0 C w tc1..; x 0 // o ; If f 1 f2 is non-increasing, then the functions wt./ are non-increasing and thresholds t 2 Œ0; 1 ; t D 1; : : : ; T exist, such that the policy ( ut D 0 if t > t ; 1 if t t ; is optimal
21 Numerical Illustration Cost distributions f 1 and f 2 : uniform with R 0 f 1.x/ dx R 0 f 2.x/ dx Transition risk mapping: mean semideviation Empirical distribution of the total cost for the risk-neutral model (blue) and the risk-averse model (orange)
22 Partially Observable Jump Process The unobserved process fy t g 0tT : Finite state Markov jump process on the space Y D f1; : : : ; ng with the generator.t/: 8 < lim ij.t/ D "#0 : 1 " P Y tc" D jˇˇyt D i if j i P k i ik.t/ if j D i The observed process fx t g 0tT : Diffusion following the SDE dx t D A.Y t ; t/ dt C B.t/ dw t ; 0 t T ; with the initial value x 0, independent of Y 0. fw t g is a Wiener process. Random final cost:.y T /
23 The Belief State Equation (Wonham, 1964) Filtration of observable events: fft X g 0tT The belief state: i.t/ D P Y t D i j Ft X ; i D 1; : : : ; n; Belief State Equation d i.s/ D. A.i; s/ NA.s/ / i.s/ ds C i.s/ B.s/ where. / i.s/ D nx ji.s/ j.s/; jd1 xa.s/ D dw s ; i.0/ D p i ; nx A.j; s/ j.s/; jd1 and fw t g 0tT is a Wiener process given by the formula W t D Z t 0 dx s NA.s/ ds : B.s/
24 Risk Filtering by BSDE (with Ruofan Yan) Suppose.t/ D p and we use a Markov risk filter % t;t 0tT. The value function for the final cost case: h V.t; p/ D % t;t Y t;p i T Structure of % t;t./ [using Coquet, Hu, Mémin, Peng (2002)] If the filter is monotonic, normalized, time consistent, and has the local property (+ minor growth conditions) then a driver g W Œ0; T R R n t;p! R exists, such that % t;t Y T D Vt, where.v ; Z/ solve backward stochastic differential equation h dv s D g.s; V s ; Z s / ds Z s dw s ; s 2 Œt; T ; V T D % T ;T Y t;p i T
25 Equivalent Forward-Backward System Under additional condition of law invariance of T ;T Œ, we obtain the following system. Forward SDE for the belief state: For i D 1; : : : ; n and 0 t s T d i.s/ D. A.i; s/ NA.s/ / i.s/ ds C i.s/ B.s/ dw s ; i.t/ D p i ; Backward SDE for the risk measure: for 0 t s T dv s D g.s; V s ; Z s / ds Z s dw s ; s 2 Œt; T ; V T D r T ;.T / The functional r T.; / is a law invariant risk measure.
26 The Running Cost Case Functional with Running Cost: Z T D Z T 0 c.t/ dt C.Y T / The value function: V.t; p/ D % t;t ŒZ T Forward SDE for the belief state: For i D 1; : : : ; n and 0 t s T d i.s/ D. A.i; s/ NA.s/ / i.s/ ds C i.s/ B.s/ dw s ; i.t/ D p i ; Backward SDE for the risk measure: for 0 t s T dv s D c.s/ C g.s; V s ; Z s / ds Z s dw s ; s 2 Œt; T ; V T D r T ;.T /
27 Controlled Process Controlled transition rates: ij.t; /, 2 U, where U is a bounded set. The rates are uniformly bounded. Piecewise-constant control: For 0 D t 0 < t 1 < t 2 < < t N D T, we define U N i D u 2 U ˇˇ u.t/ D u.tj /; 8 t 2 Œt j ; t jc1 /; 8 j D i; : : : ; N 1 where U is the set of U-valued processes, adapted to F t 0tT. Value function for a fixed control: Z tjc1 V u.t j ; p/ D tj ;t jc1 c. t j ;piu.s/; u r / ds C V u t jc1 ; t j ;piu.t jc1 / t j t j ;piu./ is the belief process restarted at t j from value p, while the system is controlled by u./ D u.t j / in the interval Œt j ; t jc1 /.
28 Dynamic Programming Optimal value function: OV.t j ; p/ D inf u2u N j V u.t j ; p/ Z tjc1 OV.t j ; p/ D inf tj ;t jc1 c. t j ;pi.r/; / dr C OV t jc1 ; t j ;pi.t jc1 / 2U t j Each tj ;t jc1 Œ is given by a controlled FBSDE system on Œt j ; t jc1 d t j ;pi i.s/ D../ / i.s/ ds C t j ;pi i.s/ A.i; s/ NA.s/ dw s ; B.s/ t j ;pi i.t j / D p i ; dv s D c t j ;pi.s/; C g.s; V s ; Z s / ds Z s dw s ; V tjc1 D OV t jc1 ; t j ;pi.t jc1 / Further research: Numerical methods for this FBSDE system
29 References A. Ruszczyński, Risk-averse dynamic programming for Markov decision processes, Mathematical Programming, Series B 125 (2010) Ö. Çavuş and A. Ruszczyński, Computational methods for risk-averse undiscounted transient Markov models, Operations Research, 62 (2), 2014, Ö. Çavuş and A. Ruszczyński, Risk-averse control of undiscounted transient Markov models, SIAM J. on Control and Optimization, 52(6), 2014, J. Fan and A. Ruszczyński, Risk measurement and risk-averse control of partially observable discrete-time Markov systems, Mathematical Methods of Operations Research, 2018, C. McGinity, D. Dentcheva and A. Ruszczyński, Risk-averse approximate dynamic programming for partially observable systems with application to clinical trials, in preparation R. Yan, J. Yao, A. Ruszczyński, Risk Filtering and Risk-Averse Control of Partially Observable Markov Jump Processes, submitted for publication. D. Dentcheva and A. Ruszczyński, Risk measures for continuous-time Markov chains, submitted for publication.
Process-Based Risk Measures for Observable and Partially Observable Discrete-Time Controlled Systems
Process-Based Risk Measures for Observable and Partially Observable Discrete-Time Controlled Systems Jingnan Fan Andrzej Ruszczyński November 5, 2014; revised April 15, 2015 Abstract For controlled discrete-time
More informationKolmogorov Equations and Markov Processes
Kolmogorov Equations and Markov Processes May 3, 013 1 Transition measures and functions Consider a stochastic process {X(t)} t 0 whose state space is a product of intervals contained in R n. We define
More informationOn continuous time contract theory
Ecole Polytechnique, France Journée de rentrée du CMAP, 3 octobre, 218 Outline 1 2 Semimartingale measures on the canonical space Random horizon 2nd order backward SDEs (Static) Principal-Agent Problem
More informationMean-Field optimization problems and non-anticipative optimal transport. Beatrice Acciaio
Mean-Field optimization problems and non-anticipative optimal transport Beatrice Acciaio London School of Economics based on ongoing projects with J. Backhoff, R. Carmona and P. Wang Robust Methods in
More informationHJB equations. Seminar in Stochastic Modelling in Economics and Finance January 10, 2011
Department of Probability and Mathematical Statistics Faculty of Mathematics and Physics, Charles University in Prague petrasek@karlin.mff.cuni.cz Seminar in Stochastic Modelling in Economics and Finance
More informationParticle Filtering Approaches for Dynamic Stochastic Optimization
Particle Filtering Approaches for Dynamic Stochastic Optimization John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge I-Sim Workshop,
More informationApproximation of BSDEs using least-squares regression and Malliavin weights
Approximation of BSDEs using least-squares regression and Malliavin weights Plamen Turkedjiev (turkedji@math.hu-berlin.de) 3rd July, 2012 Joint work with Prof. Emmanuel Gobet (E cole Polytechnique) Plamen
More informationBrownian Motion. 1 Definition Brownian Motion Wiener measure... 3
Brownian Motion Contents 1 Definition 2 1.1 Brownian Motion................................. 2 1.2 Wiener measure.................................. 3 2 Construction 4 2.1 Gaussian process.................................
More informationMarkov Chain BSDEs and risk averse networks
Markov Chain BSDEs and risk averse networks Samuel N. Cohen Mathematical Institute, University of Oxford (Based on joint work with Ying Hu, Robert Elliott, Lukas Szpruch) 2nd Young Researchers in BSDEs
More informationMarkov Chain Monte Carlo Methods for Stochastic Optimization
Markov Chain Monte Carlo Methods for Stochastic Optimization John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge U of Toronto, MIE,
More informationPROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS
PROBABILITY: LIMIT THEOREMS II, SPRING 15. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please
More informationPROBABILITY: LIMIT THEOREMS II, SPRING HOMEWORK PROBLEMS
PROBABILITY: LIMIT THEOREMS II, SPRING 218. HOMEWORK PROBLEMS PROF. YURI BAKHTIN Instructions. You are allowed to work on solutions in groups, but you are required to write up solutions on your own. Please
More informationDecomposability and time consistency of risk averse multistage programs
Decomposability and time consistency of risk averse multistage programs arxiv:1806.01497v1 [math.oc] 5 Jun 2018 A. Shapiro School of Industrial and Systems Engineering Georgia Institute of Technology Atlanta,
More informationNonlinear representation, backward SDEs, and application to the Principal-Agent problem
Nonlinear representation, backward SDEs, and application to the Principal-Agent problem Ecole Polytechnique, France April 4, 218 Outline The Principal-Agent problem Formulation 1 The Principal-Agent problem
More informationThe Wiener Itô Chaos Expansion
1 The Wiener Itô Chaos Expansion The celebrated Wiener Itô chaos expansion is fundamental in stochastic analysis. In particular, it plays a crucial role in the Malliavin calculus as it is presented in
More informationRisk-Averse Dynamic Optimization. Andrzej Ruszczyński. Research supported by the NSF award CMMI
Research supported by the NSF award CMMI-0965689 Outline 1 Risk-Averse Preferences 2 Coherent Risk Measures 3 Dynamic Risk Measurement 4 Markov Risk Measures 5 Risk-Averse Control Problems 6 Solution Methods
More informationOn the Multi-Dimensional Controller and Stopper Games
On the Multi-Dimensional Controller and Stopper Games Joint work with Yu-Jui Huang University of Michigan, Ann Arbor June 7, 2012 Outline Introduction 1 Introduction 2 3 4 5 Consider a zero-sum controller-and-stopper
More informationIntroduction Optimality and Asset Pricing
Introduction Optimality and Asset Pricing Andrea Buraschi Imperial College Business School October 2010 The Euler Equation Take an economy where price is given with respect to the numéraire, which is our
More informationMEAN FIELD GAMES WITH MAJOR AND MINOR PLAYERS
MEAN FIELD GAMES WITH MAJOR AND MINOR PLAYERS René Carmona Department of Operations Research & Financial Engineering PACM Princeton University CEMRACS - Luminy, July 17, 217 MFG WITH MAJOR AND MINOR PLAYERS
More informationON THE POLICY IMPROVEMENT ALGORITHM IN CONTINUOUS TIME
ON THE POLICY IMPROVEMENT ALGORITHM IN CONTINUOUS TIME SAUL D. JACKA AND ALEKSANDAR MIJATOVIĆ Abstract. We develop a general approach to the Policy Improvement Algorithm (PIA) for stochastic control problems
More informationSDE Coefficients. March 4, 2008
SDE Coefficients March 4, 2008 The following is a summary of GARD sections 3.3 and 6., mainly as an overview of the two main approaches to creating a SDE model. Stochastic Differential Equations (SDE)
More informationA class of globally solvable systems of BSDE and applications
A class of globally solvable systems of BSDE and applications Gordan Žitković Department of Mathematics The University of Texas at Austin Thera Stochastics - Wednesday, May 31st, 2017 joint work with Hao
More informationSequential Monte Carlo Methods for Bayesian Computation
Sequential Monte Carlo Methods for Bayesian Computation A. Doucet Kyoto Sept. 2012 A. Doucet (MLSS Sept. 2012) Sept. 2012 1 / 136 Motivating Example 1: Generic Bayesian Model Let X be a vector parameter
More informationThe concentration of a drug in blood. Exponential decay. Different realizations. Exponential decay with noise. dc(t) dt.
The concentration of a drug in blood Exponential decay C12 concentration 2 4 6 8 1 C12 concentration 2 4 6 8 1 dc(t) dt = µc(t) C(t) = C()e µt 2 4 6 8 1 12 time in minutes 2 4 6 8 1 12 time in minutes
More informationMean-Field optimization problems and non-anticipative optimal transport. Beatrice Acciaio
Mean-Field optimization problems and non-anticipative optimal transport Beatrice Acciaio London School of Economics based on ongoing projects with J. Backhoff, R. Carmona and P. Wang Thera Stochastics
More informationTowards a Bayesian model for Cyber Security
Towards a Bayesian model for Cyber Security Mark Briers (mbriers@turing.ac.uk) Joint work with Henry Clausen and Prof. Niall Adams (Imperial College London) 27 September 2017 The Alan Turing Institute
More informationA MODEL FOR THE LONG-TERM OPTIMAL CAPACITY LEVEL OF AN INVESTMENT PROJECT
A MODEL FOR HE LONG-ERM OPIMAL CAPACIY LEVEL OF AN INVESMEN PROJEC ARNE LØKKA AND MIHAIL ZERVOS Abstract. We consider an investment project that produces a single commodity. he project s operation yields
More informationDynamic Risk Measures and Nonlinear Expectations with Markov Chain noise
Dynamic Risk Measures and Nonlinear Expectations with Markov Chain noise Robert J. Elliott 1 Samuel N. Cohen 2 1 Department of Commerce, University of South Australia 2 Mathematical Insitute, University
More informationBrownian Motion. An Undergraduate Introduction to Financial Mathematics. J. Robert Buchanan. J. Robert Buchanan Brownian Motion
Brownian Motion An Undergraduate Introduction to Financial Mathematics J. Robert Buchanan 2010 Background We have already seen that the limiting behavior of a discrete random walk yields a derivation of
More informationEULER MARUYAMA APPROXIMATION FOR SDES WITH JUMPS AND NON-LIPSCHITZ COEFFICIENTS
Qiao, H. Osaka J. Math. 51 (14), 47 66 EULER MARUYAMA APPROXIMATION FOR SDES WITH JUMPS AND NON-LIPSCHITZ COEFFICIENTS HUIJIE QIAO (Received May 6, 11, revised May 1, 1) Abstract In this paper we show
More informationDynamic models. Dependent data The AR(p) model The MA(q) model Hidden Markov models. 6 Dynamic models
6 Dependent data The AR(p) model The MA(q) model Hidden Markov models Dependent data Dependent data Huge portion of real-life data involving dependent datapoints Example (Capture-recapture) capture histories
More informationDecentralized Stochastic Control with Partial Sharing Information Structures: A Common Information Approach
Decentralized Stochastic Control with Partial Sharing Information Structures: A Common Information Approach 1 Ashutosh Nayyar, Aditya Mahajan and Demosthenis Teneketzis Abstract A general model of decentralized
More informationMinimum average value-at-risk for finite horizon semi-markov decision processes
12th workshop on Markov processes and related topics Minimum average value-at-risk for finite horizon semi-markov decision processes Xianping Guo (with Y.H. HUANG) Sun Yat-Sen University Email: mcsgxp@mail.sysu.edu.cn
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond Department of Biomedical Engineering and Computational Science Aalto University January 26, 2012 Contents 1 Batch and Recursive Estimation
More informationHigher Order Linear Equations
C H A P T E R 4 Higher Order Linear Equations 4.1 1. The differential equation is in standard form. Its coefficients, as well as the function g(t) = t, are continuous everywhere. Hence solutions are valid
More informationMean-field SDE driven by a fractional BM. A related stochastic control problem
Mean-field SDE driven by a fractional BM. A related stochastic control problem Rainer Buckdahn, Université de Bretagne Occidentale, Brest Durham Symposium on Stochastic Analysis, July 1th to July 2th,
More informationMath 116 Practice for Exam 2
Math 116 Practice for Exam Generated October 8, 016 Name: SOLUTIONS Instructor: Section Number: 1. This exam has 6 questions. Note that the problems are not of equal difficulty, so you may want to skip
More informationStatistical test for some multistable processes
Statistical test for some multistable processes Ronan Le Guével Joint work in progress with A. Philippe Journées MAS 2014 1 Multistable processes First definition : Ferguson-Klass-LePage series Properties
More informationBirgit Rudloff Operations Research and Financial Engineering, Princeton University
TIME CONSISTENT RISK AVERSE DYNAMIC DECISION MODELS: AN ECONOMIC INTERPRETATION Birgit Rudloff Operations Research and Financial Engineering, Princeton University brudloff@princeton.edu Alexandre Street
More informationPartial Differential Equations with Applications to Finance Seminar 1: Proving and applying Dynkin s formula
Partial Differential Equations with Applications to Finance Seminar 1: Proving and applying Dynkin s formula Group 4: Bertan Yilmaz, Richard Oti-Aboagye and Di Liu May, 15 Chapter 1 Proving Dynkin s formula
More informationNumerical Solutions of ODEs by Gaussian (Kalman) Filtering
Numerical Solutions of ODEs by Gaussian (Kalman) Filtering Hans Kersting joint work with Michael Schober, Philipp Hennig, Tim Sullivan and Han C. Lie SIAM CSE, Atlanta March 1, 2017 Emmy Noether Group
More informationViscosity Solutions of Path-dependent Integro-Differential Equations
Viscosity Solutions of Path-dependent Integro-Differential Equations Christian Keller University of Southern California Conference on Stochastic Asymptotics & Applications Joint with 6th Western Conference
More informationNested Uncertain Differential Equations and Its Application to Multi-factor Term Structure Model
Nested Uncertain Differential Equations and Its Application to Multi-factor Term Structure Model Xiaowei Chen International Business School, Nankai University, Tianjin 371, China School of Finance, Nankai
More informationLecture 12: Diffusion Processes and Stochastic Differential Equations
Lecture 12: Diffusion Processes and Stochastic Differential Equations 1. Diffusion Processes 1.1 Definition of a diffusion process 1.2 Examples 2. Stochastic Differential Equations SDE) 2.1 Stochastic
More informationRisk-Sensitive and Robust Mean Field Games
Risk-Sensitive and Robust Mean Field Games Tamer Başar Coordinated Science Laboratory Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign Urbana, IL - 6181 IPAM
More informationInfinite-dimensional methods for path-dependent equations
Infinite-dimensional methods for path-dependent equations (Università di Pisa) 7th General AMaMeF and Swissquote Conference EPFL, Lausanne, 8 September 215 Based on Flandoli F., Zanco G. - An infinite-dimensional
More informationUniformly Uniformly-ergodic Markov chains and BSDEs
Uniformly Uniformly-ergodic Markov chains and BSDEs Samuel N. Cohen Mathematical Institute, University of Oxford (Based on joint work with Ying Hu, Robert Elliott, Lukas Szpruch) Centre Henri Lebesgue,
More informationMarkov Chain Monte Carlo Methods for Stochastic
Markov Chain Monte Carlo Methods for Stochastic Optimization i John R. Birge The University of Chicago Booth School of Business Joint work with Nicholas Polson, Chicago Booth. JRBirge U Florida, Nov 2013
More informationNumerical scheme for quadratic BSDEs and Dynamic Risk Measures
Numerical scheme for quadratic BSDEs, Numerics and Finance, Oxford Man Institute Julien Azzaz I.S.F.A., Lyon, FRANCE 4 July 2012 Joint Work In Progress with Anis Matoussi, Le Mans, FRANCE Outline Framework
More informationRisk neutral and risk averse approaches to multistage stochastic programming.
Risk neutral and risk averse approaches to multistage stochastic programming. A. Shapiro School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia 30332-0205, USA
More informationSolution of Stochastic Optimal Control Problems and Financial Applications
Journal of Mathematical Extension Vol. 11, No. 4, (2017), 27-44 ISSN: 1735-8299 URL: http://www.ijmex.com Solution of Stochastic Optimal Control Problems and Financial Applications 2 Mat B. Kafash 1 Faculty
More informationUtility Theory CHAPTER Single period utility theory
CHAPTER 7 Utility Theory 7.. Single period utility theory We wish to use a concept of utility that is able to deal with uncertainty. So we introduce the von Neumann Morgenstern utility function. The investor
More informationSMSTC (2007/08) Probability.
SMSTC (27/8) Probability www.smstc.ac.uk Contents 12 Markov chains in continuous time 12 1 12.1 Markov property and the Kolmogorov equations.................... 12 2 12.1.1 Finite state space.................................
More informationQuasi-invariant measures on the path space of a diffusion
Quasi-invariant measures on the path space of a diffusion Denis Bell 1 Department of Mathematics, University of North Florida, 4567 St. Johns Bluff Road South, Jacksonville, FL 32224, U. S. A. email: dbell@unf.edu,
More informationNumerical methods for solving stochastic differential equations
Mathematical Communications 4(1999), 251-256 251 Numerical methods for solving stochastic differential equations Rózsa Horváth Bokor Abstract. This paper provides an introduction to stochastic calculus
More information1 Stochastic Dynamic Programming
1 Stochastic Dynamic Programming Formally, a stochastic dynamic program has the same components as a deterministic one; the only modification is to the state transition equation. When events in the future
More informationUncertainty in Kalman Bucy Filtering
Uncertainty in Kalman Bucy Filtering Samuel N. Cohen Mathematical Institute University of Oxford Research Supported by: The Oxford Man Institute for Quantitative Finance The Oxford Nie Financial Big Data
More informationSystemic Risk and Stochastic Games with Delay
Systemic Risk and Stochastic Games with Delay Jean-Pierre Fouque (with René Carmona, Mostafa Mousavi and Li-Hsien Sun) PDE and Probability Methods for Interactions Sophia Antipolis (France) - March 30-31,
More informationLecture 4: Introduction to stochastic processes and stochastic calculus
Lecture 4: Introduction to stochastic processes and stochastic calculus Cédric Archambeau Centre for Computational Statistics and Machine Learning Department of Computer Science University College London
More informationVector Calculus. A primer
Vector Calculus A primer Functions of Several Variables A single function of several variables: f: R $ R, f x (, x ),, x $ = y. Partial derivative vector, or gradient, is a vector: f = y,, y x ( x $ Multi-Valued
More informationHarnack Inequalities and Applications for Stochastic Equations
p. 1/32 Harnack Inequalities and Applications for Stochastic Equations PhD Thesis Defense Shun-Xiang Ouyang Under the Supervision of Prof. Michael Röckner & Prof. Feng-Yu Wang March 6, 29 p. 2/32 Outline
More informationMATH 5616H INTRODUCTION TO ANALYSIS II SAMPLE FINAL EXAM: SOLUTIONS
MATH 5616H INTRODUCTION TO ANALYSIS II SAMPLE FINAL EXAM: SOLUTIONS You may not use notes, books, etc. Only the exam paper, a pencil or pen may be kept on your desk during the test. Calculators are not
More informationAffine Processes. Econometric specifications. Eduardo Rossi. University of Pavia. March 17, 2009
Affine Processes Econometric specifications Eduardo Rossi University of Pavia March 17, 2009 Eduardo Rossi (University of Pavia) Affine Processes March 17, 2009 1 / 40 Outline 1 Affine Processes 2 Affine
More informationReal Time Stochastic Control and Decision Making: From theory to algorithms and applications
Real Time Stochastic Control and Decision Making: From theory to algorithms and applications Evangelos A. Theodorou Autonomous Control and Decision Systems Lab Challenges in control Uncertainty Stochastic
More informationHigher order weak approximations of stochastic differential equations with and without jumps
Higher order weak approximations of stochastic differential equations with and without jumps Hideyuki TANAKA Graduate School of Science and Engineering, Ritsumeikan University Rough Path Analysis and Related
More informationReversible Coalescing-Fragmentating Wasserstein Dynamics on the Real Line
Reversible Coalescing-Fragmentating Wasserstein Dynamics on the Real Line Vitalii Konarovskyi Leipzig university Berlin-Leipzig workshop in analysis and stochastics (017) Joint work with Max von Renesse
More informationConstrained continuous-time Markov decision processes on the finite horizon
Constrained continuous-time Markov decision processes on the finite horizon Xianping Guo Sun Yat-Sen University, Guangzhou Email: mcsgxp@mail.sysu.edu.cn 16-19 July, 2018, Chengdou Outline The optimal
More informationAmbiguity and Information Processing in a Model of Intermediary Asset Pricing
Ambiguity and Information Processing in a Model of Intermediary Asset Pricing Leyla Jianyu Han 1 Kenneth Kasa 2 Yulei Luo 1 1 The University of Hong Kong 2 Simon Fraser University December 15, 218 1 /
More informationAn efficient approach to stochastic optimal control. Bert Kappen SNN Radboud University Nijmegen the Netherlands
An efficient approach to stochastic optimal control Bert Kappen SNN Radboud University Nijmegen the Netherlands Bert Kappen Examples of control tasks Motor control Bert Kappen Pascal workshop, 27-29 May
More information1. Introduction Systems evolving on widely separated time-scales represent a challenge for numerical simulations. A generic example is
COMM. MATH. SCI. Vol. 1, No. 2, pp. 385 391 c 23 International Press FAST COMMUNICATIONS NUMERICAL TECHNIQUES FOR MULTI-SCALE DYNAMICAL SYSTEMS WITH STOCHASTIC EFFECTS ERIC VANDEN-EIJNDEN Abstract. Numerical
More informationMath 116 Final Exam April 26, 2013
Math 116 Final Exam April 26, 2013 Name: Instructor: Section: 1. Do not open this exam until you are told to do so. 2. This exam has 13 pages including this cover. There are 10 problems. Note that the
More informationAuxiliary Particle Methods
Auxiliary Particle Methods Perspectives & Applications Adam M. Johansen 1 adam.johansen@bristol.ac.uk Oxford University Man Institute 29th May 2008 1 Collaborators include: Arnaud Doucet, Nick Whiteley
More informationOptimal portfolio strategies under partial information with expert opinions
1 / 35 Optimal portfolio strategies under partial information with expert opinions Ralf Wunderlich Brandenburg University of Technology Cottbus, Germany Joint work with Rüdiger Frey Research Seminar WU
More informationProf. Erhan Bayraktar (University of Michigan)
September 17, 2012 KAP 414 2:15 PM- 3:15 PM Prof. (University of Michigan) Abstract: We consider a zero-sum stochastic differential controller-and-stopper game in which the state process is a controlled
More informationFormulas for probability theory and linear models SF2941
Formulas for probability theory and linear models SF2941 These pages + Appendix 2 of Gut) are permitted as assistance at the exam. 11 maj 2008 Selected formulae of probability Bivariate probability Transforms
More informationStochastic Differential Equations
Chapter 5 Stochastic Differential Equations We would like to introduce stochastic ODE s without going first through the machinery of stochastic integrals. 5.1 Itô Integrals and Itô Differential Equations
More informationNotes on Measure Theory and Markov Processes
Notes on Measure Theory and Markov Processes Diego Daruich March 28, 2014 1 Preliminaries 1.1 Motivation The objective of these notes will be to develop tools from measure theory and probability to allow
More informationDynamic Games with Asymmetric Information: Common Information Based Perfect Bayesian Equilibria and Sequential Decomposition
Dynamic Games with Asymmetric Information: Common Information Based Perfect Bayesian Equilibria and Sequential Decomposition 1 arxiv:1510.07001v1 [cs.gt] 23 Oct 2015 Yi Ouyang, Hamidreza Tavafoghi and
More informationP 1.5 X 4.5 / X 2 and (iii) The smallest value of n for
DHANALAKSHMI COLLEGE OF ENEINEERING, CHENNAI DEPARTMENT OF ELECTRONICS AND COMMUNICATION ENGINEERING MA645 PROBABILITY AND RANDOM PROCESS UNIT I : RANDOM VARIABLES PART B (6 MARKS). A random variable X
More informationExercises for Multivariable Differential Calculus XM521
This document lists all the exercises for XM521. The Type I (True/False) exercises will be given, and should be answered, online immediately following each lecture. The Type III exercises are to be done
More informationAn Analytic Method for Solving Uncertain Differential Equations
Journal of Uncertain Systems Vol.6, No.4, pp.244-249, 212 Online at: www.jus.org.uk An Analytic Method for Solving Uncertain Differential Equations Yuhan Liu Department of Industrial Engineering, Tsinghua
More informationMathematical Formulation of Our Example
Mathematical Formulation of Our Example We define two binary random variables: open and, where is light on or light off. Our question is: What is? Computer Vision 1 Combining Evidence Suppose our robot
More informationMathematical Methods for Neurosciences. ENS - Master MVA Paris 6 - Master Maths-Bio ( )
Mathematical Methods for Neurosciences. ENS - Master MVA Paris 6 - Master Maths-Bio (2014-2015) Etienne Tanré - Olivier Faugeras INRIA - Team Tosca November 26th, 2014 E. Tanré (INRIA - Team Tosca) Mathematical
More informationOn a class of stochastic differential equations in a financial network model
1 On a class of stochastic differential equations in a financial network model Tomoyuki Ichiba Department of Statistics & Applied Probability, Center for Financial Mathematics and Actuarial Research, University
More informationOn Kusuoka Representation of Law Invariant Risk Measures
MATHEMATICS OF OPERATIONS RESEARCH Vol. 38, No. 1, February 213, pp. 142 152 ISSN 364-765X (print) ISSN 1526-5471 (online) http://dx.doi.org/1.1287/moor.112.563 213 INFORMS On Kusuoka Representation of
More informationState Space Representation of Gaussian Processes
State Space Representation of Gaussian Processes Simo Särkkä Department of Biomedical Engineering and Computational Science (BECS) Aalto University, Espoo, Finland June 12th, 2013 Simo Särkkä (Aalto University)
More informationHOMEWORK # 3 SOLUTIONS
HOMEWORK # 3 SOLUTIONS TJ HITCHMAN. Exercises from the text.. Chapter 2.4. Problem 32 We are to use variation of parameters to find the general solution to y + 2 x y = 8x. The associated homogeneous equation
More informationBrownian Motion and Stochastic Calculus
ETHZ, Spring 17 D-MATH Prof Dr Martin Larsson Coordinator A Sepúlveda Brownian Motion and Stochastic Calculus Exercise sheet 6 Please hand in your solutions during exercise class or in your assistant s
More informationOpen Problem: Approximate Planning of POMDPs in the class of Memoryless Policies
Open Problem: Approximate Planning of POMDPs in the class of Memoryless Policies Kamyar Azizzadenesheli U.C. Irvine Joint work with Prof. Anima Anandkumar and Dr. Alessandro Lazaric. Motivation +1 Agent-Environment
More informationMulti-armed bandit models: a tutorial
Multi-armed bandit models: a tutorial CERMICS seminar, March 30th, 2016 Multi-Armed Bandit model: general setting K arms: for a {1,..., K}, (X a,t ) t N is a stochastic process. (unknown distributions)
More informationStochastic modelling of epidemic spread
Stochastic modelling of epidemic spread Julien Arino Department of Mathematics University of Manitoba Winnipeg Julien Arino@umanitoba.ca 19 May 2012 1 Introduction 2 Stochastic processes 3 The SIS model
More informationProblem 1. Construct a filtered probability space on which a Brownian motion W and an adapted process X are defined and such that
Stochatic Calculu Example heet 4 - Lent 5 Michael Tehranchi Problem. Contruct a filtered probability pace on which a Brownian motion W and an adapted proce X are defined and uch that dx t = X t t dt +
More informationMean Field Competitive Binary MDPs and Structured Solutions
Mean Field Competitive Binary MDPs and Structured Solutions Minyi Huang School of Mathematics and Statistics Carleton University Ottawa, Canada MFG217, UCLA, Aug 28 Sept 1, 217 1 / 32 Outline of talk The
More informationParticle Filtering for Data-Driven Simulation and Optimization
Particle Filtering for Data-Driven Simulation and Optimization John R. Birge The University of Chicago Booth School of Business Includes joint work with Nicholas Polson. JRBirge INFORMS Phoenix, October
More informationPoisson Jumps in Credit Risk Modeling: a Partial Integro-differential Equation Formulation
Poisson Jumps in Credit Risk Modeling: a Partial Integro-differential Equation Formulation Jingyi Zhu Department of Mathematics University of Utah zhu@math.utah.edu Collaborator: Marco Avellaneda (Courant
More informationNote Set 5: Hidden Markov Models
Note Set 5: Hidden Markov Models Probabilistic Learning: Theory and Algorithms, CS 274A, Winter 2016 1 Hidden Markov Models (HMMs) 1.1 Introduction Consider observed data vectors x t that are d-dimensional
More informationLecture 2: From Linear Regression to Kalman Filter and Beyond
Lecture 2: From Linear Regression to Kalman Filter and Beyond January 18, 2017 Contents 1 Batch and Recursive Estimation 2 Towards Bayesian Filtering 3 Kalman Filter and Bayesian Filtering and Smoothing
More informationBridging the Gap between Center and Tail for Multiscale Processes
Bridging the Gap between Center and Tail for Multiscale Processes Matthew R. Morse Department of Mathematics and Statistics Boston University BU-Keio 2016, August 16 Matthew R. Morse (BU) Moderate Deviations
More informationLecture 4a: Continuous-Time Markov Chain Models
Lecture 4a: Continuous-Time Markov Chain Models Continuous-time Markov chains are stochastic processes whose time is continuous, t [0, ), but the random variables are discrete. Prominent examples of continuous-time
More informationExistence and uniqueness: Picard s theorem
Existence and uniqueness: Picard s theorem First-order equations Consider the equation y = f(x, y) (not necessarily linear). The equation dictates a value of y at each point (x, y), so one would expect
More information