Markov decision processes and interval Markov chains: exploiting the connection
|
|
|
- Coral Baker
- 5 years ago
- Views:
Transcription
1 Markov decision processes and interval Markov chains: exploiting the connection Mingmei Teo Supervisors: Prof. Nigel Bean, Dr Joshua Ross University of Adelaide July 10, 2013
2 Intervals and interval arithmetic Intervals Markov chains Problem We use the notation X = [ X, X ] to represent an interval Interval arithmetic allows us to perform arithmetic operations on intervals and can be represented as follows X Y = {x y : x X, y Y } where X and Y represent intervals and is the arithmetic operator
3 Intervals and interval arithmetic Intervals Markov chains Problem Let X = [ 1, 1]. Then we have X 2 = {x 2 : x [ 1, 1]} = [0, 1] whilst X X = {x 1 x 2 : x 1 [ 1, 1], x 2 [ 1, 1]} = [ 1, 1]. So here, we have the idea of one-sample and re-sample.
4 Intervals Markov chains Problem Computation with interval arithmetic Computational software, e.g. INTLAB Performs arithmetic operations on interval vectors and matrices Solves systems of linear equations with intervals
5 Intervals Markov chains Problem Why might interval arithmetic be useful? Point estimate of parameters with sensitivity analysis Can we avoid the need for sensitivity analysis? Is it possible to directly incorporate the uncertainty of parameter values into our model? Intervals can be used to bound our parameter values, [x error, x + error]
6 Markov chains + intervals =? Intervals Markov chains Problem Consider a discrete time Markov chain with n + 1 states, {0,..., n}, and state 0 an absorbing state Interval transition probability matrix P = [1, 1] [0, 0] [0, 0] [ ] P10, P 10. P s [ ] Pn0, P n0
7 Intervals Markov chains Problem Conditions on the interval transition probability matrix Bounds are valid probabilities, 0 P ij P ij 1 Row sums must satisfy the following, P ij 1 j j P ij
8 Time homogeneity Background Intervals Markov chains Problem Standard Markov chains: One-step transition probability matrix, P, constant over time Interval Markov chains: Time inhomogeneous interval matrix Time homogeneous interval matrix One-sample (Time homogeneous Markov chain) Re-sample (Time inhomogeneous Markov chain)
9 Intervals Markov chains Problem Hitting times and mean hitting times N i is the random variable describing the number of steps required to hit state 0 conditional on starting in state i ν i = E[N i ] is expected number of steps needed to hit state 0 conditional on starting in state i
10 Hitting times problem Intervals Markov chains Problem We want to calculate an interval hitting times vector, [ν, ν], for our interval Markov chain. That is, we want to solve [ν, ν] = (I P s ) 1 1 where I is the identity matrix, 1 is vector of ones, P s is sub-matrix of the interval matrix P and ν and ν represent the lower and upper bounds of the hitting times vector.
11 Intervals Markov chains Problem Can we solve the system of equations directly? Can we just use INTLAB and interval arithmetic to solve the system of equations? INTLAB uses an iterative method to solve the system of equations Problem: ensuring the same realisation of the interval matrix is chosen at each iteration Problem: ensuring j P ij = 1
12 Hitting times interval Intervals Markov chains Problem We seek to calculate the interval hitting times vector of an interval Markov chain by minimising and maximising the hitting times vector, ν = (I P s ) 1 1, where P 11 P 1n P s =..... P 1n P nn is a realisation of the interval P s matrix with the row sums condition obeyed.
13 Maximisation case Background Intervals Markov chains Problem We wanted to solve the following maximisation problem for k = 1,..., n. subject to max ν k = [ ] (I P s ) 1 1 k n P ij = 1, for i = 1,..., n, j=0 P ij P ij P ij, for i = 1,..., n; j = 0,..., n.
14 New formulation of the problem Intervals Markov chains Problem subject to max ν k = [ ] (I P s ) 1 1 k n P ij = 1 P i0, for i = 1,..., n, j=1 P ij P ij P ij, for i, j = 1,..., n.
15 Intervals Markov chains Problem Feasible region of maximisation problem Constraints are row-based Let F i be the feasible region of row i, for i = 1,..., n Represents the possible vectors for the i th row of the P s matrix F i is defined by bounds and linear constraints which form a convex hull
16 What can we do with this? Intervals Markov chains Problem Numerical experience suggests the optimal solution occurs at a vertex of the feasible region Look to prove this conjecture using Markov decision processes (MDPs) We want to be able to represent our maximisation problem as an MDP and exploit existing MDP theory
17 Mapping Proof Conclusions What are Markov decision processes? A way to model decision making processes to optimise a pre-defined objective in a stochastic environment Described by decision times, states, actions, rewards and transition probabilities Optimised by decision rules and policies
18 Mapping Background Mapping Proof Conclusions Lemma Our maximisation problem is a Markov decision process restricted to only consider Markovian decision rules and stationary policies. Prove this by representing our maximisation problem as an MDP
19 Mapping Proof Conclusions Proof: states, decision times and rewards States Both representations involve the same underlying Markov chain Decision times Every time step of the underlying Markov chain Infinite-horizon MDP as we allow the process to continue until absorption Reward = 1 Each step increases the time to absorption by one
20 Proof: actions Background Mapping Proof Conclusions Recall, F i is the feasible region of row i We choose to let each vertex in F i correspond to an action of the MDP when in state i To recover the full feasible region, need convex combinations of vertices convex combinations of actions
21 Proof: transition probabilities Mapping Proof Conclusions Let P (a) i be the associated probability distribution vector for an action a When an action a is chosen in state i, the corresponding P (a) i is inserted into the i th row of the matrix, P s Considering all states i = 1,..., n, we get the P s matrix
22 Mapping Proof Conclusions Proof: Markovian decision rules and stationary policy Markovian decision rules Maximisation problem involves choosing the transition probabilities of a Markov chain Stationary policy We have a time homogeneous (one-sample) interval Markov chain Means optimal P s matrix remains constant over time Hence the choice of decision rule is independent of time
23 Optimal at vertex Background Mapping Proof Conclusions Theorem There exists an optimal solution of the maximisation problem where row i of the optimal matrix, P s, represents a vertex of F i for all i = 1,..., n. Need to show there is no extra benefit from having randomised decision rules as opposed to deterministic decision rules
24 Mapping Proof Conclusions Why do we care about randomised and deterministic? Randomised decision rules convex combination of actions non-vertex of F i Deterministic decision rules single action vertex of F i Want deterministic decision rules!
25 Proof Background Mapping Proof Conclusions Proposition (Proposition of Puterman 1 ) For all v V, sup {r d + P d v} = sup {r d + P d v}. d D MD d D MR This proposition from Puterman 1 gives us that there is nothing to be gained from randomised decision rules So there exists an optimal is obtained for deterministic decision rules 1 M.L. Puterman. : Discrete Stochastic Dynamic Programming
26 Conclusions Background Mapping Proof Conclusions Proven that an optimal solution occurs at a vertex of the feasible region This theorem provides us with a useful analytic property which we can exploit when obtaining the optimal solution through numerical methods
27 What else? Background Mapping Proof Conclusions Determine if interval analysis can be used to investigate model sensitivity Vary width of intervals for parameters and see effect on mean hitting times intervals
28 Background?
29 Counter-example for an analytic solution Consider the following interval transition probability matrix, [1, 1] [0, 0] [0, 0] [0, 0] [0.3, 0.35] [0, 1] [0, 0] [0, 0.1] P =. [0.2, 0.3] [0, 1] [0, 1] [0, 1] [0.1, 0.2] [0, 1] [0, 0.3] [0, 0]
30 Counter-example for an analytic solution Our proposed analytic solution: P s = Optimal solution obtained numerically from MATLAB: Ps =
Interval Markov chains: Performance measures and sensitivity analysis
Interval Markov chains: Performance measures and sensitivity analysis Mingmei Teo Thesis submitted for the degree of Master of Philosophy in Applied Mathematics at The University of Adelaide School of
, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
The quest for finding Hamiltonian cycles
The quest for finding Hamiltonian cycles Giang Nguyen School of Mathematical Sciences University of Adelaide Travelling Salesman Problem Given a list of cities and distances between cities, what is the
Section Notes 9. Midterm 2 Review. Applied Math / Engineering Sciences 121. Week of December 3, 2018
Section Notes 9 Midterm 2 Review Applied Math / Engineering Sciences 121 Week of December 3, 2018 The following list of topics is an overview of the material that was covered in the lectures and sections
Infinite-Horizon Average Reward Markov Decision Processes
Infinite-Horizon Average Reward Markov Decision Processes Dan Zhang Leeds School of Business University of Colorado at Boulder Dan Zhang, Spring 2012 Infinite Horizon Average Reward MDP 1 Outline The average
Markov Model. Model representing the different resident states of a system, and the transitions between the different states
Markov Model Model representing the different resident states of a system, and the transitions between the different states (applicable to repairable, as well as non-repairable systems) System behavior
Discrete planning (an introduction)
Sistemi Intelligenti Corso di Laurea in Informatica, A.A. 2017-2018 Università degli Studi di Milano Discrete planning (an introduction) Nicola Basilico Dipartimento di Informatica Via Comelico 39/41-20135
MDP Preliminaries. Nan Jiang. February 10, 2019
MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process
Lecture notes for Analysis of Algorithms : Markov decision processes
Lecture notes for Analysis of Algorithms : Markov decision processes Lecturer: Thomas Dueholm Hansen June 6, 013 Abstract We give an introduction to infinite-horizon Markov decision processes (MDPs) with
Infinite-Horizon Discounted Markov Decision Processes
Infinite-Horizon Discounted Markov Decision Processes Dan Zhang Leeds School of Business University of Colorado at Boulder Dan Zhang, Spring 2012 Infinite Horizon Discounted MDP 1 Outline The expected
Decision Theory: Markov Decision Processes
Decision Theory: Markov Decision Processes CPSC 322 Lecture 33 March 31, 2006 Textbook 12.5 Decision Theory: Markov Decision Processes CPSC 322 Lecture 33, Slide 1 Lecture Overview Recap Rewards and Policies
Complexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning
Complexity of stochastic branch and bound methods for belief tree search in Bayesian reinforcement learning Christos Dimitrakakis Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands
Lecture 3: Markov Decision Processes
Lecture 3: Markov Decision Processes Joseph Modayil 1 Markov Processes 2 Markov Reward Processes 3 Markov Decision Processes 4 Extensions to MDPs Markov Processes Introduction Introduction to MDPs Markov
Chapter 2 SOME ANALYTICAL TOOLS USED IN THE THESIS
Chapter 2 SOME ANALYTICAL TOOLS USED IN THE THESIS 63 2.1 Introduction In this chapter we describe the analytical tools used in this thesis. They are Markov Decision Processes(MDP), Markov Renewal process
Stochastic Shortest Path Problems
Chapter 8 Stochastic Shortest Path Problems 1 In this chapter, we study a stochastic version of the shortest path problem of chapter 2, where only probabilities of transitions along different arcs can
(b) What is the variance of the time until the second customer arrives, starting empty, assuming that we measure time in minutes?
IEOR 3106: Introduction to Operations Research: Stochastic Models Fall 2006, Professor Whitt SOLUTIONS to Final Exam Chapters 4-7 and 10 in Ross, Tuesday, December 19, 4:10pm-7:00pm Open Book: but only
Reinforcement Learning and Deep Reinforcement Learning
Reinforcement Learning and Deep Reinforcement Learning Ashis Kumer Biswas, Ph.D. ashis.biswas@ucdenver.edu Deep Learning November 5, 2018 1 / 64 Outlines 1 Principles of Reinforcement Learning 2 The Q
REINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
Reinforcement Learning
Reinforcement Learning March May, 2013 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha
Markov decision processes
CS 2740 Knowledge representation Lecture 24 Markov decision processes Milos Hauskrecht milos@cs.pitt.edu 5329 Sennott Square Administrative announcements Final exam: Monday, December 8, 2008 In-class Only
Power Allocation over Two Identical Gilbert-Elliott Channels
Power Allocation over Two Identical Gilbert-Elliott Channels Junhua Tang School of Electronic Information and Electrical Engineering Shanghai Jiao Tong University, China Email: junhuatang@sjtu.edu.cn Parisa
Computational complexity estimates for value and policy iteration algorithms for total-cost and average-cost Markov decision processes
Computational complexity estimates for value and policy iteration algorithms for total-cost and average-cost Markov decision processes Jefferson Huang Dept. Applied Mathematics and Statistics Stony Brook
Probabilistic Planning. George Konidaris
Probabilistic Planning George Konidaris gdk@cs.brown.edu Fall 2017 The Planning Problem Finding a sequence of actions to achieve some goal. Plans It s great when a plan just works but the world doesn t
RECURSION EQUATION FOR
Math 46 Lecture 8 Infinite Horizon discounted reward problem From the last lecture: The value function of policy u for the infinite horizon problem with discount factor a and initial state i is W i, u
Introduction to Reinforcement Learning Part 1: Markov Decision Processes
Introduction to Reinforcement Learning Part 1: Markov Decision Processes Rowan McAllister Reinforcement Learning Reading Group 8 April 2015 Note I ve created these slides whilst following Algorithms for
AN ABSTRACT OF THE THESIS OF
AN ABSTRACT OF THE THESIS OF Thai Duong for the degree of Master of Science in Computer Science presented on May 20, 2015. Title: Data Collection in Sensor Networks via the Novel Fast Markov Decision Process
Linear Dependence of Stationary Distributions in Ergodic Markov Decision Processes
Linear ependence of Stationary istributions in Ergodic Markov ecision Processes Ronald Ortner epartment Mathematik und Informationstechnologie, Montanuniversität Leoben Abstract In ergodic MPs we consider
Control Theory : Course Summary
Control Theory : Course Summary Author: Joshua Volkmann Abstract There are a wide range of problems which involve making decisions over time in the face of uncertainty. Control theory draws from the fields
The Simplex and Policy Iteration Methods are Strongly Polynomial for the Markov Decision Problem with Fixed Discount
The Simplex and Policy Iteration Methods are Strongly Polynomial for the Markov Decision Problem with Fixed Discount Yinyu Ye Department of Management Science and Engineering and Institute of Computational
Stochastic Safest and Shortest Path Problems
Stochastic Safest and Shortest Path Problems Florent Teichteil-Königsbuch AAAI-12, Toronto, Canada July 24-26, 2012 Path optimization under probabilistic uncertainties Problems coming to searching for
An Introduction to Markov Decision Processes. MDP Tutorial - 1
An Introduction to Markov Decision Processes Bob Givan Purdue University Ron Parr Duke University MDP Tutorial - 1 Outline Markov Decision Processes defined (Bob) Objective functions Policies Finding Optimal
Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
Markov Decision Processes
Markov Decision Processes Lecture notes for the course Games on Graphs B. Srivathsan Chennai Mathematical Institute, India 1 Markov Chains We will define Markov chains in a manner that will be useful to
Simplex Algorithm for Countable-state Discounted Markov Decision Processes
Simplex Algorithm for Countable-state Discounted Markov Decision Processes Ilbin Lee Marina A. Epelman H. Edwin Romeijn Robert L. Smith November 16, 2014 Abstract We consider discounted Markov Decision
AM 121: Intro to Optimization Models and Methods: Fall 2018
AM 11: Intro to Optimization Models and Methods: Fall 018 Lecture 18: Markov Decision Processes Yiling Chen Lesson Plan Markov decision processes Policies and value functions Solving: average reward, discounted
Stochastic processes. MAS275 Probability Modelling. Introduction and Markov chains. Continuous time. Markov property
Chapter 1: and Markov chains Stochastic processes We study stochastic processes, which are families of random variables describing the evolution of a quantity with time. In some situations, we can treat
MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
Reinforcement Learning
1 Reinforcement Learning Chris Watkins Department of Computer Science Royal Holloway, University of London July 27, 2015 2 Plan 1 Why reinforcement learning? Where does this theory come from? Markov decision
Motivation for introducing probabilities
for introducing probabilities Reaching the goals is often not sufficient: it is important that the expected costs do not outweigh the benefit of reaching the goals. 1 Objective: maximize benefits - costs.
Classification of Countable State Markov Chains
Classification of Countable State Markov Chains Friday, March 21, 2014 2:01 PM How can we determine whether a communication class in a countable state Markov chain is: transient null recurrent positive
Markov Decision Processes
Markov Decision Processes Noel Welsh 11 November 2010 Noel Welsh () Markov Decision Processes 11 November 2010 1 / 30 Annoucements Applicant visitor day seeks robot demonstrators for exciting half hour
Distributed Optimization. Song Chong EE, KAIST
Distributed Optimization Song Chong EE, KAIST songchong@kaist.edu Dynamic Programming for Path Planning A path-planning problem consists of a weighted directed graph with a set of n nodes N, directed links
Robustness of policies in Constrained Markov Decision Processes
1 Robustness of policies in Constrained Markov Decision Processes Alexander Zadorojniy and Adam Shwartz, Senior Member, IEEE Abstract We consider the optimization of finite-state, finite-action Markov
MATH 564/STAT 555 Applied Stochastic Processes Homework 2, September 18, 2015 Due September 30, 2015
ID NAME SCORE MATH 56/STAT 555 Applied Stochastic Processes Homework 2, September 8, 205 Due September 30, 205 The generating function of a sequence a n n 0 is defined as As : a ns n for all s 0 for which
CS 7180: Behavioral Modeling and Decisionmaking
CS 7180: Behavioral Modeling and Decisionmaking in AI Markov Decision Processes for Complex Decisionmaking Prof. Amy Sliva October 17, 2012 Decisions are nondeterministic In many situations, behavior and
21 Markov Decision Processes
2 Markov Decision Processes Chapter 6 introduced Markov chains and their analysis. Most of the chapter was devoted to discrete time Markov chains, i.e., Markov chains that are observed only at discrete
The Markov Decision Process (MDP) model
Decision Making in Robots and Autonomous Agents The Markov Decision Process (MDP) model Subramanian Ramamoorthy School of Informatics 25 January, 2013 In the MAB Model We were in a single casino and the
A Shadow Simplex Method for Infinite Linear Programs
A Shadow Simplex Method for Infinite Linear Programs Archis Ghate The University of Washington Seattle, WA 98195 Dushyant Sharma The University of Michigan Ann Arbor, MI 48109 May 25, 2009 Robert L. Smith
MATH4406 Assignment 5
MATH4406 Assignment 5 Patrick Laub (ID: 42051392) October 7, 2014 1 The machine replacement model 1.1 Real-world motivation Consider the machine to be the entire world. Over time the creator has running
Optimism in the Face of Uncertainty Should be Refutable
Optimism in the Face of Uncertainty Should be Refutable Ronald ORTNER Montanuniversität Leoben Department Mathematik und Informationstechnolgie Franz-Josef-Strasse 18, 8700 Leoben, Austria, Phone number:
Bayesian Congestion Control over a Markovian Network Bandwidth Process
Bayesian Congestion Control over a Markovian Network Bandwidth Process Parisa Mansourifard 1/30 Bayesian Congestion Control over a Markovian Network Bandwidth Process Parisa Mansourifard (USC) Joint work
Game Theory and its Applications to Networks - Part I: Strict Competition
Game Theory and its Applications to Networks - Part I: Strict Competition Corinne Touati Master ENS Lyon, Fall 200 What is Game Theory and what is it for? Definition (Roger Myerson, Game Theory, Analysis
Optimisation and Operations Research
Optimisation and Operations Research Lecture 22: Linear Programming Revisited Matthew Roughan http://www.maths.adelaide.edu.au/matthew.roughan/ Lecture_notes/OORII/ School
On Finding Optimal Policies for Markovian Decision Processes Using Simulation
On Finding Optimal Policies for Markovian Decision Processes Using Simulation Apostolos N. Burnetas Case Western Reserve University Michael N. Katehakis Rutgers University February 1995 Abstract A simulation
Some notes on Markov Decision Theory
Some notes on Markov Decision Theory Nikolaos Laoutaris laoutaris@di.uoa.gr January, 2004 1 Markov Decision Theory[1, 2, 3, 4] provides a methodology for the analysis of probabilistic sequential decision
STOCHASTIC MODELS FOR RELIABILITY, AVAILABILITY, AND MAINTAINABILITY
STOCHASTIC MODELS FOR RELIABILITY, AVAILABILITY, AND MAINTAINABILITY Ph.D. Assistant Professor Industrial and Systems Engineering Auburn University RAM IX Summit November 2 nd 2016 Outline Introduction
STOCHASTIC PROCESSES Basic notions
J. Virtamo 38.3143 Queueing Theory / Stochastic processes 1 STOCHASTIC PROCESSES Basic notions Often the systems we consider evolve in time and we are interested in their dynamic behaviour, usually involving
x x 1 0 1/(N 1) (N 2)/(N 1)
Please simplify your answers to the extent reasonable without a calculator, show your work, and explain your answers, concisely. If you set up an integral or a sum that you cannot evaluate, leave it as
Markov Chains and Stochastic Sampling
Part I Markov Chains and Stochastic Sampling 1 Markov Chains and Random Walks on Graphs 1.1 Structure of Finite Markov Chains We shall only consider Markov chains with a finite, but usually very large,
1 Markov decision processes
2.997 Decision-Making in Large-Scale Systems February 4 MI, Spring 2004 Handout #1 Lecture Note 1 1 Markov decision processes In this class we will study discrete-time stochastic systems. We can describe
Markov Decision Processes and Dynamic Programming
Markov Decision Processes and Dynamic Programming A. LAZARIC (SequeL Team @INRIA-Lille) ENS Cachan - Master 2 MVA SequeL INRIA Lille MVA-RL Course How to model an RL problem The Markov Decision Process
MARKOV DECISION PROCESSES
J. Virtamo 38.3141 Teletraffic Theory / Markov decision processes 1 MARKOV DECISION PROCESSES In studying Markov processes we have up till now assumed that the system, its states and transition probabilities
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free
On Polynomial Cases of the Unichain Classification Problem for Markov Decision Processes
On Polynomial Cases of the Unichain Classification Problem for Markov Decision Processes Eugene A. Feinberg Department of Applied Mathematics and Statistics State University of New York at Stony Brook
GMDPtoolbox: a Matlab library for solving Graph-based Markov Decision Processes
GMDPtoolbox: a Matlab library for solving Graph-based Markov Decision Processes Marie-Josée Cros, Nathalie Peyrard, Régis Sabbadin UR MIAT, INRA Toulouse, France JFRB, Clermont Ferrand, 27-28 juin 2016
Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016
Course 16:198:520: Introduction To Artificial Intelligence Lecture 13 Decision Making Abdeslam Boularias Wednesday, December 7, 2016 1 / 45 Overview We consider probabilistic temporal models where the
On the Approximate Solution of POMDP and the Near-Optimality of Finite-State Controllers
On the Approximate Solution of POMDP and the Near-Optimality of Finite-State Controllers Huizhen (Janey) Yu (janey@mit.edu) Dimitri Bertsekas (dimitrib@mit.edu) Lab for Information and Decision Systems,
Minimum average value-at-risk for finite horizon semi-markov decision processes
12th workshop on Markov processes and related topics Minimum average value-at-risk for finite horizon semi-markov decision processes Xianping Guo (with Y.H. HUANG) Sun Yat-Sen University Email: mcsgxp@mail.sysu.edu.cn
Risk-Sensitive and Average Optimality in Markov Decision Processes
Risk-Sensitive and Average Optimality in Markov Decision Processes Karel Sladký Abstract. This contribution is devoted to the risk-sensitive optimality criteria in finite state Markov Decision Processes.
CS788 Dialogue Management Systems Lecture #2: Markov Decision Processes
CS788 Dialogue Management Systems Lecture #2: Markov Decision Processes Kee-Eung Kim KAIST EECS Department Computer Science Division Markov Decision Processes (MDPs) A popular model for sequential decision
http://www.math.uah.edu/stat/markov/.xhtml 1 of 9 7/16/2009 7:20 AM Virtual Laboratories > 16. Markov Chains > 1 2 3 4 5 6 7 8 9 10 11 12 1. A Markov process is a random process in which the future is
Markov Chains. Chapter 16. Markov Chains - 1
Markov Chains Chapter 16 Markov Chains - 1 Why Study Markov Chains? Decision Analysis focuses on decision making in the face of uncertainty about one future event. However, many decisions need to consider
Bayesian Congestion Control over a Markovian Network Bandwidth Process: A multiperiod Newsvendor Problem
Bayesian Congestion Control over a Markovian Network Bandwidth Process: A multiperiod Newsvendor Problem Parisa Mansourifard 1/37 Bayesian Congestion Control over a Markovian Network Bandwidth Process:
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
Markov Decision Processes and Dynamic Programming
Markov Decision Processes and Dynamic Programming A. LAZARIC (SequeL Team @INRIA-Lille) Ecole Centrale - Option DAD SequeL INRIA Lille EC-RL Course In This Lecture A. LAZARIC Markov Decision Processes
Linear Programming Methods
Chapter 11 Linear Programming Methods 1 In this chapter we consider the linear programming approach to dynamic programming. First, Bellman s equation can be reformulated as a linear program whose solution
Markov Decision Processes and Solving Finite Problems. February 8, 2017
Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:
Example: physical systems. If the state space. Example: speech recognition. Context can be. Example: epidemics. Suppose each infected
4. Markov Chains A discrete time process {X n,n = 0,1,2,...} with discrete state space X n {0,1,2,...} is a Markov chain if it has the Markov property: P[X n+1 =j X n =i,x n 1 =i n 1,...,X 0 =i 0 ] = P[X
Prof. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be
REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while
Point Process Control
Point Process Control The following note is based on Chapters I, II and VII in Brémaud s book Point Processes and Queues (1981). 1 Basic Definitions Consider some probability space (Ω, F, P). A real-valued
Probabilistic Model Checking and Strategy Synthesis for Robot Navigation
Probabilistic Model Checking and Strategy Synthesis for Robot Navigation Dave Parker University of Birmingham (joint work with Bruno Lacerda, Nick Hawes) AIMS CDT, Oxford, May 2015 Overview Probabilistic
Reductions Of Undiscounted Markov Decision Processes and Stochastic Games To Discounted Ones. Jefferson Huang
Reductions Of Undiscounted Markov Decision Processes and Stochastic Games To Discounted Ones Jefferson Huang School of Operations Research and Information Engineering Cornell University November 16, 2016
Partially Observable Markov Decision Processes (POMDPs) Pieter Abbeel UC Berkeley EECS
Partially Observable Markov Decision Processes (POMDPs) Pieter Abbeel UC Berkeley EECS Many slides adapted from Jur van den Berg Outline POMDPs Separation Principle / Certainty Equivalence Locally Optimal
Maximizing throughput in zero-buffer tandem lines with dedicated and flexible servers
Maximizing throughput in zero-buffer tandem lines with dedicated and flexible servers Mohammad H. Yarmand and Douglas G. Down Department of Computing and Software, McMaster University, Hamilton, ON, L8S
Decentralized Control of Stochastic Systems
Decentralized Control of Stochastic Systems Sanjay Lall Stanford University CDC-ECC Workshop, December 11, 2005 2 S. Lall, Stanford 2005.12.11.02 Decentralized Control G 1 G 2 G 3 G 4 G 5 y 1 u 1 y 2 u
Reinforcement learning an introduction
Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,
The L-Shaped Method. Operations Research. Anthony Papavasiliou 1 / 38
1 / 38 The L-Shaped Method Operations Research Anthony Papavasiliou Contents 2 / 38 1 The L-Shaped Method 2 Example: Capacity Expansion Planning 3 Examples with Optimality Cuts [ 5.1a of BL] 4 Examples
FEL3330 Networked and Multi-Agent Control Systems. Lecture 11: Distributed Estimation
FEL3330, Lecture 11 1 June 8, 2011 FEL3330 Networked and Multi-Agent Control Systems Lecture 11: Distributed Estimation Distributed Estimation Distributed Localization where R X is the covariance of X.
Dantzig s pivoting rule for shortest paths, deterministic MDPs, and minimum cost to time ratio cycles
Dantzig s pivoting rule for shortest paths, deterministic MDPs, and minimum cost to time ratio cycles Thomas Dueholm Hansen 1 Haim Kaplan Uri Zwick 1 Department of Management Science and Engineering, Stanford
Influence of product return lead-time on inventory control
Influence of product return lead-time on inventory control Mohamed Hichem Zerhouni, Jean-Philippe Gayon, Yannick Frein To cite this version: Mohamed Hichem Zerhouni, Jean-Philippe Gayon, Yannick Frein.
Language Acquisition and Parameters: Part II
Language Acquisition and Parameters: Part II Matilde Marcolli CS0: Mathematical and Computational Linguistics Winter 205 Transition Matrices in the Markov Chain Model absorbing states correspond to local
Reinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
An Empirical Algorithm for Relative Value Iteration for Average-cost MDPs
2015 IEEE 54th Annual Conference on Decision and Control CDC December 15-18, 2015. Osaka, Japan An Empirical Algorithm for Relative Value Iteration for Average-cost MDPs Abhishek Gupta Rahul Jain Peter
Exponential Moving Average Based Multiagent Reinforcement Learning Algorithms
Exponential Moving Average Based Multiagent Reinforcement Learning Algorithms Mostafa D. Awheda Department of Systems and Computer Engineering Carleton University Ottawa, Canada KS 5B6 Email: mawheda@sce.carleton.ca
Planning and Acting in Partially Observable Stochastic Domains
Planning and Acting in Partially Observable Stochastic Domains Leslie Pack Kaelbling*, Michael L. Littman**, Anthony R. Cassandra*** *Computer Science Department, Brown University, Providence, RI, USA
Decision Theory: Q-Learning
Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning
Markov Decision Processes Infinite Horizon Problems
Markov Decision Processes Infinite Horizon Problems Alan Fern * * Based in part on slides by Craig Boutilier and Daniel Weld 1 What is a solution to an MDP? MDP Planning Problem: Input: an MDP (S,A,R,T)
Markov chains. Randomness and Computation. Markov chains. Markov processes
Markov chains Randomness and Computation or, Randomized Algorithms Mary Cryan School of Informatics University of Edinburgh Definition (Definition 7) A discrete-time stochastic process on the state space
A linear programming approach to nonstationary infinite-horizon Markov decision processes
A linear programming approach to nonstationary infinite-horizon Markov decision processes Archis Ghate Robert L Smith July 24, 2012 Abstract Nonstationary infinite-horizon Markov decision processes (MDPs)
Notation. state space
Notation S state space i, j, w, z states Φ parameter space modelling decision rules and perturbations φ elements of parameter space δ decision rule and stationary, deterministic policy δ(i) decision rule