Q-Learning and SARSA: Machine learningbased stochastic control approaches for financial trading
|
|
|
- Madlyn Fields
- 5 years ago
- Views:
Transcription
1 Q-Learning and SARSA: Machine learningbased stochastic control approaches for financial trading Marco CORAZZA Department of Economics Ca' Foscari University of Venice CONFERENCE ON COMPUTATIONAL MANAGEMENT SCIENCE Bergamo, Italy May 30-31, June 1, 201
2 Aim QL & SARSA: Machine learning-based stochastic control approaches for financial trading To investigate the capabilities of an intelligent investor to make effective financial trading.
3 Summary QL & SARSA: Machine learning-based stochastic control approaches for financial trading 0 Introduction (EMH vs. AMH) 1 Reinforcement learning 2 Q-Learning and SARSA 3 Operational implementation 4 Application to the Italian stock market 5 Some final remarks
4 0 Introduction (1/8) The starting question Can machine learning methods be used to develop profitable trading systems? Classical answer Modern answer
5 0 Introduction (2/8) Classical answer Efficient Market Hypothesis (EMH) In financial markets, it is possible to "gain" only randomly and rarely. In a liquid financial market, asset prices fully reflect all currently available information. The EMH exists in various degrees: Weak form (past prices and volumes); Semi-strong form (public information); Strong form (public and private information).
6 0 Introduction (3/8) Classical answer
7 0 Introduction (4/8) Classical answer According to EMH, asset price movements can be described as P t+1 = E(P t+1 Ω t ) + ε t+1 = P t where t and t+1: consecutive time instants, P τ : asset price at time τ, E( ): expectation operator, P t+1 : random variable asset price at time t+1, Ω t : set of all available information at time t, ε t+1: error term at time t+1, with E(ε t+1) = 0.
8 0 Introduction (5/8) Modern answer Adaptive Market Hypothesis Financial markets are systems in which the agents interact among them in complex ways in order to maximize their profits. Agents react to the news in a not fully rational ways, making mistakes but learning and adapting their behaviors. So, financial markets are generally inefficient and the traders can more or less systematically implement profitable strategies.
9 0 Introduction (6/8) Modern answer
10 0 Introduction (7/8) Summarizing It is reasonable to assume AMH is valid. Therefore, to develop profitable trading systems it is necessary to provide them with problem-solving capabilities similar to those of the superior living beings, that is to make them "intelligent".
11 1 Reinforcement Learning: A sketch (1/3) Machine learning is a research field studying methodologies by which a computer is able to perform a finalized action given its state depending on an algorithmic learning based on past and current data. Main machine learning methodologies Supervised learning Unsupervised learning Reinforcement learning
12 1 Reinforcement Learning: A sketch (2/3)
13 1 Reinforcement Learning: A sketch (3/3) time t: the system detects the state of the environment; time t: on this basis, the systems takes an action; time t + 1: the environment returns a negative or positive reward coherently with the exactness, or less, of the action; time t + 1: the agent modifies its knowledge, that is it learns, depending on whether the action is incorrect or correct.
14 1 Reinforcement Learning: Formalization (1/5) Discrete time framework: 0, 1,, t, t + 1, ; State vector: s t R n, it contains the relevant information; Action: a t A(s t ) R, where A(s t ) is the set of the possible actions according to the state s t ; Reward function: r(s t, a t, s t+1 );
15 1 Reinforcement Learning: Formalization (2/5) System's goal: To maximize the expected value of R(s t ) = r(s t, a t, s t+1 ) + γ r(s t+1, a t+1, s t+2 ) + + +γ 2 r(s t+2, a t+2, s t+3 ) + where γ [0, 1[ is a discount factor; Reinforcement Learning's goal: To detect a policy, that is a map π(s t ) = a t, which associates to each state the action that maximizes the expected value of R(s t );
16 1 Reinforcement Learning: Formalization (3/5) Reformulated system's goal: To maximize the expexted value of R π (s t ) = r(s t, π(s t ), s t+1 ) + γ r(s t+1, π(s t+1 ), s t+2 ) + + +γ 2 r(s t+2, π(s t+2 ), s t+3 ) + ; Value functions: For maximize the expexted value of R(s t ) or R π (s t ), one introduce V π (s) = E(R π (s t ) s t = s) or Q π (s, a) = E(R π (s t ) s t = s, a t = a), where s and a are particular value of s t and a t respectively;
17 1 Reinforcement Learning: Formalization (4/5) Bellman equation: For the value equations, the following recursive relationship holds V π (s) = E[r(s t, π(s t ), s t+1 ) + γ V π (s t+1 ) s t+1 = s] from which the following approximate iterative estimator π (s t ) = E[r(s t, π(s t ), s t+1 ) + γ V kπ (s t+1 )]. V k+1
18 1 Reinforcement Learning: Formalization (5/5) In order to improve the policy at each instant time, the following approximate iterative procedure is considered π V k+1 (s t ) = E[r(s t, a, s t+1 ) + γ V kπ (s t+1 )] for some a where π V k+1 (s t ) is the update estimate at iteration k + 1 and with a t = { π (s t ) with probability 1 ε a A(s t ) with probability ε where ε [0, 1] and π (s t ) is the action which maximizes Q π (s, a).
19 2 The Q-Learning and SARSA algorithms (1/5) In the Q-Learning framework, the approximate iterative procedure becomes Q k+1 (s t, a t ) = Q k(s t, a t ) k + 1 [r(s t, a t, s t+1 ) + γ max a Q k(s t+1, a t ) Q k(s t, a t )]; In the SARSA framework, the approximate iterative procedure becomes Q k+1 (s t, a t ) = Q k(s t, a t ) k + 1 [r(s t, a t, s t+1 ) + γ Q k(s t+1, a t ) Q k(s t, a t )];
20 2 The Q-Learning and SARSA algorithms (2/5) As the system takes into account continuous s t, Q k(s t, a t ) may be approximate by Q k(s t, a t ) = Q k(s t, a t ; θ k ) where θ k is a parameter vector, randomly initialized.
21 2 The Q-Learning and SARSA algorithms (3/5) For obtaining the best estimation of Q π (s t, a t ), the following linear functional is considered Q k(s, a; θ k ) = θ 0,k + θ i,k φ(s i ) n i=1 n+m + θ i,k φ(a i ) i=n+1 where s = (s 1,, s n ) is the state vector, a = (a 1,, a m ) is the action vector, φ( ) is a proper transformation function of states.
22 2 The Q-Learning and SARSA algorithms (4/5) Specification of the transformation or squashing function φ( ): a φ(x) = d, 1 + be cx with a = 2, b = 1, c = and d = 1
23 2 The Q-Learning and SARSA algorithms (5/5) Finally, one obtain the following iterative updating rules { d k = r(s t, a t, s t+1 ) + γ max a Q π (s t+1, a t ; θ k ) Q π (s t, a t ; θ k ) θ k+1 = θ k + α k d k θ Q π (s t+1, a t ; θ k ) where α k ]0, 1] is the so called learning rate. Adaptive stochastic optimal conrol problem.
24 3 Operational implementation (1/5) Specification of the state vector s t ; Specification of the actions a t ; Specification of the reward function r(s t, a t, s t+1 ); Other.
25 3 Operational implementation (2/5) Specification of the state vector s t : s t = (e t n, e t (n 1),, e t, a t 1 ), where e t = ln ( p t p t 1 ) and with n {0, 4}; Specification of the actions a t : 1 (short position) a t = { 0 (stay out position) ; +1 (long position)
26 3 Operational implementation (3/5) Specification of the reward function r(s t, a t, s t+1 ), Sharpe ratio: where g t = a t 1 e t and with L {5, 22}; r(s t, a t, s t+1 ) = g t,l L 1 (g t i g t,l) 2 i=0 L 1 Specification of the transaction costs: 0,15%;,
27 3 Operational implementation (4/5) α = 5,0%, γ = 95%, ε {2.5%, 5.0%, 10.0%}; Number of iterations: 500; (both Q-Learning and SARSA are stochastic algorithms)
28 3 Operational implementation (5/5) Determination of the action: 1 if a t [ 1, 1 3 [ a t = 0 if a t [ 1 3, 1 3 [, where +1 if { a t [ 1 3, 1] a t = a t,i. i=1
29 4 Applications (1/9) Daily closing prices: From January 2, 1985 to September 30, 2014 (7518 data): Assicurazioni Generali FIAT Pirelli & C. Saipem Telecom Italia UniCredit
30 4 Applications (2/9) (2 algorithms of RL) X (2 values of n) X (2 values of L) X (3 values of ε) = 24 configurations
31 4 Applications (3/9) Telecom Italia Q-Learning, n = 5, L = 22, ε = 10.0%
32 4 Applications (4/9) Telecom Italia SARSA, n = 5, L = 22, ε = 10.0%
33 4 Applications (5/9) Pirelli & C. Q-Learning vs SARSA, n = 5, L = 22, ε = 10.0%
34 4 Applications (6/9) g[%]: Net average return per year; %: Percentage of time such that C(t) > C(1), with t = 2,, T = 7518; #: Average number of trade per year; g[%]net>0: Percentage of configurations for which g[%] > 0; #>50%: Percentage of configurations such that % > 0 in the 50% of the days.
35 4 Applications (7/9) Best case Assicurazioni Generali
36 4 Applications (8/9) Worst case Pirelli & C.
37 4 Applications (9/9)
38 5 Some concluding remarks Performances o Both Q-Learning and SARSA achieve satisfactory results, although they use simple states and not refined configurations. Log-returns as states o Simple choice. (I am working on some new states) Reward function o The Sharpe ratio is too basic. (I am working on some new performance measures) Estimation of Q π (s t, a t ) o Experience new estimators besides the linear one.
39 Thanks for your attention!
40 References QL & SARSA: Machine learning-based stochastic control approaches for financial trading Bertsekas D.P. and Tsitsiklis J.N. (1996) "Neuro-dynamic programming", Athena Scientific. Bertoluzzo F. and Corazza M. (2014) "Reinforcement learning for automated financial trading: Basics and applications". In: Bassis S., Esposito A. and Morabito F.C. (Eds.): Recent Advances of Neural Network Models and Applications, Springer, Lo A. (2004) "The Adaptive Markets Hypothesis: Market efficiency from an evolutionary perspective", Journal of Portfolio Management, 30, Moody J., Saffel M. (2001) "Learning to trade via Direct Reinforcement", IEEE Transactions on Neural Network, 12,
Basics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
Elements of Reinforcement Learning
Elements of Reinforcement Learning Policy: way learning algorithm behaves (mapping from state to action) Reward function: Mapping of state action pair to reward or cost Value function: long term reward,
Reinforcement Learning with Function Approximation. Joseph Christian G. Noel
Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is
Reinforcement Learning. Introduction
Reinforcement Learning Introduction Reinforcement Learning Agent interacts and learns from a stochastic environment Science of sequential decision making Many faces of reinforcement learning Optimal control
Payments System Design Using Reinforcement Learning: A Progress Report
Payments System Design Using Reinforcement Learning: A Progress Report A. Desai 1 H. Du 1 R. Garratt 2 F. Rivadeneyra 1 1 Bank of Canada 2 University of California Santa Barbara 16th Payment and Settlement
Introduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
REINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
Financial Risk and Returns Prediction with Modular Networked Learning
arxiv:1806.05876v1 [cs.lg] 15 Jun 2018 Financial Risk and Returns Prediction with Modular Networked Learning Carlos Pedro Gonçalves June 18, 2018 University of Lisbon, Instituto Superior de Ciências Sociais
The convergence limit of the temporal difference learning
The convergence limit of the temporal difference learning Ryosuke Nomura the University of Tokyo September 3, 2013 1 Outline Reinforcement Learning Convergence limit Construction of the feature vector
Reinforcement learning
Reinforcement learning Based on [Kaelbling et al., 1996, Bertsekas, 2000] Bert Kappen Reinforcement learning Reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error
Prof. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be
REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while
MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
Dueling Network Architectures for Deep Reinforcement Learning (ICML 2016)
Dueling Network Architectures for Deep Reinforcement Learning (ICML 2016) Yoonho Lee Department of Computer Science and Engineering Pohang University of Science and Technology October 11, 2016 Outline
Reinforcement Learning
Reinforcement Learning Cyber Rodent Project Some slides from: David Silver, Radford Neal CSC411: Machine Learning and Data Mining, Winter 2017 Michael Guerzhoy 1 Reinforcement Learning Supervised learning:
Artificial Intelligence
Artificial Intelligence Dynamic Programming Marc Toussaint University of Stuttgart Winter 2018/19 Motivation: So far we focussed on tree search-like solvers for decision problems. There is a second important
Procedia Computer Science 00 (2011) 000 6
Procedia Computer Science (211) 6 Procedia Computer Science Complex Adaptive Systems, Volume 1 Cihan H. Dagli, Editor in Chief Conference Organized by Missouri University of Science and Technology 211-
Open Theoretical Questions in Reinforcement Learning
Open Theoretical Questions in Reinforcement Learning Richard S. Sutton AT&T Labs, Florham Park, NJ 07932, USA, sutton@research.att.com, www.cs.umass.edu/~rich Reinforcement learning (RL) concerns the problem
Introduction to Reinforcement Learning
CSCI-699: Advanced Topics in Deep Learning 01/16/2019 Nitin Kamra Spring 2019 Introduction to Reinforcement Learning 1 What is Reinforcement Learning? So far we have seen unsupervised and supervised learning.
Reinforcement learning an introduction
Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,
Reinforcement Learning. George Konidaris
Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom
Temporal difference learning
Temporal difference learning AI & Agents for IET Lecturer: S Luz http://www.scss.tcd.ie/~luzs/t/cs7032/ February 4, 2014 Recall background & assumptions Environment is a finite MDP (i.e. A and S are finite).
Artificial Intelligence & Sequential Decision Problems
Artificial Intelligence & Sequential Decision Problems (CIV6540 - Machine Learning for Civil Engineers) Professor: James-A. Goulet Département des génies civil, géologique et des mines Chapter 15 Goulet
Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016
Course 16:198:520: Introduction To Artificial Intelligence Lecture 13 Decision Making Abdeslam Boularias Wednesday, December 7, 2016 1 / 45 Overview We consider probabilistic temporal models where the
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
Reinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
Reinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina
Reinforcement Learning Introduction Introduction Unsupervised learning has no outcome (no feedback). Supervised learning has outcome so we know what to predict. Reinforcement learning is in between it
Machine Learning and Bayesian Inference. Unsupervised learning. Can we find regularity in data without the aid of labels?
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone extension 6372 Email: sbh11@cl.cam.ac.uk www.cl.cam.ac.uk/ sbh11/ Unsupervised learning Can we find regularity
INF 5860 Machine learning for image classification. Lecture 14: Reinforcement learning May 9, 2018
Machine learning for image classification Lecture 14: Reinforcement learning May 9, 2018 Page 3 Outline Motivation Introduction to reinforcement learning (RL) Value function based methods (Q-learning)
arxiv: v1 [cs.lg] 23 Oct 2017
![arxiv: v1 [cs.lg] 23 Oct 2017](/thumbs/83/88739636.jpg "arxiv: v1 [cs.lg] 23 Oct 2017")
Accelerated Reinforcement Learning K. Lakshmanan Department of Computer Science and Engineering Indian Institute of Technology (BHU), Varanasi, India Email: lakshmanank.cse@itbhu.ac.in arxiv:1710.08070v1
Decision Theory: Q-Learning
Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning
Variance Reduction for Policy Gradient Methods. March 13, 2017
Variance Reduction for Policy Gradient Methods March 13, 2017 Reward Shaping Reward Shaping Reward Shaping Reward shaping: r(s, a, s ) = r(s, a, s ) + γφ(s ) Φ(s) for arbitrary potential Φ Theorem: r admits
Reinforcement Learning as Classification Leveraging Modern Classifiers
Reinforcement Learning as Classification Leveraging Modern Classifiers Michail G. Lagoudakis and Ronald Parr Department of Computer Science Duke University Durham, NC 27708 Machine Learning Reductions
Marks. bonus points. } Assignment 1: Should be out this weekend. } Mid-term: Before the last lecture. } Mid-term deferred exam:
Marks } Assignment 1: Should be out this weekend } All are marked, I m trying to tally them and perhaps add bonus points } Mid-term: Before the last lecture } Mid-term deferred exam: } This Saturday, 9am-10.30am,
16.4 Multiattribute Utility Functions
285 Normalized utilities The scale of utilities reaches from the best possible prize u to the worst possible catastrophe u Normalized utilities use a scale with u = 0 and u = 1 Utilities of intermediate
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
Reinforcement Learning and Control
CS9 Lecture notes Andrew Ng Part XIII Reinforcement Learning and Control We now begin our study of reinforcement learning and adaptive control. In supervised learning, we saw algorithms that tried to make
ilstd: Eligibility Traces and Convergence Analysis
ilstd: Eligibility Traces and Convergence Analysis Alborz Geramifard Michael Bowling Martin Zinkevich Richard S. Sutton Department of Computing Science University of Alberta Edmonton, Alberta {alborz,bowling,maz,sutton}@cs.ualberta.ca
Evaluation of multi armed bandit algorithms and empirical algorithm
Acta Technica 62, No. 2B/2017, 639 656 c 2017 Institute of Thermomechanics CAS, v.v.i. Evaluation of multi armed bandit algorithms and empirical algorithm Zhang Hong 2,3, Cao Xiushan 1, Pu Qiumei 1,4 Abstract.
Reinforcement Learning (1)
Reinforcement Learning 1 Reinforcement Learning (1) Machine Learning 64-360, Part II Norman Hendrich University of Hamburg, Dept. of Informatics Vogt-Kölln-Str. 30, D-22527 Hamburg hendrich@informatik.uni-hamburg.de
Nonlinear Bivariate Comovements of Asset Prices: Theory and Tests
Nonlinear Bivariate Comovements of Asset Prices: Theory and Tests M. Corazza, A.G. Malliaris, E. Scalco Department of Applied Mathematics University Ca Foscari of Venice (Italy) Department of Economics
A Gentle Introduction to Reinforcement Learning
A Gentle Introduction to Reinforcement Learning Alexander Jung 2018 1 Introduction and Motivation Consider the cleaning robot Rumba which has to clean the office room B329. In order to keep things simple,
Reinforcement Learning
1 Reinforcement Learning Chris Watkins Department of Computer Science Royal Holloway, University of London July 27, 2015 2 Plan 1 Why reinforcement learning? Where does this theory come from? Markov decision
6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
Reinforcement Learning: the basics
Reinforcement Learning: the basics Olivier Sigaud Université Pierre et Marie Curie, PARIS 6 http://people.isir.upmc.fr/sigaud August 6, 2012 1 / 46 Introduction Action selection/planning Learning by trial-and-error
Lecture 4: Approximate dynamic programming
IEOR 800: Reinforcement learning By Shipra Agrawal Lecture 4: Approximate dynamic programming Deep Q Networks discussed in the last lecture are an instance of approximate dynamic programming. These are
Short Course: Multiagent Systems. Multiagent Systems. Lecture 1: Basics Agents Environments. Reinforcement Learning. This course is about:
Short Course: Multiagent Systems Lecture 1: Basics Agents Environments Reinforcement Learning Multiagent Systems This course is about: Agents: Sensing, reasoning, acting Multiagent Systems: Distributed
Reinforcement Learning and Deep Reinforcement Learning
Reinforcement Learning and Deep Reinforcement Learning Ashis Kumer Biswas, Ph.D. ashis.biswas@ucdenver.edu Deep Learning November 5, 2018 1 / 64 Outlines 1 Principles of Reinforcement Learning 2 The Q
An Adaptive Clustering Method for Model-free Reinforcement Learning
An Adaptive Clustering Method for Model-free Reinforcement Learning Andreas Matt and Georg Regensburger Institute of Mathematics University of Innsbruck, Austria {andreas.matt, georg.regensburger}@uibk.ac.at
problem. max Both k (0) and h (0) are given at time 0. (a) Write down the Hamilton-Jacobi-Bellman (HJB) Equation in the dynamic programming
1. Endogenous Growth with Human Capital Consider the following endogenous growth model with both physical capital (k (t)) and human capital (h (t)) in continuous time. The representative household solves
Q-learning. Tambet Matiisen
Q-learning Tambet Matiisen (based on chapter 11.3 of online book Artificial Intelligence, foundations of computational agents by David Poole and Alan Mackworth) Stochastic gradient descent Experience
Bayesian Active Learning With Basis Functions
Bayesian Active Learning With Basis Functions Ilya O. Ryzhov Warren B. Powell Operations Research and Financial Engineering Princeton University Princeton, NJ 08544, USA IEEE ADPRL April 13, 2011 1 / 29
1 Problem Formulation
Book Review Self-Learning Control of Finite Markov Chains by A. S. Poznyak, K. Najim, and E. Gómez-Ramírez Review by Benjamin Van Roy This book presents a collection of work on algorithms for learning
Reinforcement Learning Part 2
Reinforcement Learning Part 2 Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ From previous tutorial Reinforcement Learning Exploration No supervision Agent-Reward-Environment
CS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
Real Time Value Iteration and the State-Action Value Function
MS&E338 Reinforcement Learning Lecture 3-4/9/18 Real Time Value Iteration and the State-Action Value Function Lecturer: Ben Van Roy Scribe: Apoorva Sharma and Tong Mu 1 Review Last time we left off discussing
Lecture 7: Value Function Approximation
Lecture 7: Value Function Approximation Joseph Modayil Outline 1 Introduction 2 3 Batch Methods Introduction Large-Scale Reinforcement Learning Reinforcement learning can be used to solve large problems,
Reinforcement Learning
Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation
Learning Control Under Uncertainty: A Probabilistic Value-Iteration Approach
Learning Control Under Uncertainty: A Probabilistic Value-Iteration Approach B. Bischoff 1, D. Nguyen-Tuong 1,H.Markert 1 anda.knoll 2 1- Robert Bosch GmbH - Corporate Research Robert-Bosch-Str. 2, 71701
Reinforcement Learning. Machine Learning, Fall 2010
Reinforcement Learning Machine Learning, Fall 2010 1 Administrativia This week: finish RL, most likely start graphical models LA2: due on Thursday LA3: comes out on Thursday TA Office hours: Today 1:30-2:30
Actor-critic methods. Dialogue Systems Group, Cambridge University Engineering Department. February 21, 2017
Actor-critic methods Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department February 21, 2017 1 / 21 In this lecture... The actor-critic architecture Least-Squares Policy Iteration
CS 287: Advanced Robotics Fall Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study
CS 287: Advanced Robotics Fall 2009 Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study Pieter Abbeel UC Berkeley EECS Assignment #1 Roll-out: nice example paper: X.
Lecture 18: Reinforcement Learning Sanjeev Arora Elad Hazan
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 18: Reinforcement Learning Sanjeev Arora Elad Hazan Some slides borrowed from Peter Bodik and David Silver Course progress Learning
Reinforcement Learning
Reinforcement Learning Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ Task Grasp the green cup. Output: Sequence of controller actions Setup from Lenz et. al.
Internet Monetization
Internet Monetization March May, 2013 Discrete time Finite A decision process (MDP) is reward process with decisions. It models an environment in which all states are and time is divided into stages. Definition
Oblivious Equilibrium: A Mean Field Approximation for Large-Scale Dynamic Games
Oblivious Equilibrium: A Mean Field Approximation for Large-Scale Dynamic Games Gabriel Y. Weintraub, Lanier Benkard, and Benjamin Van Roy Stanford University {gweintra,lanierb,bvr}@stanford.edu Abstract
Reinforcement Learning
Reinforcement Learning March May, 2013 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha
Reinforcement Learning for NLP
Reinforcement Learning for NLP Advanced Machine Learning for NLP Jordan Boyd-Graber REINFORCEMENT OVERVIEW, POLICY GRADIENT Adapted from slides by David Silver, Pieter Abbeel, and John Schulman Advanced
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu
Lecture 8: Policy Gradient
Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve
Sequential decision making under uncertainty. Department of Computer Science, Czech Technical University in Prague
Sequential decision making under uncertainty Jiří Kléma Department of Computer Science, Czech Technical University in Prague https://cw.fel.cvut.cz/wiki/courses/b4b36zui/prednasky pagenda Previous lecture:
Partially observable Markov decision processes. Department of Computer Science, Czech Technical University in Prague
Partially observable Markov decision processes Jiří Kléma Department of Computer Science, Czech Technical University in Prague https://cw.fel.cvut.cz/wiki/courses/b4b36zui/prednasky pagenda Previous lecture:
Dual Temporal Difference Learning
Dual Temporal Difference Learning Min Yang Dept. of Computing Science University of Alberta Edmonton, Alberta Canada T6G 2E8 Yuxi Li Dept. of Computing Science University of Alberta Edmonton, Alberta Canada
Least-Squares Temporal Difference Learning based on Extreme Learning Machine
Least-Squares Temporal Difference Learning based on Extreme Learning Machine Pablo Escandell-Montero, José M. Martínez-Martínez, José D. Martín-Guerrero, Emilio Soria-Olivas, Juan Gómez-Sanchis IDAL, Intelligent
Reinforcement Learning and NLP
1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value
Decision Theory: Markov Decision Processes
Decision Theory: Markov Decision Processes CPSC 322 Lecture 33 March 31, 2006 Textbook 12.5 Decision Theory: Markov Decision Processes CPSC 322 Lecture 33, Slide 1 Lecture Overview Recap Rewards and Policies
Reinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
Optimism in the Face of Uncertainty Should be Refutable
Optimism in the Face of Uncertainty Should be Refutable Ronald ORTNER Montanuniversität Leoben Department Mathematik und Informationstechnolgie Franz-Josef-Strasse 18, 8700 Leoben, Austria, Phone number:
Dynamic stochastic game and macroeconomic equilibrium
Dynamic stochastic game and macroeconomic equilibrium Tianxiao Zheng SAIF 1. Introduction We have studied single agent problems. However, macro-economy consists of a large number of agents including individuals/households,
Lecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
Q-Learning in Continuous State Action Spaces
Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental
(a) Write down the Hamilton-Jacobi-Bellman (HJB) Equation in the dynamic programming
1. Government Purchases and Endogenous Growth Consider the following endogenous growth model with government purchases (G) in continuous time. Government purchases enhance production, and the production
Reinforcement Learning
Reinforcement Learning Model-Based Reinforcement Learning Model-based, PAC-MDP, sample complexity, exploration/exploitation, RMAX, E3, Bayes-optimal, Bayesian RL, model learning Vien Ngo MLR, University
Temporal Difference Learning & Policy Iteration
Temporal Difference Learning & Policy Iteration Advanced Topics in Reinforcement Learning Seminar WS 15/16 ±0 ±0 +1 by Tobias Joppen 03.11.2015 Fachbereich Informatik Knowledge Engineering Group Prof.
A Tour of Reinforcement Learning The View from Continuous Control. Benjamin Recht University of California, Berkeley
A Tour of Reinforcement Learning The View from Continuous Control Benjamin Recht University of California, Berkeley trustable, scalable, predictable Control Theory! Reinforcement Learning is the study
Reinforcement Learning in a Nutshell
Reinforcement Learning in a Nutshell V. Heidrich-Meisner 1, M. Lauer 2, C. Igel 1 and M. Riedmiller 2 1- Institut für Neuroinformatik, Ruhr-Universität Bochum, Germany 2- Neuroinformatics Group, University
Reinforcement Learning
Reinforcement Learning Lecture 6: RL algorithms 2.0 Alexandre Proutiere, Sadegh Talebi, Jungseul Ok KTH, The Royal Institute of Technology Objectives of this lecture Present and analyse two online algorithms
Deep Reinforcement Learning: Policy Gradients and Q-Learning
Deep Reinforcement Learning: Policy Gradients and Q-Learning John Schulman Bay Area Deep Learning School September 24, 2016 Introduction and Overview Aim of This Talk What is deep RL, and should I use
Wealth, Information Acquisition and Portfolio Choice: A Correction
Wealth, Information Acquisition and Portfolio Choice: A Correction Joel Peress INSEAD There is an error in our 2004 paper Wealth, Information Acquisition and Portfolio Choice. This note shows how to correct
Reinforcement Learning using Continuous Actions. Hado van Hasselt
Reinforcement Learning using Continuous Actions Hado van Hasselt 2005 Concluding thesis for Cognitive Artificial Intelligence University of Utrecht First supervisor: Dr. Marco A. Wiering, University of
15-780: ReinforcementLearning
15-780: ReinforcementLearning J. Zico Kolter March 2, 2016 1 Outline Challenge of RL Model-based methods Model-free methods Exploration and exploitation 2 Outline Challenge of RL Model-based methods Model-free
Sequential decision making under uncertainty. Department of Computer Science, Czech Technical University in Prague
Sequential decision making under uncertainty Jiří Kléma Department of Computer Science, Czech Technical University in Prague /doku.php/courses/a4b33zui/start pagenda Previous lecture: individual rational
Non-linear Supervised High Frequency Trading Strategies with Applications in US Equity Markets
Non-linear Supervised High Frequency Trading Strategies with Applications in US Equity Markets Nan Zhou, Wen Cheng, Ph.D. Associate, Quantitative Research, J.P. Morgan nan.zhou@jpmorgan.com The 4th Annual
In: Proc. BENELEARN-98, 8th Belgian-Dutch Conference on Machine Learning, pp 9-46, 998 Linear Quadratic Regulation using Reinforcement Learning Stephan ten Hagen? and Ben Krose Department of Mathematics,
A Correction. Joel Peress INSEAD. Abstract
Wealth, Information Acquisition and ortfolio Choice A Correction Joel eress INSEAD Abstract There is an error in my 2004 paper Wealth, Information Acquisition and ortfolio Choice. This note shows how to
Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]
![Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]](/thumbs/94/119187228.jpg "Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]")
Review: TD-Learning function TD-Learning(mdp) returns a policy Class #: Reinforcement Learning, II 8s S, U(s) =0 set start-state s s 0 choose action a, using -greedy policy based on U(s) U(s) U(s)+ [r
Markov Decision Processes and Solving Finite Problems. February 8, 2017
Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:
arxiv: v1 [cs.lg] 30 Dec 2016
![arxiv: v1 [cs.lg] 30 Dec 2016](/thumbs/96/127560530.jpg "arxiv: v1 [cs.lg] 30 Dec 2016")
Adaptive Least-Squares Temporal Difference Learning Timothy A. Mann and Hugo Penedones and Todd Hester Google DeepMind London, United Kingdom {kingtim, hugopen, toddhester}@google.com Shie Mannor Electrical
Human-level control through deep reinforcement. Liia Butler
Humanlevel control through deep reinforcement Liia Butler But first... A quote "The question of whether machines can think... is about as relevant as the question of whether submarines can swim" Edsger