Off-Policy Actor-Critic
|
|
|
- Amanda Willis
- 5 years ago
- Views:
Transcription
1 Off-Policy Actor-Critic Ludovic Trottier Laval University July Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
2 Table of Contents 1 Reinforcement Learning Theory 2 Off-PAC Algorithm 3 Experimentations 4 Discussion 5 References Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
3 1 Reinforcement Learning Theory 2 Off-PAC Algorithm 3 Experimentations 4 Discussion 5 References Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
4 On vs Off Policy On-policy learning: learns the value function of the policy that is used to make decisions. (SARSA) Q(s, a) Q(s, a) + α [ r + γq(s, a ) Q(s, a) ] Off-policy learning: learns the value function of a policy different of the one used to make decisions. (Q-learning) [ ] Q(s, a) Q(s, a) + α r + γ max Q(s, a ) Q(s, a) a Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
![Ref. [3] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July 25 2012 5 / 34 Actor-Critic vs Off-Policy Actor-Critic (Off-PAC) Actor-Critic evaluates improves](/docs-images/86/94607800/images/5-1.jpg "critic behavior - actor - behavior data generated with behavior Off-PAC evaluates improves critic target - actor - target data generated with behavior Figure: Actor-Critic")
5 Ref. [3] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34 Actor-Critic vs Off-Policy Actor-Critic (Off-PAC) Actor-Critic evaluates improves critic behavior - actor - behavior data generated with behavior Off-PAC evaluates improves critic target - actor - target data generated with behavior Figure: Actor-Critic Method
6 Value Function Approximation Because of a large state-space, the value function need to be approximated. Q(s, a) = s S Pr(s s, a)[r(s, a, s ) + γv (s )] Off-policy + value function approximation fluctuation. Based on the analysis of [2], it is better to use linear approximation : ˆV (s) = v T x s, v i and x s i R e.g. tile coding binary features x s Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
![Mountain Car Tile Coding Simulation S = [ 1.5, 0.5] [ 0.07, 0.](/docs-images/86/94607800/images/7-0.jpg "07] A = { 1, 0, 1} Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic")
7 Mountain Car Tile Coding Simulation S = [ 1.5, 0.5] [ 0.07, 0.07] A = { 1, 0, 1} Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
8 Stochastic Gradient Descent Given an exemple (x t, y t ) X and an objective function Q(w, x, y) Normal gradient descent w t+1 = w t + α Q(w, x t, y t ) w w t+1 = w t + α 1 n n i=1 Q(w, x i, y i ) w Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
9 1 Reinforcement Learning Theory 2 Off-PAC Algorithm 3 Experimentations 4 Discussion 5 References Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
10 Motivations 3 problems of off-policy action-value methods : 1 Target policy is deterministic 2 Policy improvement is problematic for large action space 3 Small changes in policy evaluation large changes in policy improvement Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
11 Problem The policy is a function parametrize by a weight vector u: π u : A S [0, 1] Objective function to maximise [4] : J(u) = lim Pr(s t = s s 0, b)]v (s) with V (s) = t s S[ π(a s)q(s, a) a A Figure: Chain Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
12 Desciption of the Critic Evaluating an approximation of the true value function V (s) ˆV (s) = v T x s, v i and x s i R in an incremental way using gradient descent. Objective function to minimize : MSPBE(v) = ˆV ΠT ˆV 2 D Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
13 MSPBE Objective function Given a vector function f : f 2 D = s S[ lim t Pr(s t = s s 0, b)]f 2 (s) Define V π (s) = E [ t=0 γt R t+1 S 0 = s, π] and we get : V π (s) = a π(s, a) s Pr(s, a, s )[R(s, a) + γv π (s )] = s Pr(s, a, s )[R(s, a) + γv π (s )] = s Pr(s, a, s )R(s, a) + s Pr(s, a, s )γv π (s ) V π = E[R] + γpv π = TV π Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
14 MSPBE Objective function For a vector function f, we define the linear projection as Πf := f θ = θ T φ such that θ = arg min fθ f 2 D θ { } { } = arg min θ [ lim Pr(s t = s s 0, b)](f t θ f ) 2 (s) s S Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
15 Geometrical Interpretation Ref. [2] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
16 Exemple Iteration 0 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
17 Exemple Iteration 1 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
18 Exemple Iteration 1 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
19 Exemple Iteration 1 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
20 Exemple Iteration 2 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
21 Eligibility Traces R (1) t = r t+1 + γv t (s t+1 ) state R (2) t = r t+1 + γr t+2 + γ 2 V t (s t+2 ) action R (n) t = r t+1 + γr t γ n V t (s t+n ) Ref. [3] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
22 Eligibility Traces The return R t can be rewritten as any average of steps. e.g. R avg t = 1 2 R(2) t R(4) t Ref. [3] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
23 Eligibility Traces The λ-return used with TD methods is another way to define R avg R avg t := R λ t = (1 λ) n=1 λ n 1 R (n) t, 0 λ 1 t. Ref. [3] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
24 Off-Policy Policy-gradient Theorem Objective function to maximise : u J is too difficult to estimate. J(u) = s S[ lim t Pr(s t = s s 0, b)]v (s) We approximate the gradient with g(u). u J(u) g(u) = lim Pr(s t = s s 0, b)] t s S[ u π(a s)q(s, a) a A (product rule + approximation) Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
25 Desciption of the Actor Uses stochastic gradient descent on g(u) : [ ] g(u) = E u π(a s)q(s, a) s lim Pr( ) t a We can add a random function of the state to the equations because of [4] u π(s, a) = 0; Intuition : u π(a s) and Q(s, a) are orthogonal. a Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
26 Desciption of the Actor Rewriting the terms [ ] g(u) = E ρ(s, a)ψ(s, a)q(s, a) s lim Pr( ), a b( s) t where ρ(s, a) = π(a s) b(a s) and ψ(s, a) = uπ(a s) π(a s) We further approximate Q(s, a) with the off-policy λ-return R λ t R λ t = r t+1 + (1 λ)γ ˆV (s t ) + λγρ(s, a)r λ t+1 Forward view Backward view Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
27 1 Reinforcement Learning Theory 2 Off-PAC Algorithm 3 Experimentations 4 Discussion 5 References Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
28 Parameters 4 methods : Q(λ) Greedy-GQ Softmax-GQ Off-PAC b( s) = uniform distribution γ = 0.99 # steps = 5000, with performance testing at each 20 episodes. Gibbs distribution for target policy : π(a s) = tile coding feature extraction. Parameters sweep exp{ut φ s,a} a exp{ut φ s,a} Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
29 Results Off-PAC is successful Ref. [1] Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
30 1 Reinforcement Learning Theory 2 Off-PAC Algorithm 3 Experimentations 4 Discussion 5 References Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
31 Discussion Why using approximation (value function + policy) in small state/action space problems? Linear approximation of value function is very naive? Uniform behavior policy may cause problem for Q(λ). Is a stochastic policy really necessary? What is the complexity? Is Bellman operator T defined with deterministic policy, but applied to stochastic one?. Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
32 Conclusion 1 Off-Policy vs On-Policy methods 2 Actor-Critic for on and off-policy 3 Description of the Critic (value-function approximation, eligibility trace, stochastic gradient descent,gtd(λ)) 4 Description of the Actor (approximation of gradient, forward backward view, objective function) 5 Results 6 (Possible) problems Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
33 1 Reinforcement Learning Theory 2 Off-PAC Algorithm 3 Experimentations 4 Discussion 5 References Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
34 References Thomas Degris, Martha White, and Richard S. Sutton. Off-policy actor-critic. CoRR, abs/ , Hamid Reza Maei. Gradient Temporal-Difference Learning Algorithms. PhD thesis, University of Alberta, Fall Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, USA, 1st edition, Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In NIPS, pages , Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July / 34
PART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.
Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE
Reinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo MLR, University of Stuttgart
Reinforcement Learning: the basics
Reinforcement Learning: the basics Olivier Sigaud Université Pierre et Marie Curie, PARIS 6 http://people.isir.upmc.fr/sigaud August 6, 2012 1 / 46 Introduction Action selection/planning Learning by trial-and-error
Chapter 7: Eligibility Traces. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1
Chapter 7: Eligibility Traces R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Midterm Mean = 77.33 Median = 82 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
Off-Policy Actor-Critic
Thomas Degris thomas.degris@inria.fr Flowers Team, INRIA, Talence, ENSTA-ParisTech, Paris, France Martha White whitem@cs.ualberta.ca Richard S. Sutton sutton@cs.ualberta.ca RLAI Laboratory, Department
Temporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI
Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning
Lecture 8: Policy Gradient
Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve
Policy Gradient Methods. February 13, 2017
Policy Gradient Methods February 13, 2017 Policy Optimization Problems maximize E π [expression] π Fixed-horizon episodic: T 1 Average-cost: lim T 1 T r t T 1 r t Infinite-horizon discounted: γt r t Variable-length
Temporal difference learning
Temporal difference learning AI & Agents for IET Lecturer: S Luz http://www.scss.tcd.ie/~luzs/t/cs7032/ February 4, 2014 Recall background & assumptions Environment is a finite MDP (i.e. A and S are finite).
Reinforcement Learning
Reinforcement Learning Function Approximation Continuous state/action space, mean-square error, gradient temporal difference learning, least-square temporal difference, least squares policy iteration Vien
Reinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of
Approximation Methods in Reinforcement Learning
2018 CS420, Machine Learning, Lecture 12 Approximation Methods in Reinforcement Learning Weinan Zhang Shanghai Jiao Tong University http://wnzhang.net http://wnzhang.net/teaching/cs420/index.html Reinforcement
Off-Policy Actor-Critic
homas Degris thomas.degris@inria.fr Flowers eam, INRIA, alence, ENSA-Parisech, Paris, France Martha White whitem@cs.ualberta.ca Richard S. Sutton sutton@cs.ualberta.ca RLAI Laboratory, Department of Computing
Lecture 3: Markov Decision Processes
Lecture 3: Markov Decision Processes Joseph Modayil 1 Markov Processes 2 Markov Reward Processes 3 Markov Decision Processes 4 Extensions to MDPs Markov Processes Introduction Introduction to MDPs Markov
Machine Learning. Reinforcement learning. Hamid Beigy. Sharif University of Technology. Fall 1396
Machine Learning Reinforcement learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Machine Learning Fall 1396 1 / 32 Table of contents 1 Introduction
Notes on Reinforcement Learning
1 Introduction Notes on Reinforcement Learning Paulo Eduardo Rauber 2014 Reinforcement learning is the study of agents that act in an environment with the goal of maximizing cumulative reward signals.
Reinforcement Learning with Function Approximation. Joseph Christian G. Noel
Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
Reinforcement Learning and Deep Reinforcement Learning
Reinforcement Learning and Deep Reinforcement Learning Ashis Kumer Biswas, Ph.D. ashis.biswas@ucdenver.edu Deep Learning November 5, 2018 1 / 64 Outlines 1 Principles of Reinforcement Learning 2 The Q
Actor-critic methods. Dialogue Systems Group, Cambridge University Engineering Department. February 21, 2017
Actor-critic methods Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department February 21, 2017 1 / 21 In this lecture... The actor-critic architecture Least-Squares Policy Iteration
(Deep) Reinforcement Learning
Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 17 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015
Reinforcement Learning
Reinforcement Learning RL in continuous MDPs March April, 2015 Large/Continuous MDPs Large/Continuous state space Tabular representation cannot be used Large/Continuous action space Maximization over action
Open Theoretical Questions in Reinforcement Learning
Open Theoretical Questions in Reinforcement Learning Richard S. Sutton AT&T Labs, Florham Park, NJ 07932, USA, sutton@research.att.com, www.cs.umass.edu/~rich Reinforcement learning (RL) concerns the problem
6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
Dialogue management: Parametric approaches to policy optimisation. Dialogue Systems Group, Cambridge University Engineering Department
Dialogue management: Parametric approaches to policy optimisation Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 30 Dialogue optimisation as a reinforcement learning
Least squares temporal difference learning
Least squares temporal difference learning TD(λ) Good properties of TD Easy to implement, traces achieve the forward view Linear complexity = fast Combines easily with linear function approximation Outperforms
arxiv: v1 [cs.ai] 5 Nov 2017
![arxiv: v1 [cs.ai] 5 Nov 2017](/thumbs/96/129180284.jpg "arxiv: v1 [cs.ai] 5 Nov 2017")
arxiv:1711.01569v1 [cs.ai] 5 Nov 2017 Markus Dumke Department of Statistics Ludwig-Maximilians-Universität München markus.dumke@campus.lmu.de Abstract Temporal-difference (TD) learning is an important
Linear Least-squares Dyna-style Planning
Linear Least-squares Dyna-style Planning Hengshuai Yao Department of Computing Science University of Alberta Edmonton, AB, Canada T6G2E8 hengshua@cs.ualberta.ca Abstract World model is very important for
Reinforcement Learning In Continuous Time and Space
Reinforcement Learning In Continuous Time and Space presentation of paper by Kenji Doya Leszek Rybicki lrybicki@mat.umk.pl 18.07.2008 Leszek Rybicki lrybicki@mat.umk.pl Reinforcement Learning In Continuous
15-889e Policy Search: Gradient Methods Emma Brunskill. All slides from David Silver (with EB adding minor modificafons), unless otherwise noted
15-889e Policy Search: Gradient Methods Emma Brunskill All slides from David Silver (with EB adding minor modificafons), unless otherwise noted Outline 1 Introduction 2 Finite Difference Policy Gradient
, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
Reinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
PART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.
Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE
Olivier Sigaud. September 21, 2012
Supervised and Reinforcement Learning Tools for Motor Learning Models Olivier Sigaud Université Pierre et Marie Curie - Paris 6 September 21, 2012 1 / 64 Introduction Who is speaking? 2 / 64 Introduction
Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation
Fast Gradient-Descent Methods for Temporal-Difference Learning with Linear Function Approximation Rich Sutton, University of Alberta Hamid Maei, University of Alberta Doina Precup, McGill University Shalabh
Dual Temporal Difference Learning
Dual Temporal Difference Learning Min Yang Dept. of Computing Science University of Alberta Edmonton, Alberta Canada T6G 2E8 Yuxi Li Dept. of Computing Science University of Alberta Edmonton, Alberta Canada
Introduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
Reinforcement Learning
Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns Forward and backward view Function
Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
Reinforcement Learning Part 2
Reinforcement Learning Part 2 Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ From previous tutorial Reinforcement Learning Exploration No supervision Agent-Reward-Environment
Lecture 7: Value Function Approximation
Lecture 7: Value Function Approximation Joseph Modayil Outline 1 Introduction 2 3 Batch Methods Introduction Large-Scale Reinforcement Learning Reinforcement learning can be used to solve large problems,
Q-Learning in Continuous State Action Spaces
Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental
INF 5860 Machine learning for image classification. Lecture 14: Reinforcement learning May 9, 2018
Machine learning for image classification Lecture 14: Reinforcement learning May 9, 2018 Page 3 Outline Motivation Introduction to reinforcement learning (RL) Value function based methods (Q-learning)
The convergence limit of the temporal difference learning
The convergence limit of the temporal difference learning Ryosuke Nomura the University of Tokyo September 3, 2013 1 Outline Reinforcement Learning Convergence limit Construction of the feature vector
Markov Decision Processes and Solving Finite Problems. February 8, 2017
Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:
Lecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
Reinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina
Reinforcement Learning Introduction Introduction Unsupervised learning has no outcome (no feedback). Supervised learning has outcome so we know what to predict. Reinforcement learning is in between it
Replacing eligibility trace for action-value learning with function approximation
Replacing eligibility trace for action-value learning with function approximation Kary FRÄMLING Helsinki University of Technology PL 5500, FI-02015 TKK - Finland Abstract. The eligibility trace is one
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari
Emphatic Temporal-Difference Learning
Emphatic Temporal-Difference Learning A Rupam Mahmood Huizhen Yu Martha White Richard S Sutton Reinforcement Learning and Artificial Intelligence Laboratory Department of Computing Science, University
Temporal Difference Learning & Policy Iteration
Temporal Difference Learning & Policy Iteration Advanced Topics in Reinforcement Learning Seminar WS 15/16 ±0 ±0 +1 by Tobias Joppen 03.11.2015 Fachbereich Informatik Knowledge Engineering Group Prof.
CS599 Lecture 2 Function Approximation in RL
CS599 Lecture 2 Function Approximation in RL Look at how experience with a limited part of the state set be used to produce good behavior over a much larger part. Overview of function approximation (FA)
Internet Monetization
Internet Monetization March May, 2013 Discrete time Finite A decision process (MDP) is reward process with decisions. It models an environment in which all states are and time is divided into stages. Definition
Reinforcement Learning
Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation
REINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
Reinforcement Learning. Spring 2018 Defining MDPs, Planning
Reinforcement Learning Spring 2018 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
Reinforcement Learning
Reinforcement Learning Policy gradients Daniel Hennes 26.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Policy based reinforcement learning So far we approximated the action value
Reinforcement Learning. Machine Learning, Fall 2010
Reinforcement Learning Machine Learning, Fall 2010 1 Administrativia This week: finish RL, most likely start graphical models LA2: due on Thursday LA3: comes out on Thursday TA Office hours: Today 1:30-2:30
Reinforcement Learning for NLP
Reinforcement Learning for NLP Advanced Machine Learning for NLP Jordan Boyd-Graber REINFORCEMENT OVERVIEW, POLICY GRADIENT Adapted from slides by David Silver, Pieter Abbeel, and John Schulman Advanced
Artificial Intelligence
Artificial Intelligence Dynamic Programming Marc Toussaint University of Stuttgart Winter 2018/19 Motivation: So far we focussed on tree search-like solvers for decision problems. There is a second important
Deep reinforcement learning. Dialogue Systems Group, Cambridge University Engineering Department
Deep reinforcement learning Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 25 In this lecture... Introduction to deep reinforcement learning Value-based Deep RL Deep
Reinforcement Learning
Reinforcement Learning Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ Task Grasp the green cup. Output: Sequence of controller actions Setup from Lenz et. al.
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu
MDP Preliminaries. Nan Jiang. February 10, 2019
MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process
Reinforcement Learning. George Konidaris
Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom
Reinforcement Learning. Summer 2017 Defining MDPs, Planning
Reinforcement Learning Summer 2017 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
Lecture 9: Policy Gradient II (Post lecture) 2
Lecture 9: Policy Gradient II (Post lecture) 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver
An online kernel-based clustering approach for value function approximation
An online kernel-based clustering approach for value function approximation N. Tziortziotis and K. Blekas Department of Computer Science, University of Ioannina P.O.Box 1186, Ioannina 45110 - Greece {ntziorzi,kblekas}@cs.uoi.gr
Lecture 4: Approximate dynamic programming
IEOR 800: Reinforcement learning By Shipra Agrawal Lecture 4: Approximate dynamic programming Deep Q Networks discussed in the last lecture are an instance of approximate dynamic programming. These are
ilstd: Eligibility Traces and Convergence Analysis
ilstd: Eligibility Traces and Convergence Analysis Alborz Geramifard Michael Bowling Martin Zinkevich Richard S. Sutton Department of Computing Science University of Alberta Edmonton, Alberta {alborz,bowling,maz,sutton}@cs.ualberta.ca
ROB 537: Learning-Based Control. Announcements: Project background due Today. HW 3 Due on 10/30 Midterm Exam on 11/6.
ROB 537: Learning-Based Control Week 5, Lecture 1 Policy Gradient, Eligibility Traces, Transfer Learning (MaC Taylor Announcements: Project background due Today HW 3 Due on 10/30 Midterm Exam on 11/6 Reading:
arxiv: v1 [cs.lg] 20 Sep 2018
![arxiv: v1 [cs.lg] 20 Sep 2018](/thumbs/86/94886615.jpg "arxiv: v1 [cs.lg] 20 Sep 2018")
Predicting Periodicity with Temporal Difference Learning Kristopher De Asis, Brendan Bennett, Richard S. Sutton Reinforcement Learning and Artificial Intelligence Laboratory, University of Alberta {kldeasis,
Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms *
Proceedings of the 8 th International Conference on Applied Informatics Eger, Hungary, January 27 30, 2010. Vol. 1. pp. 87 94. Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms
ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control
ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control Lecturer: Nikolay Atanasov: natanasov@ucsd.edu Teaching Assistants: Tianyu Wang: tiw161@eng.ucsd.edu Yongxi Lu: yol070@eng.ucsd.edu
Reinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
Reinforcement Learning II
Reinforcement Learning II Andrea Bonarini Artificial Intelligence and Robotics Lab Department of Electronics and Information Politecnico di Milano E-mail: bonarini@elet.polimi.it URL:http://www.dei.polimi.it/people/bonarini
CS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
Forward Actor-Critic for Nonlinear Function Approximation in Reinforcement Learning
Forward Actor-Critic for Nonlinear Function Approximation in Reinforcement Learning Vivek Veeriah Dept. of Computing Science University of Alberta Edmonton, Canada vivekveeriah@ualberta.ca Harm van Seijen
Implicit Incremental Natural Actor Critic
Implicit Incremental Natural Actor Critic Ryo Iwaki and Minoru Asada Osaka University, -1, Yamadaoka, Suita city, Osaka, Japan {ryo.iwaki,asada}@ams.eng.osaka-u.ac.jp Abstract. The natural policy gradient
Lecture 9: Policy Gradient II 1
Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning II Daniel Hennes 11.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns
Trust Region Policy Optimization
Trust Region Policy Optimization Yixin Lin Duke University yixin.lin@duke.edu March 28, 2017 Yixin Lin (Duke) TRPO March 28, 2017 1 / 21 Overview 1 Preliminaries Markov Decision Processes Policy iteration
Generalization and Function Approximation
Generalization and Function Approximation 0 Generalization and Function Approximation Suggested reading: Chapter 8 in R. S. Sutton, A. G. Barto: Reinforcement Learning: An Introduction MIT Press, 1998.
MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
The Book: Where we are and where we re going. CSE 190: Reinforcement Learning: An Introduction. Chapter 7: Eligibility Traces. Simple Monte Carlo
CSE 190: Reinforcement Learning: An Introduction Chapter 7: Eligibility races Acknowledgment: A good number of these slides are cribbed from Rich Sutton he Book: Where we are and where we re going Part
Machine Learning I Reinforcement Learning
Machine Learning I Reinforcement Learning Thomas Rückstieß Technische Universität München December 17/18, 2009 Literature Book: Reinforcement Learning: An Introduction Sutton & Barto (free online version:
Variance Reduction for Policy Gradient Methods. March 13, 2017
Variance Reduction for Policy Gradient Methods March 13, 2017 Reward Shaping Reward Shaping Reward Shaping Reward shaping: r(s, a, s ) = r(s, a, s ) + γφ(s ) Φ(s) for arbitrary potential Φ Theorem: r admits
Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]
![Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]](/thumbs/94/119187228.jpg "Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]")
Review: TD-Learning function TD-Learning(mdp) returns a policy Class #: Reinforcement Learning, II 8s S, U(s) =0 set start-state s s 0 choose action a, using -greedy policy based on U(s) U(s) U(s)+ [r
On and Off-Policy Relational Reinforcement Learning
On and Off-Policy Relational Reinforcement Learning Christophe Rodrigues, Pierre Gérard, and Céline Rouveirol LIPN, UMR CNRS 73, Institut Galilée - Université Paris-Nord first.last@lipn.univ-paris13.fr
Deep Reinforcement Learning: Policy Gradients and Q-Learning
Deep Reinforcement Learning: Policy Gradients and Q-Learning John Schulman Bay Area Deep Learning School September 24, 2016 Introduction and Overview Aim of This Talk What is deep RL, and should I use
Notes on Tabular Methods
Notes on Tabular ethods Nan Jiang September 28, 208 Overview of the methods. Tabular certainty-equivalence Certainty-equivalence is a model-based RL algorithm, that is, it first estimates an DP model from
https://www.youtube.com/watch?v=ymvi1l746us Eligibility traces Chapter 12, plus some extra stuff! Like n-step methods, but better! Eligibility traces A mechanism that allow TD, Sarsa and Q-learning to
Reinforcement Learning and NLP
1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value
Convergence of reinforcement learning algorithms and acceleration of learning
PHYSICAL REVIEW E 67, 026706 2003 Convergence of reinforcement learning algorithms and acceleration of learning A. Potapov* and M. K. Ali Department of Physics, The University of Lethbridge, 4401 University
Reinforcement Learning
Reinforcement Learning March May, 2013 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha
CSC321 Lecture 22: Q-Learning
CSC321 Lecture 22: Q-Learning Roger Grosse Roger Grosse CSC321 Lecture 22: Q-Learning 1 / 21 Overview Second of 3 lectures on reinforcement learning Last time: policy gradient (e.g. REINFORCE) Optimize
Reinforcement learning an introduction
Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,
6 Basic Convergence Results for RL Algorithms
Learning in Complex Systems Spring 2011 Lecture Notes Nahum Shimkin 6 Basic Convergence Results for RL Algorithms We establish here some asymptotic convergence results for the basic RL algorithms, by showing
CS 287: Advanced Robotics Fall Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study
CS 287: Advanced Robotics Fall 2009 Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study Pieter Abbeel UC Berkeley EECS Assignment #1 Roll-out: nice example paper: X.
Relative Entropy Policy Search
Relative Entropy Policy Search Jan Peters, Katharina Mülling, Yasemin Altün Max Planck Institute for Biological Cybernetics, Spemannstr. 38, 72076 Tübingen, Germany {jrpeters,muelling,altun}@tuebingen.mpg.de