REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning
|
|
- Abel Nelson
- 5 years ago
- Views:
Transcription
1 REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari REINFORCE (Williams) February 28, / 38
2 Outline Reinforcement Learning: A Quick Refresher 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
3 Reinforcement Learning: A Quick Refresher Reinforcement Learning Framework System modeled as Markov Decision Process: s S - state of the system a A agent s action p(s s, a) the dynamics of the system r : S A R the reward function (possibly stochastic) π(a s) - the agent s policy (also possibly stochastic) Ronen Tamari REINFORCE (Williams) February 28, / 38
4 Reinforcement Learning: A Quick Refresher Reinforcement Learning Objective Assuming episodic problem with horizon T, the reward of the episode is: T T r (s t, a t ) = r (t) t=1 Or in the case of discounted rewards, t=1 T d (t) r (t) t=1 where d is timestep-dependent weighting, often set to d (t) = γ t, γ (0, 1) Generally, goal is to learn policy optimizing the expected return: [ T ] E d (t) r (t) t=1 Ronen Tamari REINFORCE (Williams) February 28, / 38
5 Reinforcement Learning: A Quick Refresher Reinforcement Learning Methods Value Based- try to learn a value function satisfying Bellman Equation Q-Learning Actor-Critic Algorithms Policy Search- attempt to directly learn a policy maximizing rewards Gradient Based REINFORCE (also known as likelihood-ratio methods) Gradient-Free Simulated Annealing Cross-Entropy Search Ronen Tamari REINFORCE (Williams) February 28, / 38
6 Outline Policy Gradient Methods 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
7 Policy Gradient Methods Performance Measure Denote a trajectory by τ = {(a t, s t )} T t=1 Assume policy parameterized by K parameters: θ R K Performance measure defined as [ T ] J (θ) = E pθ (τ) r (t) θ = E pθ (τ) [r (τ) θ] t=1 Ronen Tamari REINFORCE (Williams) February 28, / 38
8 Policy Gradient Methods Policy Gradient Methods Update rule: where α t is the learning rate. θ t+1 = θ t + α t θ J θ=θt The main problem is obtaining a good estimator of the policy gradient θ J θ=θt Ronen Tamari REINFORCE (Williams) February 28, / 38
9 Outline REINFORCE Framework 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
10 REINFORCE Framework Episodic REINFORCE (Williams, R.J. (1992)) REINFORCE: REward Increment = Nonnegative Factor times Offset Reinforcement times Characteristic Eligibility. Or formally, according to Williams, a learning algorithm with a weight update of the form T w ij = α ij (r b ij ) e ij (t) w ij - Policy parameterization (think of case of θ = W being a N M weights matrix) α ij - Learning rate factor. r - Reward received at end of trial or after each timestep. Can be time dependent. b ij - Reinforcement baseline- conditionally independent of action y t given policy. e ij = ln g w ij - "Characteristic Eligibility"- where g = Pr (y = ξ w, x) t=1 Ronen Tamari REINFORCE (Williams) February 28, / 38
11 REINFORCE Framework Episodic REINFORCE (Williams, R.J. (1992)) From now on we will adhere to contemporary notation common today in the deep learning community, which is an adaptation to context of MDP type problems. T w ij = α ij (r b ij ) e ij (t) t=1 T T θ = α r t b θ log π θ (y t x t ) t=1 t=1 }{{} R θ - Policy parameterization (for example, weights of NN) α - Equivalent to Gradient Descent learning rate. θ log π θ (y t x t ) - gradient of log-likelihood of policy. Ronen Tamari REINFORCE (Williams) February 28, / 38
12 REINFORCE Framework Theorem: Episodic REINFORCE Theorem For any episodic REINFORCE algorithm, the inner product of E [ θ θ] and θ E [R θ] is non-negative. Furthermore, if α > 0 then this inner product is zero only when θ E [R θ] = 0. Also, if α ij is independent of i, j then α θ E [R θ] = E [ θ θ] We will show a formulation specifically for the MDP case: [ T ] α θ J (θ) = α θ E [(r (τ) θ] = E pθ (τ) α θ log π θ (a t s t ) (r (τ) b) θ t=0 Ronen Tamari REINFORCE (Williams) February 28, / 38
13 REINFORCE Framework Deriving the Policy Gradient Proof: J (θ) = E pθ (τ) [r (τ)] = T p θ (τ) r (τ) dτ Where p θ (τ) = p (s 0 ) T t=0 p (s t+1 s t, a t ) π θ (a t s t ) due to Markovity assumption. The gradient is: θ p θ (τ) =? θ J (θ) = θ p θ (τ) r (τ) dτ T Ronen Tamari REINFORCE (Williams) February 28, / 38
14 REINFORCE Framework Deriving the Policy Gradient (cont.) Use the "log-likelihood trick": θ log (p θ (τ)) = θp θ (τ) p θ (τ) To obtain: θ J (θ) = T θ p θ (τ) r (τ) dτ = T θ log (p θ (τ)) p θ (τ) r (τ) dτ = [ T ] = E pθ (τ) [ θ log (p θ (τ)) r (τ)] = E pθ (τ) θ log π θ (a t s t ) r (τ) t=0 Since only the policy π θ (a t s t ) is dependent on θ! Ronen Tamari REINFORCE (Williams) February 28, / 38
15 REINFORCE Framework Deriving the Policy Gradient (cont.) [ T ] E pθ (τ) θ log π θ (a t s t ) r (τ) = t=0 The expectation can be replaced by sample averages to give the form: θ J (θ) 1 M M T ) θ log π θ (a t s i t i R i i=1 t=0 Where i = 1,..., M are episodes where the agent is run with the current policy. Discount factor could also be incorporated into R i. (where reward is dependent on length of episode). Ronen Tamari REINFORCE (Williams) February 28, / 38
16 REINFORCE Framework Variance Reduction Using Baseline We are only computing the expected gradient, variance may be high, as demonstrated in following example: Assume baseline b = 0 in a scenario with a single reward r for all state-action pairs. The variance of the gradient grows cubically with length of the horizon T : [ ] T T Var T r ( θ log π θ (a t s t )) = T 2 r 2 Var [ θ log π θ (a t s t )] t=0 Subtracting a baseline b from the estimation can reduce variance: [ ] T Var (Tr b) ( θ log π θ (..)) = (Tr b) 2 T Var [ θ log π θ (..)] t=0 t=0 t=0 Ronen Tamari REINFORCE (Williams) February 28, / 38
17 REINFORCE Framework Variance Reduction Using Baseline This gives the form: θ J (θ) 1 M M T ) θ log π θ (a ( T ) t s i t i rt i b i=1 t=0 t =0 Ronen Tamari REINFORCE (Williams) February 28, / 38
18 REINFORCE Framework Variance Reduction Using Baseline Doesn t introduce bias to gradient estimate θ J (θ), since: T T = θ p θ (τ) [r (τ) b] dτ = We used T E pθ (τ) [ θ log (p θ (τ)) [r (τ) b]] = θ log (p θ (τ)) p θ (τ) [r (τ) b] dτ = T θ p θ (τ) r (τ) dτ b = θ J (θ) p θ (τ) dτ = 1 = T θ p θ (τ) dτ = 0 θ p θ (τ) dτ T } {{ } =0 Ronen Tamari REINFORCE (Williams) February 28, / 38
19 REINFORCE Framework Choosing Optimal Baseline Williams work doesn t detail how to choose the baseline, though this is significant in practice for lowering variance and thus achieving good convergence. Future works explore it in detail (see for example (Sutton ( 99) and Peters (08 ) in the Further Reading section ). Ronen Tamari REINFORCE (Williams) February 28, / 38
20 Outline REINFORCE Based Algorithm 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
21 Algorithm REINFORCE Based Algorithm Episodic REINFORCE Algorithm (from Peters, J., Schaal, S. (2008).) Input: Policy parameterization θ 1.Perform trials 1,..., M (until converged): } M { {(a } i - Given trial trajectories t, st) i, r i T t t=0 i=1 - Obtain uncorrected gradient estimate g = 1 M T ( ( )) ( M i=1 t=0 θ log π θ a i t st i ) T t =0 d trt i - Estimate optimal baseline per element b = (b 1,..., b h,...b K ): b h = 1 M M i=1 ( T ( ( )) ) t=0 θ log π θh a i t st i 2 ( T ) t =0 d t r t i ( T ) 2 t=0 ( θ h log π θ (at s i t)) i - Subtract baseline: g = 1 M T ( ( )) ( M i=1 t=0 θ log π θ a i t st i ) T t =0 d trt i b - Return g = θ J (θ) (One step of SGD) Ronen Tamari REINFORCE (Williams) February 28, / 38
22 REINFORCE Based Algorithm Algorithm Convergence Original work doesn t contain analysis of asymptotic properties of REINFORCE algorithms. Empirical simulations show episodic REINFORCE algorithms to be slower due to delayed feedback (for case where reward only assigned at end of episode). Susceptibility to convergence to local optima. Baseline choice significantly affects convergence properties. Related work has shown that for this type of( method, ) theoretical convergence to true gradient is at rate of O 1 m where m denotes the number of episodes (Monte Carlo based analysis). Ronen Tamari REINFORCE (Williams) February 28, / 38
23 Outline REINFORCE in Neural Networks 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
24 REINFORCE in Neural Networks REINFORCE in Neural Networks REINFORCE originally designed (also) for use in Neural Networks and RNNs. Agent represented by stochastic output unit\s behind which are deterministic hidden units. For example, consider an RNN with a softmax output of probabilities according to which the action a t is chosen. Hidden units of RNN θ t are policy parameterization. Derivation of REINFORCE algorithm using "unfolding through time" of RNN: Weights frozen for duration of episode and updated at the end. Naturally compatible with backpropagation- θ log π θ (a t s t ) is the gradient of the corresponding RNN evaluated at timestep t. Ronen Tamari REINFORCE (Williams) February 28, / 38
25 REINFORCE in Neural Networks Example: Recurrent Attention Model (Mnih et al., 2015) Policy parameterized by RNN At each step 2 types of actions (l t glimpse location and a t classification) controlled by 2 sub-networks. Goal is to learn stochastic policy π ((l t, a t ) s 1:t ; θ) maximizing rewards. Ronen Tamari REINFORCE (Williams) February 28, / 38
26 REINFORCE in Neural Networks Example: Recurrent Attention Model (Mnih et al., 2015) Trajectory given by s 1:t = x 1, l 1, a 1,..., x t 1, l t 1, a t 1, x t At each step 2 types of actions (l t glimpse location and a t classification) controlled by 2 sub-networks. Reward R = T t=1 r t where r T = 1 for correct classification and 0 otherwise. Ronen Tamari REINFORCE (Williams) February 28, / 38
27 REINFORCE in Neural Networks Example: Recurrent Attention Model (Mnih et al., 2015) Gradient estimate of the form: θ J (θ) 1 M M T i=1 t=1 ( )) ( ( T θ log π θ at s i 1:t i rt i b t t =1 ) Ronen Tamari REINFORCE (Williams) February 28, / 38
28 REINFORCE in Neural Networks Example: Recurrent Attention Model (Mnih et al., 2015) - Dynamic Environment Same approach used to train agent to play simple game in dynamic environment. Ronen Tamari REINFORCE (Williams) February 28, / 38
29 Outline Comparison with Related Algorithms 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
30 Comparison with Related Algorithms Q-Learning: A reminder* The value function: V π [ (s) = E a π( s) r (s, a) + γv π ( s )] s p( s,a) The Q-Function is the value of taking action a at state s: [ ( Q (s, a) = r (s, a) + γe s p( s,a) V s )] The optimal value satisfies the Bellman Equation: V (s) = max E [ π a π( s) r (s, a) + γv ( s )] s p( s,a) *Slides based on Noga Zaslavsky s presentation Ronen Tamari REINFORCE (Williams) February 28, / 38
31 Comparison with Related Algorithms Q-Learning: A reminder* If we know V then an optimal policy is to decide deterministically a (s) = arg max Q (s, a) a Learning Q means learning an optimal policy *Slides based on Noga Zaslavsky s presentation Ronen Tamari REINFORCE (Williams) February 28, / 38
32 Comparison with Related Algorithms Q-Learning Elegant mathematical characterization and converges to optimal Q-function for MDP problems. Model free - only uses Q-function and not system dynamics p(s t+1 s t, a t ) Two main issues complicate the picture: What if problem isn t Markovian (often arises in multi-agent settings)? Assume it is but partially observable... (POMDP) Partial observability makes the learning problem much harder. Continuous or very large action-state space Still possible to learn with function approximation (Deepmind Atari-style games). max operator in Bellman equation makes function approximation difficult. Sensitive to noise and in practice hard to train. Ronen Tamari REINFORCE (Williams) February 28, / 38
33 Comparison with Related Algorithms Q-Learning vs. REINFORCE Q-Learning better adapted to classic observable Markovian setting, REINFORCE-based learning more relevant the further problem is from that setting (hidden and continuous states). Ronen Tamari REINFORCE (Williams) February 28, / 38
34 Outline Conclusions 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
35 Conclusions Conclusions Policy parameterization suited to complex dynamic systems. Slow convergence. Limited theoretical analysis. Integrates naturally with Deep Learning, RNNs. General approach, serves as basis for more effective algorithms. Not coupled as tightly as Q-Learning with Markovity assumptions. Ronen Tamari REINFORCE (Williams) February 28, / 38
36 Further Reading Conclusions Sutton, R. S., Mcallester, D., Singh, S., Mansour, Y. (1999). Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems 12, Peters, J., Schaal, S. (2008). Reinforcement learning of motor skills with policy gradients. Neural Networks, 21(4), Kober, J., Bagnell, J. A., Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11), Shalev-shwartz, S., Ben-zrihem, N., Cohen, A., Shashua, A. (n.d.). Long-term Planning by Short-term Prediction, Ronen Tamari REINFORCE (Williams) February 28, / 38
37 Outline 1 Reinforcement Learning: A Quick Refresher 2 Policy Gradient Methods 3 REINFORCE Framework 4 REINFORCE Based Algorithm 5 REINFORCE in Neural Networks 6 Comparison with Related Algorithms 7 Conclusions Ronen Tamari REINFORCE (Williams) February 28, / 38
38 Thank You Ronen Tamari REINFORCE (Williams) February 28, / 38
Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
More informationPolicy Gradient Methods. February 13, 2017
Policy Gradient Methods February 13, 2017 Policy Optimization Problems maximize E π [expression] π Fixed-horizon episodic: T 1 Average-cost: lim T 1 T r t T 1 r t Infinite-horizon discounted: γt r t Variable-length
More informationIntroduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
More informationReinforcement Learning and NLP
1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value
More informationDeep Reinforcement Learning: Policy Gradients and Q-Learning
Deep Reinforcement Learning: Policy Gradients and Q-Learning John Schulman Bay Area Deep Learning School September 24, 2016 Introduction and Overview Aim of This Talk What is deep RL, and should I use
More informationCSC321 Lecture 22: Q-Learning
CSC321 Lecture 22: Q-Learning Roger Grosse Roger Grosse CSC321 Lecture 22: Q-Learning 1 / 21 Overview Second of 3 lectures on reinforcement learning Last time: policy gradient (e.g. REINFORCE) Optimize
More informationREINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
More informationMarkov Decision Processes and Solving Finite Problems. February 8, 2017
Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:
More informationLecture 9: Policy Gradient II 1
Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John
More informationLecture 8: Policy Gradient
Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve
More informationLecture 18: Reinforcement Learning Sanjeev Arora Elad Hazan
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 18: Reinforcement Learning Sanjeev Arora Elad Hazan Some slides borrowed from Peter Bodik and David Silver Course progress Learning
More informationUsing Gaussian Processes for Variance Reduction in Policy Gradient Algorithms *
Proceedings of the 8 th International Conference on Applied Informatics Eger, Hungary, January 27 30, 2010. Vol. 1. pp. 87 94. Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms
More informationReinforcement Learning via Policy Optimization
Reinforcement Learning via Policy Optimization Hanxiao Liu November 22, 2017 1 / 27 Reinforcement Learning Policy a π(s) 2 / 27 Example - Mario 3 / 27 Example - ChatBot 4 / 27 Applications - Video Games
More informationCS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
More informationDialogue management: Parametric approaches to policy optimisation. Dialogue Systems Group, Cambridge University Engineering Department
Dialogue management: Parametric approaches to policy optimisation Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 30 Dialogue optimisation as a reinforcement learning
More informationINF 5860 Machine learning for image classification. Lecture 14: Reinforcement learning May 9, 2018
Machine learning for image classification Lecture 14: Reinforcement learning May 9, 2018 Page 3 Outline Motivation Introduction to reinforcement learning (RL) Value function based methods (Q-learning)
More information1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5
Table of contents 1 Introduction 2 2 Markov Decision Processes 2 3 Future Cumulative Reward 3 4 Q-Learning 4 4.1 The Q-value.............................................. 4 4.2 The Temporal Difference.......................................
More informationProf. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be
REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while
More informationELEC-E8119 Robotics: Manipulation, Decision Making and Learning Policy gradient approaches. Ville Kyrki
ELEC-E8119 Robotics: Manipulation, Decision Making and Learning Policy gradient approaches Ville Kyrki 9.10.2017 Today Direct policy learning via policy gradient. Learning goals Understand basis and limitations
More informationReinforcement learning an introduction
Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,
More informationReinforcement Learning
Reinforcement Learning Policy gradients Daniel Hennes 26.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Policy based reinforcement learning So far we approximated the action value
More information(Deep) Reinforcement Learning
Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 17 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015
More informationReinforcement Learning
Reinforcement Learning Policy Search: Actor-Critic and Gradient Policy search Mario Martin CS-UPC May 28, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 28, 2018 / 63 Goal of this lecture So far
More informationLecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free
More informationTrust Region Policy Optimization
Trust Region Policy Optimization Yixin Lin Duke University yixin.lin@duke.edu March 28, 2017 Yixin Lin (Duke) TRPO March 28, 2017 1 / 21 Overview 1 Preliminaries Markov Decision Processes Policy iteration
More informationIntroduction of Reinforcement Learning
Introduction of Reinforcement Learning Deep Reinforcement Learning Reference Textbook: Reinforcement Learning: An Introduction http://incompleteideas.net/sutton/book/the-book.html Lectures of David Silver
More information6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
More informationLecture 9: Policy Gradient II (Post lecture) 2
Lecture 9: Policy Gradient II (Post lecture) 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver
More informationQ-Learning in Continuous State Action Spaces
Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental
More informationMachine Learning I Continuous Reinforcement Learning
Machine Learning I Continuous Reinforcement Learning Thomas Rückstieß Technische Universität München January 7/8, 2010 RL Problem Statement (reminder) state s t+1 ENVIRONMENT reward r t+1 new step r t
More informationBalancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu
More informationMARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
More informationIntroduction to Reinforcement Learning
CSCI-699: Advanced Topics in Deep Learning 01/16/2019 Nitin Kamra Spring 2019 Introduction to Reinforcement Learning 1 What is Reinforcement Learning? So far we have seen unsupervised and supervised learning.
More informationLecture 8: Policy Gradient I 2
Lecture 8: Policy Gradient I 2 Emma Brunskill CS234 Reinforcement Learning. Winter 2018 Additional reading: Sutton and Barto 2018 Chp. 13 2 With many slides from or derived from David Silver and John Schulman
More informationDeep Reinforcement Learning. Scratching the surface
Deep Reinforcement Learning Scratching the surface Deep Reinforcement Learning Scenario of Reinforcement Learning Observation State Agent Action Change the environment Don t do that Reward Environment
More informationLecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
More informationLecture 23: Reinforcement Learning
Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
More informationReinforcement Learning for NLP
Reinforcement Learning for NLP Advanced Machine Learning for NLP Jordan Boyd-Graber REINFORCEMENT OVERVIEW, POLICY GRADIENT Adapted from slides by David Silver, Pieter Abbeel, and John Schulman Advanced
More informationCMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING
More informationReinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
More informationVariance Reduction for Policy Gradient Methods. March 13, 2017
Variance Reduction for Policy Gradient Methods March 13, 2017 Reward Shaping Reward Shaping Reward Shaping Reward shaping: r(s, a, s ) = r(s, a, s ) + γφ(s ) Φ(s) for arbitrary potential Φ Theorem: r admits
More informationToday s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes
Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks
More informationReinforcement Learning and Deep Reinforcement Learning
Reinforcement Learning and Deep Reinforcement Learning Ashis Kumer Biswas, Ph.D. ashis.biswas@ucdenver.edu Deep Learning November 5, 2018 1 / 64 Outlines 1 Principles of Reinforcement Learning 2 The Q
More information15-889e Policy Search: Gradient Methods Emma Brunskill. All slides from David Silver (with EB adding minor modificafons), unless otherwise noted
15-889e Policy Search: Gradient Methods Emma Brunskill All slides from David Silver (with EB adding minor modificafons), unless otherwise noted Outline 1 Introduction 2 Finite Difference Policy Gradient
More informationLecture 3: Markov Decision Processes
Lecture 3: Markov Decision Processes Joseph Modayil 1 Markov Processes 2 Markov Reward Processes 3 Markov Decision Processes 4 Extensions to MDPs Markov Processes Introduction Introduction to MDPs Markov
More informationChapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationMS&E338 Reinforcement Learning Lecture 1 - April 2, Introduction
MS&E338 Reinforcement Learning Lecture 1 - April 2, 2018 Introduction Lecturer: Ben Van Roy Scribe: Gabriel Maher 1 Reinforcement Learning Introduction In reinforcement learning (RL) we consider an agent
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning II Daniel Hennes 11.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns
More informationSpatial Transformation
Spatial Transformation Presented by Liqun Chen June 30, 2017 1 Overview 2 Spatial Transformer Networks 3 STN experiments 4 Recurrent Models of Visual Attention (RAM) 5 Recurrent Models of Visual Attention
More informationReinforcement Learning. Introduction
Reinforcement Learning Introduction Reinforcement Learning Agent interacts and learns from a stochastic environment Science of sequential decision making Many faces of reinforcement learning Optimal control
More informationTutorial on Policy Gradient Methods. Jan Peters
Tutorial on Policy Gradient Methods Jan Peters Outline 1. Reinforcement Learning 2. Finite Difference vs Likelihood-Ratio Policy Gradients 3. Likelihood-Ratio Policy Gradients 4. Conclusion General Setup
More informationReinforcement Learning
Reinforcement Learning Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ Task Grasp the green cup. Output: Sequence of controller actions Setup from Lenz et. al.
More informationReinforcement Learning as Variational Inference: Two Recent Approaches
Reinforcement Learning as Variational Inference: Two Recent Approaches Rohith Kuditipudi Duke University 11 August 2017 Outline 1 Background 2 Stein Variational Policy Gradient 3 Soft Q-Learning 4 Closing
More informationReinforcement Learning. Spring 2018 Defining MDPs, Planning
Reinforcement Learning Spring 2018 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
More informationCS 598 Statistical Reinforcement Learning. Nan Jiang
CS 598 Statistical Reinforcement Learning Nan Jiang Overview What s this course about? A grad-level seminar course on theory of RL 3 What s this course about? A grad-level seminar course on theory of RL
More informationReinforcement Learning II. George Konidaris
Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2017 Reinforcement Learning π : S A max R = t=0 t r t MDPs Agent interacts with an environment At each time t: Receives sensor signal Executes
More informationDeep Reinforcement Learning via Policy Optimization
Deep Reinforcement Learning via Policy Optimization John Schulman July 3, 2017 Introduction Deep Reinforcement Learning: What to Learn? Policies (select next action) Deep Reinforcement Learning: What to
More informationReinforcement Learning II. George Konidaris
Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2018 Reinforcement Learning π : S A max R = t=0 t r t MDPs Agent interacts with an environment At each time t: Receives sensor signal Executes
More informationDiscrete Latent Variable Models
Discrete Latent Variable Models Stefano Ermon, Aditya Grover Stanford University Lecture 14 Stefano Ermon, Aditya Grover (AI Lab) Deep Generative Models Lecture 14 1 / 29 Summary Major themes in the course
More informationReinforcement Learning Part 2
Reinforcement Learning Part 2 Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ From previous tutorial Reinforcement Learning Exploration No supervision Agent-Reward-Environment
More informationApproximate Q-Learning. Dan Weld / University of Washington
Approximate Q-Learning Dan Weld / University of Washington [Many slides taken from Dan Klein and Pieter Abbeel / CS188 Intro to AI at UC Berkeley materials available at http://ai.berkeley.edu.] Q Learning
More informationMachine Learning. Reinforcement learning. Hamid Beigy. Sharif University of Technology. Fall 1396
Machine Learning Reinforcement learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Machine Learning Fall 1396 1 / 32 Table of contents 1 Introduction
More informationNotes on Reinforcement Learning
1 Introduction Notes on Reinforcement Learning Paulo Eduardo Rauber 2014 Reinforcement learning is the study of agents that act in an environment with the goal of maximizing cumulative reward signals.
More informationApproximation Methods in Reinforcement Learning
2018 CS420, Machine Learning, Lecture 12 Approximation Methods in Reinforcement Learning Weinan Zhang Shanghai Jiao Tong University http://wnzhang.net http://wnzhang.net/teaching/cs420/index.html Reinforcement
More informationReinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina
Reinforcement Learning Introduction Introduction Unsupervised learning has no outcome (no feedback). Supervised learning has outcome so we know what to predict. Reinforcement learning is in between it
More informationReinforcement Learning as Classification Leveraging Modern Classifiers
Reinforcement Learning as Classification Leveraging Modern Classifiers Michail G. Lagoudakis and Ronald Parr Department of Computer Science Duke University Durham, NC 27708 Machine Learning Reductions
More informationOpen Theoretical Questions in Reinforcement Learning
Open Theoretical Questions in Reinforcement Learning Richard S. Sutton AT&T Labs, Florham Park, NJ 07932, USA, sutton@research.att.com, www.cs.umass.edu/~rich Reinforcement learning (RL) concerns the problem
More informationActor-critic methods. Dialogue Systems Group, Cambridge University Engineering Department. February 21, 2017
Actor-critic methods Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department February 21, 2017 1 / 21 In this lecture... The actor-critic architecture Least-Squares Policy Iteration
More informationMachine Learning and Bayesian Inference. Unsupervised learning. Can we find regularity in data without the aid of labels?
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone extension 6372 Email: sbh11@cl.cam.ac.uk www.cl.cam.ac.uk/ sbh11/ Unsupervised learning Can we find regularity
More informationReinforcement Learning
Reinforcement Learning March May, 2013 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha
More informationDeep reinforcement learning. Dialogue Systems Group, Cambridge University Engineering Department
Deep reinforcement learning Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 25 In this lecture... Introduction to deep reinforcement learning Value-based Deep RL Deep
More informationBasics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
More informationChapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationReinforcement Learning
Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns Forward and backward view Function
More informationDeep Reinforcement Learning SISL. Jeremy Morton (jmorton2) November 7, Stanford Intelligent Systems Laboratory
Deep Reinforcement Learning Jeremy Morton (jmorton2) November 7, 2016 SISL Stanford Intelligent Systems Laboratory Overview 2 1 Motivation 2 Neural Networks 3 Deep Reinforcement Learning 4 Deep Learning
More informationDecision Theory: Q-Learning
Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning
More informationLearning Tetris. 1 Tetris. February 3, 2009
Learning Tetris Matt Zucker Andrew Maas February 3, 2009 1 Tetris The Tetris game has been used as a benchmark for Machine Learning tasks because its large state space (over 2 200 cell configurations are
More informationReinforcement Learning: the basics
Reinforcement Learning: the basics Olivier Sigaud Université Pierre et Marie Curie, PARIS 6 http://people.isir.upmc.fr/sigaud August 6, 2012 1 / 46 Introduction Action selection/planning Learning by trial-and-error
More informationReinforcement Learning and Control
CS9 Lecture notes Andrew Ng Part XIII Reinforcement Learning and Control We now begin our study of reinforcement learning and adaptive control. In supervised learning, we saw algorithms that tried to make
More informationReinforcement Learning for Continuous. Action using Stochastic Gradient Ascent. Hajime KIMURA, Shigenobu KOBAYASHI JAPAN
Reinforcement Learning for Continuous Action using Stochastic Gradient Ascent Hajime KIMURA, Shigenobu KOBAYASHI Tokyo Institute of Technology, 4259 Nagatsuda, Midori-ku Yokohama 226-852 JAPAN Abstract:
More informationActive Policy Iteration: Efficient Exploration through Active Learning for Value Function Approximation in Reinforcement Learning
Active Policy Iteration: fficient xploration through Active Learning for Value Function Approximation in Reinforcement Learning Takayuki Akiyama, Hirotaka Hachiya, and Masashi Sugiyama Department of Computer
More informationHidden Markov Models (HMM) and Support Vector Machine (SVM)
Hidden Markov Models (HMM) and Support Vector Machine (SVM) Professor Joongheon Kim School of Computer Science and Engineering, Chung-Ang University, Seoul, Republic of Korea 1 Hidden Markov Models (HMM)
More informationPolicy Gradient Reinforcement Learning for Robotics
Policy Gradient Reinforcement Learning for Robotics Michael C. Koval mkoval@cs.rutgers.edu Michael L. Littman mlittman@cs.rutgers.edu May 9, 211 1 Introduction Learning in an environment with a continuous
More informationMachine Learning I Reinforcement Learning
Machine Learning I Reinforcement Learning Thomas Rückstieß Technische Universität München December 17/18, 2009 Literature Book: Reinforcement Learning: An Introduction Sutton & Barto (free online version:
More informationReinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of
More informationReinforcement Learning
Reinforcement Learning Ron Parr CompSci 7 Department of Computer Science Duke University With thanks to Kris Hauser for some content RL Highlights Everybody likes to learn from experience Use ML techniques
More informationReinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
More informationAdministration. CSCI567 Machine Learning (Fall 2018) Outline. Outline. HW5 is available, due on 11/18. Practice final will also be available soon.
Administration CSCI567 Machine Learning Fall 2018 Prof. Haipeng Luo U of Southern California Nov 7, 2018 HW5 is available, due on 11/18. Practice final will also be available soon. Remaining weeks: 11/14,
More informationDeep Reinforcement Learning
Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 50 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015
More informationReading Response: Due Wednesday. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1
Reading Response: Due Wednesday R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Another Example Get to the top of the hill as quickly as possible. reward = 1 for each step where
More informationTemporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI
Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning
More informationAdvanced Policy Gradient Methods: Natural Gradient, TRPO, and More. March 8, 2017
Advanced Policy Gradient Methods: Natural Gradient, TRPO, and More March 8, 2017 Defining a Loss Function for RL Let η(π) denote the expected return of π [ ] η(π) = E s0 ρ 0,a t π( s t) γ t r t We collect
More informationReinforcement Learning: An Introduction
Introduction Betreuer: Freek Stulp Hauptseminar Intelligente Autonome Systeme (WiSe 04/05) Forschungs- und Lehreinheit Informatik IX Technische Universität München November 24, 2004 Introduction What is
More informationThis question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
More informationLecture 3: The Reinforcement Learning Problem
Lecture 3: The Reinforcement Learning Problem Objectives of this lecture: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationDeep Reinforcement Learning. STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017
Deep Reinforcement Learning STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017 Outline Introduction to Reinforcement Learning AlphaGo (Deep RL for Computer Go)
More informationINTRODUCTION TO EVOLUTION STRATEGY ALGORITHMS. James Gleeson Eric Langlois William Saunders
INTRODUCTION TO EVOLUTION STRATEGY ALGORITHMS James Gleeson Eric Langlois William Saunders REINFORCEMENT LEARNING CHALLENGES f(θ) is a discrete function of theta How do we get a gradient θ f? Credit assignment
More informationMDP Preliminaries. Nan Jiang. February 10, 2019
MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process
More informationMarkov Decision Processes
Markov Decision Processes Noel Welsh 11 November 2010 Noel Welsh () Markov Decision Processes 11 November 2010 1 / 30 Annoucements Applicant visitor day seeks robot demonstrators for exciting half hour
More information