Reinforcement learning an introduction
|
|
- Nigel Harmon
- 6 years ago
- Views:
Transcription
1 Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013
2 Reinforcement Learning What is it? Learning from interaction Learning about, from, and while interacting with an external environment Learning what to do how to map situations to actions so as to maximize a numerical reward signal
3 Reinforcement Learning Key features? Learner is not told which actions to take Trial-and-Error search The need to explore and exploit Possibility of delayed reward Sacrifice short-term gains for greater long-term gains Considers the whole problem of a goal-directed agent interacting with an uncertain environment
4 Supervised versus Unsupervised Training Info = desired (target) outputs Training Info = evaluations ( rewards / penalties ) Inputs Supervised Learning System Outputs Inputs states Reinforcement Learning System Outputs actions Error = (target output actual output) Objective: get as much reward as possible
5 The Agent-Environment Interface Agent and environment interact at discrete time steps :... Agent observes state at step t : produces gets resulting reward : and resulting next state : s t action at step t : a t t t+ 1 R r s a t+ 1 r t +1 s t +1 a t +1 s t A( s t S ) t = 01,,2, r t +2 s t +2 t +2 a r t +3 st a t +3
6 Learning how to behave Policy at step t, π π ( s, a) = probability that t t : a mapping from states to action probabilities a t = a when s t = s Reinforcement learning methods specify how the agent changes its policy as a result of experience. Roughly, the agent s goal is to get as much reward as it can over the long run.
7 Goals and Rewards A goal should specify what we want to achieve, not how we want to achieve it. Suppose the sequence of rewards after step t is : What rt + 1, rt + 2, rt + 3,... do we want to maximize? In general, we want to maximize the expected return, { R }, for each step t. E t
8 What s the objective? Episodic tasks: interaction breaks naturally into episodes, e.g., plays of a game. R t = rt rt + 2 r T Immediate reward Long term reward Continuing tasks: interaction does not have natural episodes. k= 0 R t = r t +1 + γ r t +2 + γ 2 r t +3 +L = γ k r t +k +1, where γ, 0 γ 1, is the discount rate.
9 Goals and Rewards : example Pole-balancing: Reward = +1 every time step before falling down, Reward = 0 when pole fall down OR Reward = 0 every time step before falling down, Reward = -1 when pole falls down. and take γ < 1
10 The Objective Maximize Longterm Total Discounted Reward Find a policy π : s S a A (could be stochastic) that maximizes the value (expected future reward) of each s : { } V π (s) = E r t +1 + γ r t +2 + γ 2 r t s t = s,π { } = E{ r t +1 } + γ E r t +2 + γ r t s t +1 = s,π rewards
11 Optimal Value Functions and Policies There exist optimal value functions V* and corresponding optimal policies p* which maximize the values for all states simultaneously V * (s) = max p V p (s) s V ( s) = maxq( s, a) a
12 Optimal Value Functions and Policies There exist optimal value functions V* and corresponding optimal policies p* which maximize the values for all states simultaneously V * (s) = max p V p (s) s V ( s) = maxq( s, a) a
13 Optimal Value Functions and Policies V*(s) π*(s)
14 Optimal Value Functions and Policies GOAL +10 Find shortest path to goal Rewards can be delayed: only receive reward when reaching goal Unknown environment Consequences of an action can only be discovered by trying it and observing the result (new state s', reward r) START States: Location Actions: Move N,E,S,W Transitions: move 1 step in selected direction (except at borders) Rewards: +10 if next loc == goal, 0 else
15 Q-Learning, basic RL algorithm One - step Q - learning : Q( s t,a t ) Q( s t,a t ) + α r t +1 + γ max [ Q( s t +1,a) Q( s t,a t )] a
16 A very simple MDP example r =4 d 0.3 r =10 4 b r =2 a 1 c r =1 1 r = if transition probabilities are known use DP to solve system of equations
17 RL is model free r =4 d 0.3 r=10 4 b r=2 a 1 c r =1 1 r =
18 RL is model free d r=10 4 b 2 r=2 a 1 c 5 r =1 r =5 3 r =4 6
19 Q-learning Q(1,d)=0 r =4 d Q(2,b)= 0 b r=10 4 Q(1,a)=0 2 a r =2 1 c 5 r=1 3 r =5 Q(2,c)=0 Q( s t,a t ) Q( s t,a t ) + α r t +1 + γ max a α=0.1, γ=0.9 [ Q( s t +1,a) Q( s t,a t )] 6
20 Q-learning Q(1,d)=0 r =4 d Q(2,b)= 0 b r=10 4 Q(1,a)= a r =2 1 c 5 r=1 r =5 3 Q(2,c)=0 α=0.1, γ=0.9 6 ( ( ) ) Q ( Qs ( t 1,a t ) + α r t +1 + γ max Q sq t,a1,a t QQ1,a 1,a [ ] a [ % ] [ Q s2, t +1 a,a % ) ( s t,a t )] a ( ) Q( 1,a ) ( ( ) ) Q0 ( 1,a ) + [ 1+ α r t 0.9max{0,0} +1 + γ ( 0] ) Q( 1,a ) Q 2, a % a %
21 Q-learning Q(1,d)=0 r =4 d Q(2,b)= 01 r=10 4 Q(1,a)=0.1 2 b a r =2 1 c 5 r=1 r =5 3 Q(2,c)=0 α=0.1, γ=0.9 6 ( ( ) ) Q Q( s( t 2,b,a t ) + α r t +1 + γ max Q sq t,a2,b t Q 2,b [ Q s t +1,a) ( s t,a t )] a % ( 4, a %) Q( 2,b) ] Q 4, a" % a" ( ) Q & ( 2,b & )' ( ) Q0 ( + 2,b 0.1) # $ 10 +α$ + % r0.9max{0,...,0} t+1 +γ 0
22 Q-learning Q * (1,d)=4 r =4 d Q * (2,b)= 4.4 r =10 b 4 Q * (1,a)=1.9 2 a r =2 1 c 5 r =1 r =5 3 Q * (2,c)=5 6
23 Q-learning: optimal policy Q * (1,d)=4 r =4 d Q * (2,b)= r =10 4 b Q * (1,a)= r =2 a 1 c r =1 1 r = Q * (2,c)=5 1 6
24 Role of γ γ < 1 expresses that reward in the near future is more important than reward in the long term But also bounds the total expected reward γ.1+γ 2.1+γ γ.2 +γ 2.2 +γ = 2(1+γ.1+γ 2.1+γ )
25 Backup scheme DP s t r t +1 s t +1 T T T T T T T T T T
26 Backup sheme RL s t s t +1 r t +1 T T T T T T T T T T T T T T T T T T T T
27 Bootstraps and Samples Bootstrapping: update involves an estimate MC does not bootstrap DP bootstraps RL bootstraps Sampling: update does not involve an expected value MC samples DP does not sample RL samples MC Monte Carlo, RL Reinforcement Learning, DP Dynamic Programming
28 Model-free RL taxonomy Value Based (Critic only): o Learn Value Function o Policy is implicit (e.g. Greedy ) Policy Based (Actor only): o Explicitly store Policy o Directly update Policy (e.g. using gradient, evolution,...) Actor-Critic: o Learn Policy o Learn Value function o Update policy using Value Function Value Based Q-learning, SARSA Actor Critic Policy Based Policy gradient
29 Q-Learning versus SARSA One - step Q - learning : Q( s t,a t ) Q( s t,a t ) + α r t +1 + γ max [ Q( s t +1,a) Q( s t,a t )] a One - step QSARSA - learning : Q( s t,a t ) Q( s t,a t ) + α r t +1 + γ max [ Q( s t +1,a ) Q( s t,a t )] a With a the actual selected action
30 Real life applications Games Robotics Mechatronics Planning and scheduling Tutoring Routing in networks Self driving cars AlphaGO.
31 Offline learning Ok, you've learned a value function, How do you pick actions? Q» Q * Greedy Action Selection: Always select the action that looks best: p(s) = arg maxq(s, a) a
32 Online learning Try to perform well during learning, but still converge to the optimal action Balance exploration versus exploitation N- armbandits : minimise regret. Re gret = t r a * t ( ) r i i=1
33 Online learning Try to perform well during learning, but still converge to the optimal action Balance exploration versus exploitation N- armbandits : minimise regret. Regret = t r a * t ( ) r i i=1 Actual reward received Expected total reward for optimal action
34 Online learning Try to perform well during learning, but still converge to the optimal action Balance exploration versus exploitation N- armbandits : minimise regret. Re gret = t r a * t ( ) r i i=1 Total reward received Expected total reward for optimal action
35 The n-armed Bandit Problem a 1 a 2 a 3. a n
36 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values
37 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values
38 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values Trial1 4
39 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values Trial1 4 Trial2 7
40 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values Trial1 4 Trial2 7 Trial3 4
41 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values Trial1 4 Trial2 7 Trial3 4. Trialn 6
42 The n-armed Bandit Problem a 1 a 2 a 3. a n Expected Values Trial1 4 Trial2 7 Trial3 4. Trialn 6 Trialn+1????
43 Commonly used exploration techniques in RL e-greedy action selection: a t = { a t * with probability 1 ε random action with probability ε Boltzmann distribution action selection: τ very large -> random action selection τ very small -> greedy action selection 43
44 10-Armed Testbed n = 10 possible actions Each Q * (a) is chosen randomly from a normal distribution: each r t 1000 plays is also normal: repeat the whole thing 2000 times and average the results
45 e-greedy Methods on the 10-Armed Testbed
46 Optimistic Initial Values All methods so far depend on Q 0 (a), i.e., they are biased. Suppose instead we initialize the action values optimistically, i.e., on the 10-armed testbed, use Q 0 (a) = 5 for all a
47 Some convergence issues Most RL algorithms are proven to converge in Markovian environments But RL can also be applied to non-markovian environments, provided careful exploration The environment can be Non-Markovian to the RL learner due to : - limited sensors (POMDP) - discretisation of state space - use function approximators - other agents in environment
48 The Markov Property A state should summarize past sensations so as to retain all essential information, which is the information needed in order to behave optimally, i.e., it should have the Markov Property: { } = Pr s t +1 = " s,r t +1 = r s t,a t,r t,s t 1,a t 1,,r 1,s 0,a 0 for all " s, r, and histories s t,a t,r t,s t 1,a t 1,,r 1,s 0,a 0. Pr{ s t +1 = s ",r t +1 = r s t,a t } Knowing how you got is this state is not giving you extra information
49 Partially observable decision problem POMDP
50 Non-makovian due to discretisation Position Velocity Angle Angular velocity
51 Less Non-makovian due to better function approximation Example of fuzzy covering Position Velocity Angle Angular velocity [Nowé, 94]
52 Non-makovian due to other agents I R S A I R S A
53 Some convergence issues Q-learning in guaranteed to converge in a Markovian setting Tsitsiklis J.N. Asynchronous Stochastic Approximation and Q-learning. Machine Learning, Vol. 16: , 1994.
54 Proof by Tsitsiklis On the convergence of Q-learning Stochastic approximation : q i ( t +1) q i ( t) + α i ( t) F i q i ( t) [ ( ) q i ( t) + w i ( t) ] ( q ) q =,..., 1 q n Q(s,a) Learning factor α i 0 t 0 t ( t) α 2 i = ( t) < Contraction mapping F ( q(t) ) Q * < q(t) Q* Noise term q vector, but with possibly outdated components q i i i ( t) = ( q 1 ( τ 1 ( t) ),...,q n ( τ n ( t) )), i with 0 τ j ( t) t
55 Proof by Tsitsiklis, cont. On the convergence of Q-learning One - step Q - learning : Q( s t,a t ) Q( s t,a t ) + α r t +1 + γ max [ Q( s t +1,a) Q( s t,a t )] a Can be written as : Q( s t,a t ) Q s t,a t & '( ( ) + α E r t +1 + γ max [ Q( s t +1,a)] Q ( s,a t t) ) + + r t +1 + γ max * a a ( Q( s t +1,a)) E r t +1 + γ max [ Q( s t +1,a)] a, -. / 01
56 Proof by Tsitsiklis, cont. Relating Q-learning to stochastic approximation Q( s t,a t ) Q s t,a t & '( ( ) + α E r t +1 + γ max [ Q( s t +1,a)] Q ( s,a t t) ) + + r t +1 + γ max * a a [ Q( s t +1,a)] ( Q( s t +1,a)) E r t +1 + γ max a, -. / 01 i th component Can vary in time Contraction mapping Bellman operator Noise term Q i ( t +1) Q i ( t) + α i ( t) F i Q i ( t) [ ( ) Q i ( t) + w i ( t) ]
57 Proof by Tsitsiklis, cont. Stochastic approximation, as a vector q i F i F i + noise t q j
58 Wrap up Reinforcement learner told what to do, not how to do it. Proven to converge in Markovian environments. The more non-markovian, the more careful the exploration must be. When to use RL? When no model of the environment is available or too difficult to obtain or not flexible. However domain knowledge can be easily incorporated V(s) and Q(s,a) often expressed by function approximation(e.g.neural networks, fuzzy coverings)
59 Extensions for practical applications Representations of states Discrete versus continuous (tile coding, fuzzy, NN, radial basis functions) Fixed versus adaptive discretisation Take advantage of asynchronous updates Propagate interesting information more quickly: Prioritized sweeping, eligibility traces Incorporate domain knowledge Initialise policy Steer exploration Combine with model information (planning)
60 Planning Two uses of real experience: w model learning: to improve the model w direct RL: to directly improve the value function and policy Improving value function and/or policy via a model is sometimes called indirect RL or model-based R L or planning.
61 RL COMO Multi-agent reinforcement learning Multi-type ant systems Distributed scheduling Distributed control Multi-criteria 61
62 More on RL? Visit: ai.vub.ac.be Thanks for your attention!
Prof. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be
REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while
More informationCS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
More informationIntroduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
More informationReinforcement Learning. George Konidaris
Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom
More informationToday s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes
Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks
More informationReinforcement Learning
Reinforcement Learning 1 Reinforcement Learning Mainly based on Reinforcement Learning An Introduction by Richard Sutton and Andrew Barto Slides are mainly based on the course material provided by the
More informationMachine Learning. Reinforcement learning. Hamid Beigy. Sharif University of Technology. Fall 1396
Machine Learning Reinforcement learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Machine Learning Fall 1396 1 / 32 Table of contents 1 Introduction
More informationBasics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
More informationBalancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm
Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu
More informationReinforcement Learning II
Reinforcement Learning II Andrea Bonarini Artificial Intelligence and Robotics Lab Department of Electronics and Information Politecnico di Milano E-mail: bonarini@elet.polimi.it URL:http://www.dei.polimi.it/people/bonarini
More informationReinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
More informationARTIFICIAL INTELLIGENCE. Reinforcement learning
INFOB2KI 2018-2019 Utrecht University The Netherlands ARTIFICIAL INTELLIGENCE Reinforcement learning Lecturer: Silja Renooij These slides are part of the INFOB2KI Course Notes available from www.cs.uu.nl/docs/vakken/b2ki/schema.html
More informationMachine Learning I Reinforcement Learning
Machine Learning I Reinforcement Learning Thomas Rückstieß Technische Universität München December 17/18, 2009 Literature Book: Reinforcement Learning: An Introduction Sutton & Barto (free online version:
More informationAutonomous Helicopter Flight via Reinforcement Learning
Autonomous Helicopter Flight via Reinforcement Learning Authors: Andrew Y. Ng, H. Jin Kim, Michael I. Jordan, Shankar Sastry Presenters: Shiv Ballianda, Jerrolyn Hebert, Shuiwang Ji, Kenley Malveaux, Huy
More informationAdministration. CSCI567 Machine Learning (Fall 2018) Outline. Outline. HW5 is available, due on 11/18. Practice final will also be available soon.
Administration CSCI567 Machine Learning Fall 2018 Prof. Haipeng Luo U of Southern California Nov 7, 2018 HW5 is available, due on 11/18. Practice final will also be available soon. Remaining weeks: 11/14,
More informationReinforcement Learning Active Learning
Reinforcement Learning Active Learning Alan Fern * Based in part on slides by Daniel Weld 1 Active Reinforcement Learning So far, we ve assumed agent has a policy We just learned how good it is Now, suppose
More informationNotes on Reinforcement Learning
1 Introduction Notes on Reinforcement Learning Paulo Eduardo Rauber 2014 Reinforcement learning is the study of agents that act in an environment with the goal of maximizing cumulative reward signals.
More informationChapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationREINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
More informationMARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
More informationReinforcement Learning and Control
CS9 Lecture notes Andrew Ng Part XIII Reinforcement Learning and Control We now begin our study of reinforcement learning and adaptive control. In supervised learning, we saw algorithms that tried to make
More informationLecture 23: Reinforcement Learning
Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:
More informationReinforcement Learning with Function Approximation. Joseph Christian G. Noel
Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is
More informationPlanning in Markov Decision Processes
Carnegie Mellon School of Computer Science Deep Reinforcement Learning and Control Planning in Markov Decision Processes Lecture 3, CMU 10703 Katerina Fragkiadaki Markov Decision Process (MDP) A Markov
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
More informationLecture 3: The Reinforcement Learning Problem
Lecture 3: The Reinforcement Learning Problem Objectives of this lecture: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationOpen Theoretical Questions in Reinforcement Learning
Open Theoretical Questions in Reinforcement Learning Richard S. Sutton AT&T Labs, Florham Park, NJ 07932, USA, sutton@research.att.com, www.cs.umass.edu/~rich Reinforcement learning (RL) concerns the problem
More informationChapter 7: Eligibility Traces. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1
Chapter 7: Eligibility Traces R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Midterm Mean = 77.33 Median = 82 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction
More informationChristopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
More informationReinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
More information6 Reinforcement Learning
6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,
More informationReinforcement Learning. Machine Learning, Fall 2010
Reinforcement Learning Machine Learning, Fall 2010 1 Administrativia This week: finish RL, most likely start graphical models LA2: due on Thursday LA3: comes out on Thursday TA Office hours: Today 1:30-2:30
More informationReinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo MLR, University of Stuttgart
More informationLecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
More informationChapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
More informationREINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari
More informationLecture 25: Learning 4. Victor R. Lesser. CMPSCI 683 Fall 2010
Lecture 25: Learning 4 Victor R. Lesser CMPSCI 683 Fall 2010 Final Exam Information Final EXAM on Th 12/16 at 4:00pm in Lederle Grad Res Ctr Rm A301 2 Hours but obviously you can leave early! Open Book
More informationReinforcement Learning Part 2
Reinforcement Learning Part 2 Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ From previous tutorial Reinforcement Learning Exploration No supervision Agent-Reward-Environment
More informationLecture 2: Learning from Evaluative Feedback. or Bandit Problems
Lecture 2: Learning from Evaluative Feedback or Bandit Problems 1 Edward L. Thorndike (1874-1949) Puzzle Box 2 Learning by Trial-and-Error Law of Effect: Of several responses to the same situation, those
More informationCourse basics. CSE 190: Reinforcement Learning: An Introduction. Last Time. Course goals. The website for the class is linked off my homepage.
Course basics CSE 190: Reinforcement Learning: An Introduction The website for the class is linked off my homepage. Grades will be based on programming assignments, homeworks, and class participation.
More informationDeep Reinforcement Learning. STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017
Deep Reinforcement Learning STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017 Outline Introduction to Reinforcement Learning AlphaGo (Deep RL for Computer Go)
More informationCMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING
More informationReinforcement Learning. Up until now we have been
Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying,
More informationReinforcement Learning: the basics
Reinforcement Learning: the basics Olivier Sigaud Université Pierre et Marie Curie, PARIS 6 http://people.isir.upmc.fr/sigaud August 6, 2012 1 / 46 Introduction Action selection/planning Learning by trial-and-error
More informationTemporal difference learning
Temporal difference learning AI & Agents for IET Lecturer: S Luz http://www.scss.tcd.ie/~luzs/t/cs7032/ February 4, 2014 Recall background & assumptions Environment is a finite MDP (i.e. A and S are finite).
More informationDecision Theory: Q-Learning
Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning
More informationReview: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]
Review: TD-Learning function TD-Learning(mdp) returns a policy Class #: Reinforcement Learning, II 8s S, U(s) =0 set start-state s s 0 choose action a, using -greedy policy based on U(s) U(s) U(s)+ [r
More informationQ-learning. Tambet Matiisen
Q-learning Tambet Matiisen (based on chapter 11.3 of online book Artificial Intelligence, foundations of computational agents by David Poole and Alan Mackworth) Stochastic gradient descent Experience
More informationTemporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI
Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning
More informationIntroduction to Reinforcement Learning
CSCI-699: Advanced Topics in Deep Learning 01/16/2019 Nitin Kamra Spring 2019 Introduction to Reinforcement Learning 1 What is Reinforcement Learning? So far we have seen unsupervised and supervised learning.
More informationReinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of
More informationReinforcement Learning
Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation
More informationReinforcement Learning
Reinforcement Learning Ron Parr CompSci 7 Department of Computer Science Duke University With thanks to Kris Hauser for some content RL Highlights Everybody likes to learn from experience Use ML techniques
More informationReinforcement Learning. Spring 2018 Defining MDPs, Planning
Reinforcement Learning Spring 2018 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
More informationCSC321 Lecture 22: Q-Learning
CSC321 Lecture 22: Q-Learning Roger Grosse Roger Grosse CSC321 Lecture 22: Q-Learning 1 / 21 Overview Second of 3 lectures on reinforcement learning Last time: policy gradient (e.g. REINFORCE) Optimize
More informationMarks. bonus points. } Assignment 1: Should be out this weekend. } Mid-term: Before the last lecture. } Mid-term deferred exam:
Marks } Assignment 1: Should be out this weekend } All are marked, I m trying to tally them and perhaps add bonus points } Mid-term: Before the last lecture } Mid-term deferred exam: } This Saturday, 9am-10.30am,
More informationGeneralization and Function Approximation
Generalization and Function Approximation 0 Generalization and Function Approximation Suggested reading: Chapter 8 in R. S. Sutton, A. G. Barto: Reinforcement Learning: An Introduction MIT Press, 1998.
More informationAn Adaptive Clustering Method for Model-free Reinforcement Learning
An Adaptive Clustering Method for Model-free Reinforcement Learning Andreas Matt and Georg Regensburger Institute of Mathematics University of Innsbruck, Austria {andreas.matt, georg.regensburger}@uibk.ac.at
More informationCS 7180: Behavioral Modeling and Decisionmaking
CS 7180: Behavioral Modeling and Decisionmaking in AI Markov Decision Processes for Complex Decisionmaking Prof. Amy Sliva October 17, 2012 Decisions are nondeterministic In many situations, behavior and
More informationReinforcement Learning: An Introduction
Introduction Betreuer: Freek Stulp Hauptseminar Intelligente Autonome Systeme (WiSe 04/05) Forschungs- und Lehreinheit Informatik IX Technische Universität München November 24, 2004 Introduction What is
More informationReinforcement Learning (1)
Reinforcement Learning 1 Reinforcement Learning (1) Machine Learning 64-360, Part II Norman Hendrich University of Hamburg, Dept. of Informatics Vogt-Kölln-Str. 30, D-22527 Hamburg hendrich@informatik.uni-hamburg.de
More information15-780: Graduate Artificial Intelligence. Reinforcement learning (RL)
15-780: Graduate Artificial Intelligence Reinforcement learning (RL) From MDPs to RL We still use the same Markov model with rewards and actions But there are a few differences: 1. We do not assume we
More informationElements of Reinforcement Learning
Elements of Reinforcement Learning Policy: way learning algorithm behaves (mapping from state to action) Reward function: Mapping of state action pair to reward or cost Value function: long term reward,
More informationCS 598 Statistical Reinforcement Learning. Nan Jiang
CS 598 Statistical Reinforcement Learning Nan Jiang Overview What s this course about? A grad-level seminar course on theory of RL 3 What s this course about? A grad-level seminar course on theory of RL
More informationReinforcement learning
Reinforcement learning Based on [Kaelbling et al., 1996, Bertsekas, 2000] Bert Kappen Reinforcement learning Reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error
More informationRL 3: Reinforcement Learning
RL 3: Reinforcement Learning Q-Learning Michael Herrmann University of Edinburgh, School of Informatics 20/01/2015 Last time: Multi-Armed Bandits (10 Points to remember) MAB applications do exist (e.g.
More informationLecture 8: Policy Gradient
Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve
More informationReinforcement Learning. Introduction
Reinforcement Learning Introduction Reinforcement Learning Agent interacts and learns from a stochastic environment Science of sequential decision making Many faces of reinforcement learning Optimal control
More informationAn Introduction to Reinforcement Learning
1 / 58 An Introduction to Reinforcement Learning Lecture 01: Introduction Dr. Johannes A. Stork School of Computer Science and Communication KTH Royal Institute of Technology January 19, 2017 2 / 58 ../fig/reward-00.jpg
More informationReinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina
Reinforcement Learning Introduction Introduction Unsupervised learning has no outcome (no feedback). Supervised learning has outcome so we know what to predict. Reinforcement learning is in between it
More informationReading Response: Due Wednesday. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1
Reading Response: Due Wednesday R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Another Example Get to the top of the hill as quickly as possible. reward = 1 for each step where
More informationReinforcement Learning
Reinforcement Learning Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ Task Grasp the green cup. Output: Sequence of controller actions Setup from Lenz et. al.
More informationCSE 190: Reinforcement Learning: An Introduction. Chapter 8: Generalization and Function Approximation. Pop Quiz: What Function Are We Approximating?
CSE 190: Reinforcement Learning: An Introduction Chapter 8: Generalization and Function Approximation Objectives of this chapter: Look at how experience with a limited part of the state set be used to
More informationCS 570: Machine Learning Seminar. Fall 2016
CS 570: Machine Learning Seminar Fall 2016 Class Information Class web page: http://web.cecs.pdx.edu/~mm/mlseminar2016-2017/fall2016/ Class mailing list: cs570@cs.pdx.edu My office hours: T,Th, 2-3pm or
More informationDeep Reinforcement Learning: Policy Gradients and Q-Learning
Deep Reinforcement Learning: Policy Gradients and Q-Learning John Schulman Bay Area Deep Learning School September 24, 2016 Introduction and Overview Aim of This Talk What is deep RL, and should I use
More informationMonte Carlo is important in practice. CSE 190: Reinforcement Learning: An Introduction. Chapter 6: Temporal Difference Learning.
Monte Carlo is important in practice CSE 190: Reinforcement Learning: An Introduction Chapter 6: emporal Difference Learning When there are just a few possibilitieo value, out of a large state space, Monte
More informationMDP Preliminaries. Nan Jiang. February 10, 2019
MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process
More informationReinforcement Learning
Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns Forward and backward view Function
More informationThis question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
More informationCOMP3702/7702 Artificial Intelligence Lecture 11: Introduction to Machine Learning and Reinforcement Learning. Hanna Kurniawati
COMP3702/7702 Artificial Intelligence Lecture 11: Introduction to Machine Learning and Reinforcement Learning Hanna Kurniawati Today } What is machine learning? } Where is it used? } Types of machine learning
More informationPART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.
Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE
More informationDual Memory Model for Using Pre-Existing Knowledge in Reinforcement Learning Tasks
Dual Memory Model for Using Pre-Existing Knowledge in Reinforcement Learning Tasks Kary Främling Helsinki University of Technology, PL 55, FI-25 TKK, Finland Kary.Framling@hut.fi Abstract. Reinforcement
More informationDecision Theory: Markov Decision Processes
Decision Theory: Markov Decision Processes CPSC 322 Lecture 33 March 31, 2006 Textbook 12.5 Decision Theory: Markov Decision Processes CPSC 322 Lecture 33, Slide 1 Lecture Overview Recap Rewards and Policies
More informationIntroduction to Reinforcement Learning. Part 5: Temporal-Difference Learning
Introduction to Reinforcement Learning Part 5: emporal-difference Learning What everybody should know about emporal-difference (D) learning Used to learn value functions without human input Learns a guess
More informationLecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free
More informationReinforcement Learning. Summer 2017 Defining MDPs, Planning
Reinforcement Learning Summer 2017 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
More informationChapter 6: Temporal Difference Learning
Chapter 6: emporal Difference Learning Objectives of this chapter: Introduce emporal Difference (D) learning Focus first on policy evaluation, or prediction, methods Compare efficiency of D learning with
More informationMarkov Decision Processes and Solving Finite Problems. February 8, 2017
Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:
More informationQ-Learning for Markov Decision Processes*
McGill University ECSE 506: Term Project Q-Learning for Markov Decision Processes* Authors: Khoa Phan khoa.phan@mail.mcgill.ca Sandeep Manjanna sandeep.manjanna@mail.mcgill.ca (*Based on: Convergence of
More informationMarkov Decision Processes Chapter 17. Mausam
Markov Decision Processes Chapter 17 Mausam Planning Agent Static vs. Dynamic Fully vs. Partially Observable Environment What action next? Deterministic vs. Stochastic Perfect vs. Noisy Instantaneous vs.
More informationThe Book: Where we are and where we re going. CSE 190: Reinforcement Learning: An Introduction. Chapter 7: Eligibility Traces. Simple Monte Carlo
CSE 190: Reinforcement Learning: An Introduction Chapter 7: Eligibility races Acknowledgment: A good number of these slides are cribbed from Rich Sutton he Book: Where we are and where we re going Part
More informationToday s Outline. Recap: MDPs. Bellman Equations. Q-Value Iteration. Bellman Backup 5/7/2012. CSE 473: Artificial Intelligence Reinforcement Learning
CSE 473: Artificial Intelligence Reinforcement Learning Dan Weld Today s Outline Reinforcement Learning Q-value iteration Q-learning Exploration / exploitation Linear function approximation Many slides
More informationMachine Learning I Continuous Reinforcement Learning
Machine Learning I Continuous Reinforcement Learning Thomas Rückstieß Technische Universität München January 7/8, 2010 RL Problem Statement (reminder) state s t+1 ENVIRONMENT reward r t+1 new step r t
More informationReinforcement Learning as Classification Leveraging Modern Classifiers
Reinforcement Learning as Classification Leveraging Modern Classifiers Michail G. Lagoudakis and Ronald Parr Department of Computer Science Duke University Durham, NC 27708 Machine Learning Reductions
More informationCourse 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016
Course 16:198:520: Introduction To Artificial Intelligence Lecture 13 Decision Making Abdeslam Boularias Wednesday, December 7, 2016 1 / 45 Overview We consider probabilistic temporal models where the
More informationState Space Abstractions for Reinforcement Learning
State Space Abstractions for Reinforcement Learning Rowan McAllister and Thang Bui MLG RCC 6 November 24 / 24 Outline Introduction Markov Decision Process Reinforcement Learning State Abstraction 2 Abstraction
More informationReinforcement Learning with Recurrent Neural Networks
Institut für Informatik Neuroinformatics Group Reinforcement Learning with Recurrent Neural Networks Dissertation zur Erlangung der Doktorwürde der Universität Osnabrück Fachbereich Mathematik/Informatik
More informationInternet Monetization
Internet Monetization March May, 2013 Discrete time Finite A decision process (MDP) is reward process with decisions. It models an environment in which all states are and time is divided into stages. Definition
More informationSome AI Planning Problems
Course Logistics CS533: Intelligent Agents and Decision Making M, W, F: 1:00 1:50 Instructor: Alan Fern (KEC2071) Office hours: by appointment (see me after class or send email) Emailing me: include CS533
More informationArtificial Intelligence
Artificial Intelligence Dynamic Programming Marc Toussaint University of Stuttgart Winter 2018/19 Motivation: So far we focussed on tree search-like solvers for decision problems. There is a second important
More information