Animal learning theory

Size: px
Start display at page:

Download "Animal learning theory"

Transcription

1 Animal learning theory Based on [Sutton and Barto, 1990, Dayan and Abbott, 2001] Bert Kappen

2 [Sutton and Barto, 1990] Classical conditioning: - A conditioned stimulus (CS) and unconditioned stimulus (US) are paired in close succession - The US produces a response UR - After sufficient pairing, the CS produces a response CR similar to UR Example: rabbit eye blink condition - CS is sound of buzzer, US air puff to rabbit eye, UR is eye blink - After CS-US pairing, the buzzer alone yields CR eye blink Bert Kappen 1

3 [Sutton and Barto, 1990] Classical theory proposes that a causal relation between CS and US is learned. V = (level of US processing) (level of CS processing) = Reinforcement Eligibility V is association strength. - US processing is about reinforcement (positive or negative) - CS processing is about attention (which stimulus is credited) We focus on reinforcement processing with very simple eligibility models. Bert Kappen 2

4 Trial level vs. Real-time theories [Sutton and Barto, 1990] Trial level theories - ignore temporal aspects of individual trial and updates apply at end of trial - Rescorla-Wagner model - mostly used in experiments Real-time theories. - consider detailed timing aspects of the stimuli and rewards - TD learning Bert Kappen 3

5 Rescorla-Wagner (RW) model [Sutton and Barto, 1990] Central idea is that learning occurs whenever events violate expectations. In particular, when expected US V differs from the actual US level λ: V i := V i + V i V i = β (λ V) } {{ } V = reinforcement V i X i i α i X i }{{} saliency X i = 0, 1 to indicate absence/presence of CS i. α i is saliency (attention) of CS i (fixed). Term α i X i is often ignored for single stimulus. V i is the association strength of stimulus CS i to US. λ denotes the level of US, combining, in an unspecified way, the USs intensity, duration, and temporal relationship with the CS. 1 1 For instance, strenght, duration of air puff and timing relative to CS. Bert Kappen 4

6 RW is linear regression [Sutton and Barto, 1990] Consider sequence {(X i,t, λ t ), t = 1,..., T}. Define error C = 1 2 T λ t t=1 X i,t V i i 2 Minimizing C wrt V i gives best predictive model. Identical to RW rule with α i = 1. V i = β C = β V λ t i X i,t V i X i,t i Bert Kappen 5

7 Rescorla-Wagner model [Sutton and Barto, 1990] Rescola Wagner rule for multiple inputs can predict various phenomena: - Blocking: learned s 1 r prevents learning of association s 2 r - Inhibition: s 2 reduces prediction when combined with any predicting stimulus Bert Kappen 6

8 Rescorla-Wagner model [Sutton and Barto, 1990] Fails to explain second order conditioning: B A US - Empirical finding: if A predicts US and B predicts A, then B predicts US. - A US relation correctly reinforced by RW - B A relation not reinforced by RW because US λ = 0 in B A trials Solution is to assume that A also generates a λ Bert Kappen 7

9 Temporal aspect play important role [Sutton and Barto, 1990] Bert Kappen 8

10 Real-Time Theories of Eligibility [Sutton and Barto, 1990] Conditioning depends on time interval between CS and US. Mechanisms for coincidence of CS and US: - Stimulus trace (Hull 1939) is internal representation. Disadvantage of using stimulus trace for learning is that long inter-stimulus intervals (ISIs) require broad stimulus trace, which disagrees with fast, precisely timed responses. - Eligibility trace X (Knopf 1972) is similar to stimulus trace but only used for learning Bert Kappen 9

11 From RW to a real time theory [Sutton and Barto, 1990] Left :Trial based reinforcement λ is area under the curve. Right: Real-time reinforcement is future Problems: area under the curve. - Prediction is constant for all times prior to US. This disagrees with empirical observation Solution is imminence weighting to reduce long time association. Bert Kappen 10

12 Temporal difference learning [Sutton and Barto, 1990] Subject predicts at time t the imminence weighted area (c) rather than the unweighted area (a) which may be infinite. Bert Kappen 11

13 Temporal difference learning Dayan and Abbott 9.2 Consider time interval 0 t < with u(t) stimulus, λ(t) the reward. The expected (discounted=imminence weighted) future reward is R(t) = E γ τ t λ(τ) τ=t The aim of reinforcement learning is to learn a signal v(t) that predicts the expected future reward R(t) from the past stimulus. v(t) = T w(τ)u(t τ) τ=0 with w(τ) adaptive parameters. Minimize quadratic error C = 1 2 t=0 (v(t) R(t)) 2 with respect w(τ) yields w(τ) = ɛ C(t) w(τ) = ɛ t=0 (R(t) v(t)) u(t τ) τ = 0,..., T Bert Kappen 12

14 Temporal difference learning Dayan and Abbott 9.2 Problem is that we do not know R(t). But we can use the recursive relation R(t) = E γ τ t λ(τ) = λ(t) + γe γ τ t 1 λ(τ) = λ(t) + γr(t + 1) λ(t) + γv(t + 1) τ=t τ=t+1 We approximate R(t + 1) by v(t + 1). Update at time t becomes: w(τ) = ɛ (R(t) v(t)) u(t τ) ɛ (λ(t) + γv(t + 1) v(t)) u(t τ) Bert Kappen 13

15 Temporal difference learning Dayan and Abbott 9.2 When v(t) does not depend on stimulus, but is simply a parameter that is estimated for each t: C = 1 2 (R(t) v(t)) 2 t=0 v(t) = ɛ C v(t) = ɛ t=0 (R(t) v(t)) ɛ (r(t) + γv(t + 1) v(t)) t=0 Bert Kappen 14

16 Inter-Stimulus-Interval Dependency Dayan and Abbott 9.2 Rabbit CR (eye blink) to CS (sound) as a function of CS to US (puff) time interval shows good agreement with TD model. Bert Kappen 15

17 Fig. 9.2 Dayan and Abbott 9.2 Stimulus is u(t) = δ t,100, thus Eq. 9.6 v(t) = w(τ)u(t τ) = w(t 100) t 100 τ = 0 0 t < 100 Denote τ = t 100 with t 100, then v(t) = w(τ). Delta rule Eq. 9.10: w(τ) = ɛδ(t)u(t τ) = ɛδ(t)δ t τ,100 v(t) = w(τ) = ɛδ(t) δ(t) = r(t) + v(t + 1) v(t) and w(τ) = 0 if τ t 100. Reward is r = δ t,200. Bert Kappen 16

18 First iteration: Second iteration: Third iteration: Dayan and Abbott 9.2 v(t) = 0 δ(t) = δ t,200 v(t) = ɛδ t,200 δ(t) = (1 ɛ)δ t,200 + ɛδ t,199 v(t) = (2ɛ ɛ 2 )δ t,200 + ɛδ t,199 δ(t) =... δ(t) = v(t) + r(t) Bert Kappen 17

19 Dopamine Dayan and Abbott 9.2 Activity of dopamine neurons in ventral tegmental area (VTA) of monkey encode δ in fig Monkey button press after stimulus (sound) to receive reward (fruit juice). - A: Left panels, trials locked to stimulus, right panels trials locked to reward. Top row is early trials, bottom row is late trials. VTA cells respond to reward in early trials and to stimulus in late trials. - B Top: stimulus locked trials after learning as in bottom fig. A - B bottom: witholding the (expected) reward yields inhibition Bert Kappen 18

20 Instrumental conditioning/static action choice Dayan Abbott 9.3 Conditioning: Classical: reward (punishment) independent of action taken (Pavlov dog) Instrumental: reward depends on action taken Reward timing: immediate: reward follows directly after action (static action choice) delayed: reward follows after sequence of actions Bert Kappen 19

21 Foraging of bees Dayan Abbott 9.3 Foraging of bees as example of learning behavior with immediate reward: bees visit flowers of two colors (blue, yellow), prefering color with higher reward (nectar) when rewards are swapped, bees adjust their preference Quantity of nectar r b,y is stochastic from (fixed) distribution q r,b (r). This is like a two-armed bandit problem. The policy of the bees is given by a probability p b,y with p b + p r parametrize this as = 1. We can p b = e βm b e βm b + e βm y p y = e βm y e βm b + e βm y The parameters m b,y are called action values, that do no require normalization. The parameter β controls exploration-exploitation: β = 0: exploration, β = : exploitation Bert Kappen 20

22 Foraging of bees Dayan Abbott 9.3 Two learning strategies: learn action value as past expected reward (indirect actor) learn action value to maximize future expected reward (direct actor) Bert Kappen 21

23 Indirect actor Dayan Abbott 9.3 We need a learning rule to estimate m b = r b = q b (r b )r b m y = r b ry = q y (r y )r y r y Consider the cost function E(m b ) = 1 2 when de dm b = r b q b (r)(m b r b ) 2. The function is minimized q b (r b )(m b r b ) = m b r b = 0 r We use stochastic gradient descent, replacing r b by a single sample r b : m b := m b ɛ de dm b = m b + ɛ(r b m b ) and similar for m y : m y := m y + ɛ(r y m y ). This is the Delta rule. Bert Kappen 22

24 Indirect actor Dayan Abbott 9.3 r b = 1, r y = 2 for first 100 visits with qb (r) = 1 2 δ r, δ r,2, q y (r) = 1 2 δ r, δ r,4. Rewards are reversed for second 100 visits. A) Visits selected according action values with β = 1 and learning of m b,y with delta rule. B) Cumulated reward for β = 1 and (C,D) for β = 50 shows better exploitation but worse exploration (sometimes) for large β. This behavior is seen in real bees. Bert Kappen 23

25 Indirect actor Dayan Abbott 9.3 Real bees are risk averse. Blue flowers provide 2 µl nectar, yellow flowers provide 6µl on 1/3 of the trials and zero otherwise. After 15 trials reversed. A) mean preference of 5 bees for blue flowers over 30 trials, each consisting of 40 visits. B) action value is concave function of nectar volume prefers low risk option. C) preference of single model bee (ɛ = 0.3, β = 23/8) Bert Kappen 24

26 Direct actor Dayan Abbott 9.3 Direct actor method estimates m b,y such as to maximize the expected future reward r = p b r b + p y ry We use stochastic gradient ascent on r : m b := m b + ɛβp b p y ( rb r y ) = mb + ɛβp b (1 p b ) r b ɛβp y p b ry m y := m y + ɛβp b p y ( ry rb ) = m y + ɛβp y (1 p y ) r y ɛβpb p y r b where we used dp b dm b = βp b p y dp y dm b = βp b p y Bert Kappen 25

27 Direct actor Dayan Abbott 9.3 The stochastic version is δm b = ɛ(1 p b )r b δm y = ɛ p y r b i f b is selected δm b = ɛ p b r y δm y = ɛ(1 p y )r b i f y is selected For multiple actions the rule generalizes to, when action a is taken: δm a = ɛ(δ a,a p a )r a a (For instance, take a = b and a = b, y gives first line above). We will use this in the actor-critic algorithm (DA 9.4). Bert Kappen 26

28 Direct actor Dayan Abbott 9.3 Setting as before; two different runs. A,B) Succesful learning. C,D) Failure to learn to switch due to larger difference m y m b Illustrates that learning methods that directly optimize expected future reward may be too greedy and have poor exploration. Bert Kappen 27

29 Delayed reward/sequential decision making Dayan Abbott 9.4 Reward obtained after sequence of actions. Rat moves without back tracking. After reward removed from maze and restart. Delayed reward problem: Choice at A has no direct reward Bert Kappen 28

30 Delayed reward/sequential decision making Dayan Abbott 9.4 Policy iteration (see [Kaelbling et al., 1996] 3.2.2): Loop: Policy evaluation: Compute value V π for policy π. Run Bellman backup until convergence Policy improvement: Improve π Bert Kappen 29

31 Delayed reward/sequential decision making Dayan Abbott 9.4 Actor Critic (see [Kaelbling et al., 1996] 4.1): Loop: Critic: use TD eval. V(state) using current policy Actor: improve policy p(state) Bert Kappen 30

32 Policy evaluation with TD Dayan Abbott 9.4 Policy is random left/right at each turn. v(b) = 1 2 (0 + 5) = 2.5 v(c) = 1 2 (0 + 2) = 1 v(a) = 1 (v(b) + v(c)) = Learn through TD learning (v(s) = w(s), s = A, B, C): v(s) := v(s) + ɛδ δ = r(s) + v(s ) v(s) Bert Kappen 31

33 Policy improvement Dayan Abbott 9.4 We use the direct actor method, which updates all m a upon taking action a as: m a := m a + ɛ(δ a,a p a )r a a In the maze, the actions (and action values) depend on the state and reward is delayed r a r a + v(s ) v(s): m a (s) := m a (s) + ɛ(δ a,a p a (s))δ δ = r a + v(s ) v(s) p a (s) = e βm a(s) a eβm a (s) Example: the values of the random policy are v(a) = 1.75 v(b) = 2.5 v(c) = 1 Bert Kappen 32

34 When in state A: Dayan Abbott 9.4 δ = 0 + v(b) v(a) = 0.75 f or right turn δ = 0 + v(c) v(a) = 0.75 f or le f t turn The learning rule increase m right (A) and decreases m left (A). β controls the exploration. Actor critic learning: show probability of left turn versus trials. Slow convergence of C due to less visits of right arm. Bert Kappen 33

35 Generalizations Dayan Abbott 9.4 Generalizations of the basis Actor-Critic model are: 1. State s may not be directly observable, but instead a vector u i (s) of sensory information is available The value function v(s) can be a parametrized function of u i (s). For instance a linear parametrization, the TD learning rule is v(s) = w i u i (s) w i := w i + ɛδu i (s) δ = r(s) + v(s ) v(s) i The action value m a (s) also becomes a function of u i instead of s directly: m a (s) = M ai u i (s) i M a i := M a i + ɛ(δ aa p a (s))δu i (s) Three-term learning rule: The connection between neurons i and a is changed depending on a third term δ 2. Discounted reward. Earlier reward/punishment more important than later. δ = r(s) + γv(s ) v(s) Bert Kappen 34

36 3. Updating value of past states in each learning step: TD(λ) Dayan Abbott 9.4 Bert Kappen 35

37 Water Maze Dayan Abbott 9.4 Rat is place in a large pool of milky water and has to swim around until find a small platform that is submerged. After several trials, the rat learns to locate the platform. (A) u i (s) is activity of place cell array, s is physical location (B) Critic: v(s) = i w i u i (s) is value function. (C upper) Actor: m a (s) = i M ai u i (s) is the action value (C lower) Bert Kappen 36

38 References Dayan Abbott 9.4 References [Dayan and Abbott, 2001] Dayan, P. and Abbott, L. (2001). Theoretical Neuroscience. Computational and Mathematical Modeling of Neural Systems. MIT Press, New York. [Kaelbling et al., 1996] Kaelbling, L., Littman, M., and Moore, A. (1996). Reinforcement learning: a survey. Journal of Artificial Intelligence research, 4: [Sutton and Barto, 1990] Sutton, R. S. and Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement., pages Cambridge University Press. Bert Kappen 37

Reinforcement learning

Reinforcement learning einforcement learning How to learn to make decisions in sequential problems (like: chess, a maze) Why is this difficult? Temporal credit assignment Prediction can help Further reading For modeling: Chapter

More information

Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associate

Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associate Key concepts: Reward: external «cons umable» s5mulus Value: internal representa5on of u5lity of a state, ac5on, s5mulus W the learned weight associated to the value Teaching signal : difference between

More information

Reinforcement learning

Reinforcement learning Reinforcement learning Based on [Kaelbling et al., 1996, Bertsekas, 2000] Bert Kappen Reinforcement learning Reinforcement learning is the problem faced by an agent that must learn behavior through trial-and-error

More information

Rewards and Value Systems: Conditioning and Reinforcement Learning

Rewards and Value Systems: Conditioning and Reinforcement Learning Rewards and Value Systems: Conditioning and Reinforcement Learning Acknowledgements: Many thanks to P. Dayan and L. Abbott for making the figures of their book available online. Also thanks to R. Sutton

More information

Introduction to Reinforcement Learning. CMPT 882 Mar. 18

Introduction to Reinforcement Learning. CMPT 882 Mar. 18 Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and

More information

Temporal difference learning

Temporal difference learning Temporal difference learning AI & Agents for IET Lecturer: S Luz http://www.scss.tcd.ie/~luzs/t/cs7032/ February 4, 2014 Recall background & assumptions Environment is a finite MDP (i.e. A and S are finite).

More information

Reinforcement Learning In Continuous Time and Space

Reinforcement Learning In Continuous Time and Space Reinforcement Learning In Continuous Time and Space presentation of paper by Kenji Doya Leszek Rybicki lrybicki@mat.umk.pl 18.07.2008 Leszek Rybicki lrybicki@mat.umk.pl Reinforcement Learning In Continuous

More information

CS599 Lecture 1 Introduction To RL

CS599 Lecture 1 Introduction To RL CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming

More information

Reinforcement Learning

Reinforcement Learning Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of

More information

Reinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina

Reinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina Reinforcement Learning Introduction Introduction Unsupervised learning has no outcome (no feedback). Supervised learning has outcome so we know what to predict. Reinforcement learning is in between it

More information

Temporal Difference Learning & Policy Iteration

Temporal Difference Learning & Policy Iteration Temporal Difference Learning & Policy Iteration Advanced Topics in Reinforcement Learning Seminar WS 15/16 ±0 ±0 +1 by Tobias Joppen 03.11.2015 Fachbereich Informatik Knowledge Engineering Group Prof.

More information

Basics of reinforcement learning

Basics of reinforcement learning Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system

More information

Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm

Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Balancing and Control of a Freely-Swinging Pendulum Using a Model-Free Reinforcement Learning Algorithm Michail G. Lagoudakis Department of Computer Science Duke University Durham, NC 2778 mgl@cs.duke.edu

More information

MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti

MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti 1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early

More information

Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015

Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)

More information

CO6: Introduction to Computational Neuroscience

CO6: Introduction to Computational Neuroscience CO6: Introduction to Computational Neuroscience Lecturer: J Lussange Ecole Normale Supérieure 29 rue d Ulm e-mail: johann.lussange@ens.fr Solutions to the 2nd exercise sheet If you have any questions regarding

More information

Elements of Reinforcement Learning

Elements of Reinforcement Learning Elements of Reinforcement Learning Policy: way learning algorithm behaves (mapping from state to action) Reward function: Mapping of state action pair to reward or cost Value function: long term reward,

More information

Reinforcement Learning

Reinforcement Learning Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo MLR, University of Stuttgart

More information

Temporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI

Temporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning

More information

Today s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes

Today s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks

More information

Introduction to Reinforcement Learning

Introduction to Reinforcement Learning Introduction to Reinforcement Learning Rémi Munos SequeL project: Sequential Learning http://researchers.lille.inria.fr/ munos/ INRIA Lille - Nord Europe Machine Learning Summer School, September 2011,

More information

Reinforcement Learning: the basics

Reinforcement Learning: the basics Reinforcement Learning: the basics Olivier Sigaud Université Pierre et Marie Curie, PARIS 6 http://people.isir.upmc.fr/sigaud August 6, 2012 1 / 46 Introduction Action selection/planning Learning by trial-and-error

More information

Reinforcement Learning. Spring 2018 Defining MDPs, Planning

Reinforcement Learning. Spring 2018 Defining MDPs, Planning Reinforcement Learning Spring 2018 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state

More information

Q-Learning in Continuous State Action Spaces

Q-Learning in Continuous State Action Spaces Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental

More information

Reinforcement Learning with Function Approximation. Joseph Christian G. Noel

Reinforcement Learning with Function Approximation. Joseph Christian G. Noel Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is

More information

Reinforcement Learning for Continuous. Action using Stochastic Gradient Ascent. Hajime KIMURA, Shigenobu KOBAYASHI JAPAN

Reinforcement Learning for Continuous. Action using Stochastic Gradient Ascent. Hajime KIMURA, Shigenobu KOBAYASHI JAPAN Reinforcement Learning for Continuous Action using Stochastic Gradient Ascent Hajime KIMURA, Shigenobu KOBAYASHI Tokyo Institute of Technology, 4259 Nagatsuda, Midori-ku Yokohama 226-852 JAPAN Abstract:

More information

ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control

ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control Lecturer: Nikolay Atanasov: natanasov@ucsd.edu Teaching Assistants: Tianyu Wang: tiw161@eng.ucsd.edu Yongxi Lu: yol070@eng.ucsd.edu

More information

Chapter 7: Eligibility Traces. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1

Chapter 7: Eligibility Traces. R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Chapter 7: Eligibility Traces R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction 1 Midterm Mean = 77.33 Median = 82 R. S. Sutton and A. G. Barto: Reinforcement Learning: An Introduction

More information

Reinforcement Learning

Reinforcement Learning 1 Reinforcement Learning Chris Watkins Department of Computer Science Royal Holloway, University of London July 27, 2015 2 Plan 1 Why reinforcement learning? Where does this theory come from? Markov decision

More information

Open Theoretical Questions in Reinforcement Learning

Open Theoretical Questions in Reinforcement Learning Open Theoretical Questions in Reinforcement Learning Richard S. Sutton AT&T Labs, Florham Park, NJ 07932, USA, sutton@research.att.com, www.cs.umass.edu/~rich Reinforcement learning (RL) concerns the problem

More information

RL 3: Reinforcement Learning

RL 3: Reinforcement Learning RL 3: Reinforcement Learning Q-Learning Michael Herrmann University of Edinburgh, School of Informatics 20/01/2015 Last time: Multi-Armed Bandits (10 Points to remember) MAB applications do exist (e.g.

More information

Grundlagen der Künstlichen Intelligenz

Grundlagen der Künstlichen Intelligenz Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and

More information

Prof. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be

Prof. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while

More information

Reinforcement Learning and Time Perception - a Model of Animal Experiments

Reinforcement Learning and Time Perception - a Model of Animal Experiments Reinforcement Learning and Time Perception - a Model of Animal Experiments J. L. Shapiro Department of Computer Science University of Manchester Manchester, M13 9PL U.K. jls@cs.man.ac.uk John Wearden Department

More information

Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))]

Review: TD-Learning. TD (SARSA) Learning for Q-values. Bellman Equations for Q-values. P (s, a, s )[R(s, a, s )+ Q (s, (s ))] Review: TD-Learning function TD-Learning(mdp) returns a policy Class #: Reinforcement Learning, II 8s S, U(s) =0 set start-state s s 0 choose action a, using -greedy policy based on U(s) U(s) U(s)+ [r

More information

An Adaptive Clustering Method for Model-free Reinforcement Learning

An Adaptive Clustering Method for Model-free Reinforcement Learning An Adaptive Clustering Method for Model-free Reinforcement Learning Andreas Matt and Georg Regensburger Institute of Mathematics University of Innsbruck, Austria {andreas.matt, georg.regensburger}@uibk.ac.at

More information

6 Basic Convergence Results for RL Algorithms

6 Basic Convergence Results for RL Algorithms Learning in Complex Systems Spring 2011 Lecture Notes Nahum Shimkin 6 Basic Convergence Results for RL Algorithms We establish here some asymptotic convergence results for the basic RL algorithms, by showing

More information

Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016

Course 16:198:520: Introduction To Artificial Intelligence Lecture 13. Decision Making. Abdeslam Boularias. Wednesday, December 7, 2016 Course 16:198:520: Introduction To Artificial Intelligence Lecture 13 Decision Making Abdeslam Boularias Wednesday, December 7, 2016 1 / 45 Overview We consider probabilistic temporal models where the

More information

, and rewards and transition matrices as shown below:

, and rewards and transition matrices as shown below: CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount

More information

1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5

1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5 Table of contents 1 Introduction 2 2 Markov Decision Processes 2 3 Future Cumulative Reward 3 4 Q-Learning 4 4.1 The Q-value.............................................. 4 4.2 The Temporal Difference.......................................

More information

Lecture 23: Reinforcement Learning

Lecture 23: Reinforcement Learning Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:

More information

Reinforcement learning an introduction

Reinforcement learning an introduction Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,

More information

Dialogue management: Parametric approaches to policy optimisation. Dialogue Systems Group, Cambridge University Engineering Department

Dialogue management: Parametric approaches to policy optimisation. Dialogue Systems Group, Cambridge University Engineering Department Dialogue management: Parametric approaches to policy optimisation Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 30 Dialogue optimisation as a reinforcement learning

More information

Reinforcement Learning

Reinforcement Learning Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation

More information

Machine Learning I Reinforcement Learning

Machine Learning I Reinforcement Learning Machine Learning I Reinforcement Learning Thomas Rückstieß Technische Universität München December 17/18, 2009 Literature Book: Reinforcement Learning: An Introduction Sutton & Barto (free online version:

More information

Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation

Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free

More information

arxiv: v1 [cs.ai] 1 Jul 2015

arxiv: v1 [cs.ai] 1 Jul 2015 arxiv:507.00353v [cs.ai] Jul 205 Harm van Seijen harm.vanseijen@ualberta.ca A. Rupam Mahmood ashique@ualberta.ca Patrick M. Pilarski patrick.pilarski@ualberta.ca Richard S. Sutton sutton@cs.ualberta.ca

More information

Reinforcement Learning. Odelia Schwartz 2016

Reinforcement Learning. Odelia Schwartz 2016 Reinforcement Learning Odelia Schwartz 2016 Forms of learning? Forms of learning Unsupervised learning Supervised learning Reinforcement learning Forms of learning Unsupervised learning Supervised learning

More information

On the Convergence of Optimistic Policy Iteration

On the Convergence of Optimistic Policy Iteration Journal of Machine Learning Research 3 (2002) 59 72 Submitted 10/01; Published 7/02 On the Convergence of Optimistic Policy Iteration John N. Tsitsiklis LIDS, Room 35-209 Massachusetts Institute of Technology

More information

REINFORCEMENT LEARNING

REINFORCEMENT LEARNING REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents

More information

The Nature of Learning - A study of Reinforcement Learning Methodology

The Nature of Learning - A study of Reinforcement Learning Methodology The Nature of Learning - A study of Reinforcement Learning Methodology Anders Christian Kølle 1 Department of Computer Science University of Copenhagen November 21, 23 1 andersk@diku.dk One needs to explore

More information

(Deep) Reinforcement Learning

(Deep) Reinforcement Learning Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 17 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015

More information

Summary of part I: prediction and RL

Summary of part I: prediction and RL Summary of part I: prediction and RL Prediction is important for action selection The problem: prediction of future reward The algorithm: temporal difference learning Neural implementation: dopamine dependent

More information

Reinforcement Learning II

Reinforcement Learning II Reinforcement Learning II Andrea Bonarini Artificial Intelligence and Robotics Lab Department of Electronics and Information Politecnico di Milano E-mail: bonarini@elet.polimi.it URL:http://www.dei.polimi.it/people/bonarini

More information

An Introduction to Reinforcement Learning

An Introduction to Reinforcement Learning 1 / 58 An Introduction to Reinforcement Learning Lecture 01: Introduction Dr. Johannes A. Stork School of Computer Science and Communication KTH Royal Institute of Technology January 19, 2017 2 / 58 ../fig/reward-00.jpg

More information

Machine Learning. Reinforcement learning. Hamid Beigy. Sharif University of Technology. Fall 1396

Machine Learning. Reinforcement learning. Hamid Beigy. Sharif University of Technology. Fall 1396 Machine Learning Reinforcement learning Hamid Beigy Sharif University of Technology Fall 1396 Hamid Beigy (Sharif University of Technology) Machine Learning Fall 1396 1 / 32 Table of contents 1 Introduction

More information

arxiv: v1 [cs.ai] 5 Nov 2017

arxiv: v1 [cs.ai] 5 Nov 2017 arxiv:1711.01569v1 [cs.ai] 5 Nov 2017 Markus Dumke Department of Statistics Ludwig-Maximilians-Universität München markus.dumke@campus.lmu.de Abstract Temporal-difference (TD) learning is an important

More information

Lecture 25: Learning 4. Victor R. Lesser. CMPSCI 683 Fall 2010

Lecture 25: Learning 4. Victor R. Lesser. CMPSCI 683 Fall 2010 Lecture 25: Learning 4 Victor R. Lesser CMPSCI 683 Fall 2010 Final Exam Information Final EXAM on Th 12/16 at 4:00pm in Lederle Grad Res Ctr Rm A301 2 Hours but obviously you can leave early! Open Book

More information

Bias-Variance Error Bounds for Temporal Difference Updates

Bias-Variance Error Bounds for Temporal Difference Updates Bias-Variance Bounds for Temporal Difference Updates Michael Kearns AT&T Labs mkearns@research.att.com Satinder Singh AT&T Labs baveja@research.att.com Abstract We give the first rigorous upper bounds

More information

Reinforcement Learning and NLP

Reinforcement Learning and NLP 1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value

More information

Reinforcement learning crash course

Reinforcement learning crash course crash course Quentin Huys Translational Neuromodeling Unit, University of Zurich and ETH Zurich University Hospital of Psychiatry Zurich Computational Psychiatry Course Zurich, 1.9.2016 Overview : rough

More information

Computational Reinforcement Learning: An Introduction

Computational Reinforcement Learning: An Introduction Computational Reinforcement Learning: An Introduction Andrew Barto Autonomous Learning Laboratory School of Computer Science University of Massachusetts Amherst barto@cs.umass.edu 1 Artificial Intelligence

More information

Reinforcement Learning. Yishay Mansour Tel-Aviv University

Reinforcement Learning. Yishay Mansour Tel-Aviv University Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak

More information

The convergence limit of the temporal difference learning

The convergence limit of the temporal difference learning The convergence limit of the temporal difference learning Ryosuke Nomura the University of Tokyo September 3, 2013 1 Outline Reinforcement Learning Convergence limit Construction of the feature vector

More information

Reinforcement Learning in a Nutshell

Reinforcement Learning in a Nutshell Reinforcement Learning in a Nutshell V. Heidrich-Meisner 1, M. Lauer 2, C. Igel 1 and M. Riedmiller 2 1- Institut für Neuroinformatik, Ruhr-Universität Bochum, Germany 2- Neuroinformatics Group, University

More information

Markov Decision Processes and Solving Finite Problems. February 8, 2017

Markov Decision Processes and Solving Finite Problems. February 8, 2017 Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:

More information

Lecture 9: Policy Gradient II 1

Lecture 9: Policy Gradient II 1 Lecture 9: Policy Gradient II 1 Emma Brunskill CS234 Reinforcement Learning. Winter 2019 Additional reading: Sutton and Barto 2018 Chp. 13 1 With many slides from or derived from David Silver and John

More information

POMDPs and Policy Gradients

POMDPs and Policy Gradients POMDPs and Policy Gradients MLSS 2006, Canberra Douglas Aberdeen Canberra Node, RSISE Building Australian National University 15th February 2006 Outline 1 Introduction What is Reinforcement Learning? Types

More information

Deep Reinforcement Learning SISL. Jeremy Morton (jmorton2) November 7, Stanford Intelligent Systems Laboratory

Deep Reinforcement Learning SISL. Jeremy Morton (jmorton2) November 7, Stanford Intelligent Systems Laboratory Deep Reinforcement Learning Jeremy Morton (jmorton2) November 7, 2016 SISL Stanford Intelligent Systems Laboratory Overview 2 1 Motivation 2 Neural Networks 3 Deep Reinforcement Learning 4 Deep Learning

More information

Policy Gradient Reinforcement Learning for Robotics

Policy Gradient Reinforcement Learning for Robotics Policy Gradient Reinforcement Learning for Robotics Michael C. Koval mkoval@cs.rutgers.edu Michael L. Littman mlittman@cs.rutgers.edu May 9, 211 1 Introduction Learning in an environment with a continuous

More information

MDP Preliminaries. Nan Jiang. February 10, 2019

MDP Preliminaries. Nan Jiang. February 10, 2019 MDP Preliminaries Nan Jiang February 10, 2019 1 Markov Decision Processes In reinforcement learning, the interactions between the agent and the environment are often described by a Markov Decision Process

More information

PART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.

PART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B. Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE

More information

Reinforcement Learning

Reinforcement Learning Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value

More information

On and Off-Policy Relational Reinforcement Learning

On and Off-Policy Relational Reinforcement Learning On and Off-Policy Relational Reinforcement Learning Christophe Rodrigues, Pierre Gérard, and Céline Rouveirol LIPN, UMR CNRS 73, Institut Galilée - Université Paris-Nord first.last@lipn.univ-paris13.fr

More information

CS 287: Advanced Robotics Fall Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study

CS 287: Advanced Robotics Fall Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study CS 287: Advanced Robotics Fall 2009 Lecture 14: Reinforcement Learning with Function Approximation and TD Gammon case study Pieter Abbeel UC Berkeley EECS Assignment #1 Roll-out: nice example paper: X.

More information

Lecture 8: Policy Gradient

Lecture 8: Policy Gradient Lecture 8: Policy Gradient Hado van Hasselt Outline 1 Introduction 2 Finite Difference Policy Gradient 3 Monte-Carlo Policy Gradient 4 Actor-Critic Policy Gradient Introduction Vapnik s rule Never solve

More information

State Space Abstractions for Reinforcement Learning

State Space Abstractions for Reinforcement Learning State Space Abstractions for Reinforcement Learning Rowan McAllister and Thang Bui MLG RCC 6 November 24 / 24 Outline Introduction Markov Decision Process Reinforcement Learning State Abstraction 2 Abstraction

More information

Reinforcement Learning II. George Konidaris

Reinforcement Learning II. George Konidaris Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2017 Reinforcement Learning π : S A max R = t=0 t r t MDPs Agent interacts with an environment At each time t: Receives sensor signal Executes

More information

CSC242: Intro to AI. Lecture 23

CSC242: Intro to AI. Lecture 23 CSC242: Intro to AI Lecture 23 Administrivia Posters! Tue Apr 24 and Thu Apr 26 Idea! Presentation! 2-wide x 4-high landscape pages Learning so far... Input Attributes Alt Bar Fri Hun Pat Price Rain Res

More information

Lecture 1: March 7, 2018

Lecture 1: March 7, 2018 Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights

More information

Olivier Sigaud. September 21, 2012

Olivier Sigaud. September 21, 2012 Supervised and Reinforcement Learning Tools for Motor Learning Models Olivier Sigaud Université Pierre et Marie Curie - Paris 6 September 21, 2012 1 / 64 Introduction Who is speaking? 2 / 64 Introduction

More information

Reinforcement Learning. George Konidaris

Reinforcement Learning. George Konidaris Reinforcement Learning George Konidaris gdk@cs.brown.edu Fall 2017 Machine Learning Subfield of AI concerned with learning from data. Broadly, using: Experience To Improve Performance On Some Task (Tom

More information

Reinforcement Learning II. George Konidaris

Reinforcement Learning II. George Konidaris Reinforcement Learning II George Konidaris gdk@cs.brown.edu Fall 2018 Reinforcement Learning π : S A max R = t=0 t r t MDPs Agent interacts with an environment At each time t: Receives sensor signal Executes

More information

Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms *

Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms * Proceedings of the 8 th International Conference on Applied Informatics Eger, Hungary, January 27 30, 2010. Vol. 1. pp. 87 94. Using Gaussian Processes for Variance Reduction in Policy Gradient Algorithms

More information

Reinforcement Learning in Non-Stationary Continuous Time and Space Scenarios

Reinforcement Learning in Non-Stationary Continuous Time and Space Scenarios Reinforcement Learning in Non-Stationary Continuous Time and Space Scenarios Eduardo W. Basso 1, Paulo M. Engel 1 1 Instituto de Informática Universidade Federal do Rio Grande do Sul (UFRGS) Caixa Postal

More information

6 Reinforcement Learning

6 Reinforcement Learning 6 Reinforcement Learning As discussed above, a basic form of supervised learning is function approximation, relating input vectors to output vectors, or, more generally, finding density functions p(y,

More information

Augmented Rescorla-Wagner and Maximum Likelihood Estimation.

Augmented Rescorla-Wagner and Maximum Likelihood Estimation. Augmented Rescorla-Wagner and Maximum Likelihood Estimation. Alan Yuille Department of Statistics University of California at Los Angeles Los Angeles, CA 90095 yuille@stat.ucla.edu In Advances in Neural

More information

ilstd: Eligibility Traces and Convergence Analysis

ilstd: Eligibility Traces and Convergence Analysis ilstd: Eligibility Traces and Convergence Analysis Alborz Geramifard Michael Bowling Martin Zinkevich Richard S. Sutton Department of Computing Science University of Alberta Edmonton, Alberta {alborz,bowling,maz,sutton}@cs.ualberta.ca

More information

Actor-critic methods. Dialogue Systems Group, Cambridge University Engineering Department. February 21, 2017

Actor-critic methods. Dialogue Systems Group, Cambridge University Engineering Department. February 21, 2017 Actor-critic methods Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department February 21, 2017 1 / 21 In this lecture... The actor-critic architecture Least-Squares Policy Iteration

More information

CMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro

CMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING

More information

REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning

REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari

More information

An Introduction to Reinforcement Learning

An Introduction to Reinforcement Learning An Introduction to Reinforcement Learning Shivaram Kalyanakrishnan shivaram@cse.iitb.ac.in Department of Computer Science and Engineering Indian Institute of Technology Bombay April 2018 What is Reinforcement

More information

Decision Theory: Q-Learning

Decision Theory: Q-Learning Decision Theory: Q-Learning CPSC 322 Decision Theory 5 Textbook 12.5 Decision Theory: Q-Learning CPSC 322 Decision Theory 5, Slide 1 Lecture Overview 1 Recap 2 Asynchronous Value Iteration 3 Q-Learning

More information

QUICR-learning for Multi-Agent Coordination

QUICR-learning for Multi-Agent Coordination QUICR-learning for Multi-Agent Coordination Adrian K. Agogino UCSC, NASA Ames Research Center Mailstop 269-3 Moffett Field, CA 94035 adrian@email.arc.nasa.gov Kagan Tumer NASA Ames Research Center Mailstop

More information

Exponential Moving Average Based Multiagent Reinforcement Learning Algorithms

Exponential Moving Average Based Multiagent Reinforcement Learning Algorithms Artificial Intelligence Review manuscript No. (will be inserted by the editor) Exponential Moving Average Based Multiagent Reinforcement Learning Algorithms Mostafa D. Awheda Howard M. Schwartz Received:

More information

Q-learning. Tambet Matiisen

Q-learning. Tambet Matiisen Q-learning Tambet Matiisen (based on chapter 11.3 of online book Artificial Intelligence, foundations of computational agents by David Poole and Alan Mackworth) Stochastic gradient descent Experience

More information

Marks. bonus points. } Assignment 1: Should be out this weekend. } Mid-term: Before the last lecture. } Mid-term deferred exam:

Marks. bonus points. } Assignment 1: Should be out this weekend. } Mid-term: Before the last lecture. } Mid-term deferred exam: Marks } Assignment 1: Should be out this weekend } All are marked, I m trying to tally them and perhaps add bonus points } Mid-term: Before the last lecture } Mid-term deferred exam: } This Saturday, 9am-10.30am,

More information

Kalman Based Temporal Difference Neural Network for Policy Generation under Uncertainty (KBTDNN)

Kalman Based Temporal Difference Neural Network for Policy Generation under Uncertainty (KBTDNN) Kalman Based Temporal Difference Neural Network for Policy Generation under Uncertainty (KBTDNN) Alp Sardag and H.Levent Akin Bogazici University Department of Computer Engineering 34342 Bebek, Istanbul,

More information

Convergence of reinforcement learning algorithms and acceleration of learning

Convergence of reinforcement learning algorithms and acceleration of learning PHYSICAL REVIEW E 67, 026706 2003 Convergence of reinforcement learning algorithms and acceleration of learning A. Potapov* and M. K. Ali Department of Physics, The University of Lethbridge, 4401 University

More information

Off-Policy Actor-Critic

Off-Policy Actor-Critic Off-Policy Actor-Critic Ludovic Trottier Laval University July 25 2012 Ludovic Trottier (DAMAS Laboratory) Off-Policy Actor-Critic July 25 2012 1 / 34 Table of Contents 1 Reinforcement Learning Theory

More information