Supplementary Material: Control of Memory, Active Perception, and Action in Minecraft
|
|
|
- Walter Powell
- 5 years ago
- Views:
Transcription
1 Supplementary Material: Control of Memory, Active Perception, and Action in Minecraft Junhyuk Oh Valliappa Chockalingam Satinder Singh Honglak Lee Computer Science & Engineering, University of Michigan A. Implementation Details A.1. Hyperparameters For all architectures, the first convolution layer consists of 32, 4 4, filters with a stride of 2 and a padding of 1. The second convolution layer consists of 64, 4 4, filters with a stride of 2 and a padding of 1. In Deep Q-Learning, batch size of 32 and discount factor of 0.99 are used. We used a replay memory size of 10 6 for random mazes and for I-Maze and Pattern Matching tasks. We linearly interpolated ɛ from 1 to 0.1 for the initial 10 6 steps in the ɛ-greedy policy. We chose the best learning rate from {001, 0025, 005, 01} that does not lead to value function explosion depending on the tasks and architectures. The chosen learning rates are shown in Table 1. The parameter is updated after every 4 steps. RMSProp was used with a momentum of 0.95 and a momentum of squared gradients of Gradients were clipped at l2- norm of 20 to prevent divergence. We used soft target Q- network updates with a momentum of as suggested by (Lillicrap et al., 2016). Table 1. Learning rates. TASK DQN DRQN MQN RMQN FRMQN I-MAZE MATCHING SINGLE SEQ SINGLE+I SEQ+I A.2. Map Generation for Pattern Matching There are a total of 512 possible visual patterns in a 3 3 room with blocks of two colors. We randomly picked 250 patterns and generated two maps for each pattern: one that contains the same pattern in two rooms and another that has a different randomly generated pattern in one of the rooms that is randomly selected. This produces 500 maps, 250 with identical rooms, and 250 with different rooms, which are used for training. For evaluating generalization, we picked another exclusive set of 250 visual patterns, and generated 500 maps by following the same procedure.
2 (a) I-Maze (Train) (b) I-Maze (Unseen) (c) Pattern Matching (Train) (d) Pattern Matching (Unseen) 0.5 DQN DRQN MQN RMQN FRMQN (e) Single (Train) (f) Single (Unseen) (g) Single (Unseen-L) 0.5 (h) Sequential (Train) (i) Sequential (Unseen) (j) Sequential (Unseen-L) (k) Single+I (Train) (l) Single+I (Unseen) (m) Single+I (Unseen-L) 0.5 (n) Sequential+I (Train) (o) Sequential+I (Unseen) (p) Sequential+I (Unseen-L) Figure 1. Learning curves. X-axis and y-axis correspond to the number of training epochs (1 epoch = 10K steps) and the average reward respectively. For I-Maze, Unseen represents unseen maps with different sizes. For Pattern Matching, Unseen represents maps with different visual patterns. For the rest plots, Unseen and Unseen-L indicate unseen topologies with the same sizes and larger sizes of maps, respectively. The performance was measured from 4 runs for random mazes and 10 runs for I-Maze and Pattern Matching.
3 t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 t = 32 t = 33 t = 34 t = 35 t = 36 Figure 2. FRMQN s play on an unseen and larger I-maze. The agent successfully completes the task by visiting the red block given the yellow indicator.
4 t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7 t = 8 t = 9 t = 10 Figure 3. FRMQN s play on a Pattern Matching task. The agent starts by looking at one room and then turns twice to look at the other room. Upon observing the two rooms, the agent uses backward actions repeatedly to move along the vertical corridor. Finally, once it is at the end of the corridor, it decides to turn and move forward to the blue block as the visual patterns of the rooms were identical. Note that the agent s performance is near optimal. t = 1 t = 2 t = 3 t = 4 t = 5 t = 6 t = 7 t = 8 t = 9 Figure 4. FRMQN s play on a Pattern Matching task. The agent successfully goes to the red goal, given that the visual patterns of the two rooms were different. t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 Figure 5. FRMQN s play on an unseen random maze with Single Goal task. As the case with most of the tasks, the agent starts by looking down quickly to see the important stimuli (e.g., goal blocks and indicators) more clearly. The agent then looks around it s vicinity and explores corridors. As soon as the agent sees the blue block, it goes to the block and successfully complete the task.
5 t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 t = 32 t = 33 Figure 6. FRMQN s play in an unseen and larger random maze with Single Goal task. The agent explores the map from the top-left side to top-right side, and successfully finds and visits the blue block.
6 t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 t = 32 t = 33 t = 34 t = 35 t = 36 t = 49 t = 50 t = 51 t = 52 t = 53 t = 54 t = 55 t = 56 t = 57 t = 58 t = 59 t = 60 t = 73 t = 74 t = 75 t = 76 t = 77 t = 78 t = 79 t = 80 t = 81 t = 82 t = 83 t = 84 t = 97 t = 98 t = 99 t = 100 Figure 7. FRMQN s play in an unseen and larger random maze with Single Goal task. Even though the agent explores the entire map in a reasonable way, it fails to find the blue block within 100 steps. This can occur occasionally, especially when the map is quite large and intrinsically complex (e.g., contains many walls giving rise to deep partial observability).
7 t = 13 t = 14 t = 15 t = 16 t = 17 t = 18 t = 19 t = 20 t = 21 t = 22 Figure 8. FRMQN s play on a random maze with Sequential Goals task. As the case with most of the other tasks, the agent begins by looking down. It then looks around for the red block (t=1-6). Upon finding the red block, it visits it (t=13). The agents then looks for the blue block (t=14-18), successfully finds it, and hence complete the sequence and task (t=22). Notably the agent does not keep searching for the red block after visiting the red block. This is where memory can be crucial. t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 Figure 9. FRMQN s play on an unseen and larger random maze with Sequential Goals task. With the context of the visual observations containing the blue block (t=5,15,16,17), the agent avoids the blue block and keeps searching for the red block based on its memory because it has not visited the red block. After finding and visiting the red block (t=28), it directly goes to the blue block (t=31), completing the sequence in the correct order, and hence the task.
8 t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 t = 32 t = 33 t = 34 t = 35 t = 36 t = 49 t = 50 t = 51 t = 52 t = 53 t = 54 t = 55 t = 56 t = 57 t = 58 t = 59 t = 60 t = 73 t = 74 t = 75 t = 76 t = 77 t = 78 t = 79 t = 80 t = 81 t = 82 t = 83 t = 84 t = 85 t = 86 t = 87 t = 88 t = 89 t = 90 t = 91 t = 92 t = 93 t = 94 t = 95 t = 96 t = 97 t = 98 t = 99 t = 100 Figure 10. FRMQN s play on an unseen and larger random maze with Sequential Goals task. At t=25, the agent finds both the red and blue blocks. However, visiting the red block (the first goal in the task) by following the corridor found would lead to visiting the blue block, and result in a negative reward. Thus, the agent attempts to search for another route to reach the red goal (t=49-100). But, the agent fails to find another route within the time limit (t=100).
9 t = 25 Figure 11. FRMQN s play in a random maze with Single Goal with Indicator task. Upon seeing that the indicator is green in color (t=2), the agent proceeds to explore the map. During its search, it comes across a corridor with a red block (t=11). This is where memory helps. The agent avoids the red block (having observed that the indicator is green). From this we can infer that the agent utilizes its memory appropriately. Later, it successfully completes the task by finding and visiting the blue block (t=22-25). t = 13 t = 14 t = 15 t = 16 t = 17 t = 18 t = 20 t = 21 Figure 12. FRMQN s play in a random maze with Single Goal with Indicator task. The agent visits the red block correctly, given the yellow indicator.
10 t = 13 t = 14 t = 15 t = 16 t = 17 t = 18 t = 19 t = 20 t = 21 t = 22 t = 23 Figure 13. FRMQN s play in an unseen and larger random maze with Single Goal with Indicator task. While the correct goal is far from the indicator, the agent is able to memorize the color of the indicator observed at the beginning (t=3) and visits the correct goal (t=23). Figure 14. FRMQN s play in a random maze with Sequential Goals with Indicator task. The agent observes that the indicator is yellow at t=3. The agent can see that the red block is near at t=4. Since the task is to first visit the blue block and then the red block if the indicator is yellow, it avoids moving towards the red block (t=6). The agent proceeds by turning and exploring some other corridors. Finally, it gets a glimpse of both the blue and red block (t=14). Using this visual observation along with retrieving from memory visual observations with the indicator present, the agent correctly goes to the blue block and then proceeds to the red block, hence completing the task (t=15-24).
11 t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 t = 32 t = 33 t = 34 t = 35 t = 36 t = 37 t = 38 t = 39 t = 40 t = 41 t = 42 t = 43 t = 44 t = 45 t = 46 t = 47 t = 48 t = 49 Figure 15. FRMQN s play in an unseen and larger random maze with Sequential Goals with Indicator task. The agent visits the red block first (t=19), given the green indicator (t=3). The agent looks for the blue block which is far apart from the red block (t=19-47). Once the agent finds the blue block, it finishes the task by completing the sequence (t=49).
12 t = 25 t = 26 t = 27 t = 28 t = 29 t = 30 t = 31 t = 32 t = 33 t = 34 t = 35 t = 36 t = 37 t = 38 t = 39 t = 40 t = 41 t = 42 t = 43 t = 44 t = 45 t = 46 t = 47 t = 48 t = 49 t = 50 Figure 16. FRMQN s play in a random maze with Sequential Goals with Indicator task. Given that the indicator is green, the agent has to visit the red block first and the blue block later. For this reason, the agent tries to avoid the blue block and search for another route to the red block (t=4-50). However, since there is no path to the red block (that avoids the blue block), the agent keeps searching until the episode terminates.
13 References Supplementary Material: Control of Memory, Active Perception, and Action in Minecraft Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander, Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David, and Wierstra, Daan. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Multiagent (Deep) Reinforcement Learning
Multiagent (Deep) Reinforcement Learning MARTIN PILÁT (MARTIN.PILAT@MFF.CUNI.CZ) Reinforcement learning The agent needs to learn to perform tasks in environment No prior knowledge about the effects of
Introduction to Reinforcement Learning
CSCI-699: Advanced Topics in Deep Learning 01/16/2019 Nitin Kamra Spring 2019 Introduction to Reinforcement Learning 1 What is Reinforcement Learning? So far we have seen unsupervised and supervised learning.
arxiv: v1 [cs.ne] 4 Dec 2015
![arxiv: v1 [cs.ne] 4 Dec 2015](/thumbs/74/70189552.jpg "arxiv: v1 [cs.ne] 4 Dec 2015")
Q-Networks for Binary Vector Actions arxiv:1512.01332v1 [cs.ne] 4 Dec 2015 Naoto Yoshida Tohoku University Aramaki Aza Aoba 6-6-01 Sendai 980-8579, Miyagi, Japan naotoyoshida@pfsl.mech.tohoku.ac.jp Abstract
Reinforcement Learning and Control
CS9 Lecture notes Andrew Ng Part XIII Reinforcement Learning and Control We now begin our study of reinforcement learning and adaptive control. In supervised learning, we saw algorithms that tried to make
Deep Q-Learning with Recurrent Neural Networks
Deep Q-Learning with Recurrent Neural Networks Clare Chen cchen9@stanford.edu Vincent Ying vincenthying@stanford.edu Dillon Laird dalaird@cs.stanford.edu Abstract Deep reinforcement learning models have
Q-Learning in Continuous State Action Spaces
Q-Learning in Continuous State Action Spaces Alex Irpan alexirpan@berkeley.edu December 5, 2015 Contents 1 Introduction 1 2 Background 1 3 Q-Learning 2 4 Q-Learning In Continuous Spaces 4 5 Experimental
CS230: Lecture 9 Deep Reinforcement Learning
CS230: Lecture 9 Deep Reinforcement Learning Kian Katanforoosh Menti code: 21 90 15 Today s outline I. Motivation II. Recycling is good: an introduction to RL III. Deep Q-Learning IV. Application of Deep
Human-level control through deep reinforcement. Liia Butler
Humanlevel control through deep reinforcement Liia Butler But first... A quote "The question of whether machines can think... is about as relevant as the question of whether submarines can swim" Edsger
Supplementary Material for Asynchronous Methods for Deep Reinforcement Learning
Supplementary Material for Asynchronous Methods for Deep Reinforcement Learning May 2, 216 1 Optimization Details We investigated two different optimization algorithms with our asynchronous framework stochastic
Deep reinforcement learning. Dialogue Systems Group, Cambridge University Engineering Department
Deep reinforcement learning Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 25 In this lecture... Introduction to deep reinforcement learning Value-based Deep RL Deep
(Deep) Reinforcement Learning
Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 17 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015
Neural Map. Structured Memory for Deep RL. Emilio Parisotto
Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student Machine Learning Department Carnegie Mellon University Supervised Learning Most deep learning problems are
INF 5860 Machine learning for image classification. Lecture 14: Reinforcement learning May 9, 2018
Machine learning for image classification Lecture 14: Reinforcement learning May 9, 2018 Page 3 Outline Motivation Introduction to reinforcement learning (RL) Value function based methods (Q-learning)
Variational Deep Q Network
Variational Deep Q Network Yunhao Tang Department of IEOR Columbia University yt2541@columbia.edu Alp Kucukelbir Department of Computer Science Columbia University alp@cs.columbia.edu Abstract We propose
arxiv: v2 [cs.lg] 7 Mar 2018
![arxiv: v2 [cs.lg] 7 Mar 2018](/thumbs/76/73786584.jpg "arxiv: v2 [cs.lg] 7 Mar 2018")
Comparing Deep Reinforcement Learning and Evolutionary Methods in Continuous Control arxiv:1712.00006v2 [cs.lg] 7 Mar 2018 Shangtong Zhang 1, Osmar R. Zaiane 2 12 Dept. of Computing Science, University
1 Introduction 2. 4 Q-Learning The Q-value The Temporal Difference The whole Q-Learning process... 5
Table of contents 1 Introduction 2 2 Markov Decision Processes 2 3 Future Cumulative Reward 3 4 Q-Learning 4 4.1 The Q-value.............................................. 4 4.2 The Temporal Difference.......................................
Deep Reinforcement Learning via Policy Optimization
Deep Reinforcement Learning via Policy Optimization John Schulman July 3, 2017 Introduction Deep Reinforcement Learning: What to Learn? Policies (select next action) Deep Reinforcement Learning: What to
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
Dueling Network Architectures for Deep Reinforcement Learning (ICML 2016)
Dueling Network Architectures for Deep Reinforcement Learning (ICML 2016) Yoonho Lee Department of Computer Science and Engineering Pohang University of Science and Technology October 11, 2016 Outline
Approximate Q-Learning. Dan Weld / University of Washington
Approximate Q-Learning Dan Weld / University of Washington [Many slides taken from Dan Klein and Pieter Abbeel / CS188 Intro to AI at UC Berkeley materials available at http://ai.berkeley.edu.] Q Learning
arxiv: v1 [cs.lg] 5 Dec 2015
![arxiv: v1 [cs.lg] 5 Dec 2015](/thumbs/82/85482236.jpg "arxiv: v1 [cs.lg] 5 Dec 2015")
Deep Attention Recurrent Q-Network Ivan Sorokin, Alexey Seleznev, Mikhail Pavlov, Aleksandr Fedorov, Anastasiia Ignateva 5vision 5visionteam@gmail.com arxiv:1512.1693v1 [cs.lg] 5 Dec 215 Abstract A deep
Sampling diverse neural networks for exploration in reinforcement learning
Sampling diverse neural networks for exploration in reinforcement learning Maxime Wabartha School of Computer Science McGill University maxime.wabartha@mail.mcgill.ca Vincent François-Lavet School of Computer
Policy Gradient Methods. February 13, 2017
Policy Gradient Methods February 13, 2017 Policy Optimization Problems maximize E π [expression] π Fixed-horizon episodic: T 1 Average-cost: lim T 1 T r t T 1 r t Infinite-horizon discounted: γt r t Variable-length
Reinforcement Learning
Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns Forward and backward view Function
arxiv: v2 [cs.lg] 2 Oct 2018
![arxiv: v2 [cs.lg] 2 Oct 2018](/thumbs/87/96205093.jpg "arxiv: v2 [cs.lg] 2 Oct 2018")
AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control arxiv:171.4423v2 [cs.lg] 2 Oct 218 Seungyul Han Dept. of Electrical Engineering KAIST Daejeon, South Korea 34141 sy.han@kaist.ac.kr
Machine Learning and Bayesian Inference. Unsupervised learning. Can we find regularity in data without the aid of labels?
Machine Learning and Bayesian Inference Dr Sean Holden Computer Laboratory, Room FC6 Telephone extension 6372 Email: sbh11@cl.cam.ac.uk www.cl.cam.ac.uk/ sbh11/ Unsupervised learning Can we find regularity
Learning Control for Air Hockey Striking using Deep Reinforcement Learning
Learning Control for Air Hockey Striking using Deep Reinforcement Learning Ayal Taitler, Nahum Shimkin Faculty of Electrical Engineering Technion - Israel Institute of Technology May 8, 2017 A. Taitler,
Machine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
Reinforcement Learning Part 2
Reinforcement Learning Part 2 Dipendra Misra Cornell University dkm@cs.cornell.edu https://dipendramisra.wordpress.com/ From previous tutorial Reinforcement Learning Exploration No supervision Agent-Reward-Environment
Reinforcement Learning as Variational Inference: Two Recent Approaches
Reinforcement Learning as Variational Inference: Two Recent Approaches Rohith Kuditipudi Duke University 11 August 2017 Outline 1 Background 2 Stein Variational Policy Gradient 3 Soft Q-Learning 4 Closing
CMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING
ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control
ECE276B: Planning & Learning in Robotics Lecture 16: Model-free Control Lecturer: Nikolay Atanasov: natanasov@ucsd.edu Teaching Assistants: Tianyu Wang: tiw161@eng.ucsd.edu Yongxi Lu: yol070@eng.ucsd.edu
Lecture 23: Reinforcement Learning
Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:
Learning in State-Space Reinforcement Learning CIS 32
Learning in State-Space Reinforcement Learning CIS 32 Functionalia Syllabus Updated: MIDTERM and REVIEW moved up one day. MIDTERM: Everything through Evolutionary Agents. HW 2 Out - DUE Sunday before the
Reinforcement Learning
Reinforcement Learning From the basics to Deep RL Olivier Sigaud ISIR, UPMC + INRIA http://people.isir.upmc.fr/sigaud September 14, 2017 1 / 54 Introduction Outline Some quick background about discrete
TTIC 31230, Fundamentals of Deep Learning David McAllester, April Vanishing and Exploding Gradients. ReLUs. Xavier Initialization
TTIC 31230, Fundamentals of Deep Learning David McAllester, April 2017 Vanishing and Exploding Gradients ReLUs Xavier Initialization Batch Normalization Highway Architectures: Resnets, LSTMs and GRUs Causes
Chapter 6: Temporal Difference Learning
Chapter 6: emporal Difference Learning Objectives of this chapter: Introduce emporal Difference (D) learning Focus first on policy evaluation, or prediction, methods hen extend to control methods R. S.
Statistical Machine Learning
Statistical Machine Learning Lecture 9 Numerical optimization and deep learning Niklas Wahlström Division of Systems and Control Department of Information Technology Uppsala University niklas.wahlstrom@it.uu.se
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning Lei Lei Ruoxuan Xiong December 16, 2017 1 Introduction Deep Neural Network
Payments System Design Using Reinforcement Learning: A Progress Report
Payments System Design Using Reinforcement Learning: A Progress Report A. Desai 1 H. Du 1 R. Garratt 2 F. Rivadeneyra 1 1 Bank of Canada 2 University of California Santa Barbara 16th Payment and Settlement
Q-learning. Tambet Matiisen
Q-learning Tambet Matiisen (based on chapter 11.3 of online book Artificial Intelligence, foundations of computational agents by David Poole and Alan Mackworth) Stochastic gradient descent Experience
Artificial Neural Networks Examination, June 2005
Artificial Neural Networks Examination, June 2005 Instructions There are SIXTY questions. (The pass mark is 30 out of 60). For each question, please select a maximum of ONE of the given answers (either
Temporal Difference. Learning KENNETH TRAN. Principal Research Engineer, MSR AI
Temporal Difference Learning KENNETH TRAN Principal Research Engineer, MSR AI Temporal Difference Learning Policy Evaluation Intro to model-free learning Monte Carlo Learning Temporal Difference Learning
Administration. CSCI567 Machine Learning (Fall 2018) Outline. Outline. HW5 is available, due on 11/18. Practice final will also be available soon.
Administration CSCI567 Machine Learning Fall 2018 Prof. Haipeng Luo U of Southern California Nov 7, 2018 HW5 is available, due on 11/18. Practice final will also be available soon. Remaining weeks: 11/14,
Introduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning II Daniel Hennes 11.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns
Lecture 10 - Planning under Uncertainty (III)
Lecture 10 - Planning under Uncertainty (III) Jesse Hoey School of Computer Science University of Waterloo March 27, 2018 Readings: Poole & Mackworth (2nd ed.)chapter 12.1,12.3-12.9 1/ 34 Reinforcement
The Predictron: End-To-End Learning and Planning
David Silver * 1 Hado van Hasselt * 1 Matteo Hessel * 1 Tom Schaul * 1 Arthur Guez * 1 Tim Harley 1 Gabriel Dulac-Arnold 1 David Reichert 1 Neil Rabinowitz 1 Andre Barreto 1 Thomas Degris 1 Abstract One
Forward Actor-Critic for Nonlinear Function Approximation in Reinforcement Learning
Forward Actor-Critic for Nonlinear Function Approximation in Reinforcement Learning Vivek Veeriah Dept. of Computing Science University of Alberta Edmonton, Canada vivekveeriah@ualberta.ca Harm van Seijen
PART A and ONE question from PART B; or ONE question from PART A and TWO questions from PART B.
Advanced Topics in Machine Learning, GI13, 2010/11 Advanced Topics in Machine Learning, GI13, 2010/11 Answer any THREE questions. Each question is worth 20 marks. Use separate answer books Answer any THREE
Reinforcement Learning
Reinforcement Learning Function approximation Mario Martin CS-UPC May 18, 2018 Mario Martin (CS-UPC) Reinforcement Learning May 18, 2018 / 65 Recap Algorithms: MonteCarlo methods for Policy Evaluation
Do not tear exam apart!
6.036: Final Exam: Fall 2017 Do not tear exam apart! This is a closed book exam. Calculators not permitted. Useful formulas on page 1. The problems are not necessarily in any order of di culty. Record
arxiv: v5 [cs.lg] 9 Sep 2016
![arxiv: v5 [cs.lg] 9 Sep 2016](/thumbs/80/81962840.jpg "arxiv: v5 [cs.lg] 9 Sep 2016")
HIGH-DIMENSIONAL CONTINUOUS CONTROL USING GENERALIZED ADVANTAGE ESTIMATION John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan and Pieter Abbeel Department of Electrical Engineering and Computer
Serious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks. Cannot approximate (learn) non-linear functions
BACK-PROPAGATION NETWORKS Serious limitations of (single-layer) perceptrons: Cannot learn non-linearly separable tasks Cannot approximate (learn) non-linear functions Difficult (if not impossible) to design
CSC321 Lecture 22: Q-Learning
CSC321 Lecture 22: Q-Learning Roger Grosse Roger Grosse CSC321 Lecture 22: Q-Learning 1 / 21 Overview Second of 3 lectures on reinforcement learning Last time: policy gradient (e.g. REINFORCE) Optimize
, and rewards and transition matrices as shown below:
CSE 50a. Assignment 7 Out: Tue Nov Due: Thu Dec Reading: Sutton & Barto, Chapters -. 7. Policy improvement Consider the Markov decision process (MDP) with two states s {0, }, two actions a {0, }, discount
Reinforcement Learning. Donglin Zeng, Department of Biostatistics, University of North Carolina
Reinforcement Learning Introduction Introduction Unsupervised learning has no outcome (no feedback). Supervised learning has outcome so we know what to predict. Reinforcement learning is in between it
Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions
2018 IEEE International Workshop on Machine Learning for Signal Processing (MLSP 18) Recurrent Neural Networks with Flexible Gates using Kernel Activation Functions Authors: S. Scardapane, S. Van Vaerenbergh,
Introduction to Reinforcement Learning. Part 5: Temporal-Difference Learning
Introduction to Reinforcement Learning Part 5: emporal-difference Learning What everybody should know about emporal-difference (D) learning Used to learn value functions without human input Learns a guess
REINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
An Off-policy Policy Gradient Theorem Using Emphatic Weightings
An Off-policy Policy Gradient Theorem Using Emphatic Weightings Ehsan Imani, Eric Graves, Martha White Reinforcement Learning and Artificial Intelligence Laboratory Department of Computing Science University
Learning to Query, Reason, and Answer Questions On Ambiguous Texts
Learning to Query, Reason, and Answer Questions On Ambiguous Texts Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Joseph P. Bigus, Murray Campbell, Ban Kawas, Kartik Talamadupula, Gerald Tesauro, Satinder
David Silver, Google DeepMind
Tutorial: Deep Reinforcement Learning David Silver, Google DeepMind Outline Introduction to Deep Learning Introduction to Reinforcement Learning Value-Based Deep RL Policy-Based Deep RL Model-Based Deep
An application of the temporal difference algorithm to the truck backer-upper problem
ESANN 214 proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. Bruges (Belgium), 23-25 April 214, i6doc.com publ., ISBN 978-28741995-7. Available
Trust Region Policy Optimization
Trust Region Policy Optimization Yixin Lin Duke University yixin.lin@duke.edu March 28, 2017 Yixin Lin (Duke) TRPO March 28, 2017 1 / 21 Overview 1 Preliminaries Markov Decision Processes Policy iteration
Chapter 6: Temporal Difference Learning
Chapter 6: emporal Difference Learning Objectives of this chapter: Introduce emporal Difference (D) learning Focus first on policy evaluation, or prediction, methods Compare efficiency of D learning with
Today s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes
Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks
Christopher Watkins and Peter Dayan. Noga Zaslavsky. The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015
Q-Learning Christopher Watkins and Peter Dayan Noga Zaslavsky The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (67679) November 1, 2015 Noga Zaslavsky Q-Learning (Watkins & Dayan, 1992)
Convolutional Neural Networks
Convolutional Neural Networks Books» http://www.deeplearningbook.org/ Books http://neuralnetworksanddeeplearning.com/.org/ reviews» http://www.deeplearningbook.org/contents/linear_algebra.html» http://www.deeplearningbook.org/contents/prob.html»
REGRET MINIMIZATION FOR PARTIALLY OBSERVABLE DEEP REINFORCEMENT LEARNING
REGRET MINIMIZATION FOR PARTIALLY OBSERVABLE DEEP REINFORCEMENT LEARNING Peter Jin, Sergey Levine, and Kurt Keutzer Berkeley AI Research Lab, Department of Electrical Engineering and Computer Sciences
Monte Carlo is important in practice. CSE 190: Reinforcement Learning: An Introduction. Chapter 6: Temporal Difference Learning.
Monte Carlo is important in practice CSE 190: Reinforcement Learning: An Introduction Chapter 6: emporal Difference Learning When there are just a few possibilitieo value, out of a large state space, Monte
CS181 Practical4 Reinforcement Learning
CS181 Practical4 Reinforcement Learning Ben Cook and Scott Shaffer April 2016 1 Technical Approach 1.1 Basic Assumptions The general approach to solving the Swingy Monkey game is to directly approximate
Basics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
Reinforcement. Function Approximation. Learning with KATJA HOFMANN. Researcher, MSR Cambridge
Reinforcement Learning with Function Approximation KATJA HOFMANN Researcher, MSR Cambridge Representation and Generalization in RL Focus on training stability Learning generalizable value functions Navigating
Reinforcement Learning and NLP
1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value
Hidden Markov Models (HMM) and Support Vector Machine (SVM)
Hidden Markov Models (HMM) and Support Vector Machine (SVM) Professor Joongheon Kim School of Computer Science and Engineering, Chung-Ang University, Seoul, Republic of Korea 1 Hidden Markov Models (HMM)
Chapter 3: The Reinforcement Learning Problem
Chapter 3: The Reinforcement Learning Problem Objectives of this chapter: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which
Deep Reinforcement Learning SISL. Jeremy Morton (jmorton2) November 7, Stanford Intelligent Systems Laboratory
Deep Reinforcement Learning Jeremy Morton (jmorton2) November 7, 2016 SISL Stanford Intelligent Systems Laboratory Overview 2 1 Motivation 2 Neural Networks 3 Deep Reinforcement Learning 4 Deep Learning
Reinforcement Learning
Reinforcement Learning RL in continuous MDPs March April, 2015 Large/Continuous MDPs Large/Continuous state space Tabular representation cannot be used Large/Continuous action space Maximization over action
Multi-Batch Experience Replay for Fast Convergence of Continuous Action Control
1 Multi-Batch Experience Replay for Fast Convergence of Continuous Action Control Seungyul Han and Youngchul Sung Abstract arxiv:1710.04423v1 [cs.lg] 12 Oct 2017 Policy gradient methods for direct policy
15-889e Policy Search: Gradient Methods Emma Brunskill. All slides from David Silver (with EB adding minor modificafons), unless otherwise noted
15-889e Policy Search: Gradient Methods Emma Brunskill All slides from David Silver (with EB adding minor modificafons), unless otherwise noted Outline 1 Introduction 2 Finite Difference Policy Gradient
Dual Memory Model for Using Pre-Existing Knowledge in Reinforcement Learning Tasks
Dual Memory Model for Using Pre-Existing Knowledge in Reinforcement Learning Tasks Kary Främling Helsinki University of Technology, PL 55, FI-25 TKK, Finland Kary.Framling@hut.fi Abstract. Reinforcement
Markov Decision Processes and Solving Finite Problems. February 8, 2017
Markov Decision Processes and Solving Finite Problems February 8, 2017 Overview of Upcoming Lectures Feb 8: Markov decision processes, value iteration, policy iteration Feb 13: Policy gradients Feb 15:
Need for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
CS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
arxiv: v1 [cs.ai] 5 Nov 2017
![arxiv: v1 [cs.ai] 5 Nov 2017](/thumbs/96/129180284.jpg "arxiv: v1 [cs.ai] 5 Nov 2017")
arxiv:1711.01569v1 [cs.ai] 5 Nov 2017 Markus Dumke Department of Statistics Ludwig-Maximilians-Universität München markus.dumke@campus.lmu.de Abstract Temporal-difference (TD) learning is an important
arxiv: v1 [cs.ai] 19 May 2017
![arxiv: v1 [cs.ai] 19 May 2017](/thumbs/74/70608829.jpg "arxiv: v1 [cs.ai] 19 May 2017")
Model-Based Planning in Discrete Action Spaces Mikael Henaff Courant Institute, New York University mbh305@nyu.edu William F. Whitney Courant Institute, New York University ww1114@nyu.edu arxiv:1705.07177v1
SAMPLE-EFFICIENT DEEP REINFORCEMENT LEARN-
SAMPLE-EFFICIENT DEEP REINFORCEMENT LEARN- ING VIA EPISODIC BACKWARD UPDATE Anonymous authors Paper under double-blind review ABSTRACT We propose Episodic Backward Update - a new algorithm to boost the
Combining PPO and Evolutionary Strategies for Better Policy Search
Combining PPO and Evolutionary Strategies for Better Policy Search Jennifer She 1 Abstract A good policy search algorithm needs to strike a balance between being able to explore candidate policies and
Comparing Deep Reinforcement Learning and Evolutionary Methods in Continuous Control
Comparing Deep Reinforcement Learning and Evolutionary Methods in Continuous Control Shangtong Zhang Dept. of Computing Science University of Alberta shangtong.zhang@ualberta.ca Osmar R. Zaiane Dept. of
Reinforcement Learning
1 Reinforcement Learning Chris Watkins Department of Computer Science Royal Holloway, University of London July 27, 2015 2 Plan 1 Why reinforcement learning? Where does this theory come from? Markov decision
Visual meta-learning for planning and control
Visual meta-learning for planning and control Seminar on Current Works in Computer Vision @ Chair of Pattern Recognition and Image Processing. Samuel Roth Winter Semester 2018/19 Albert-Ludwigs-Universität
The Predictron: End-To-End Learning and Planning
David Silver * 1 Hado van Hasselt * 1 Matteo Hessel * 1 Tom Schaul * 1 Arthur Guez * 1 Tim Harley 1 Gabriel Dulac-Arnold 1 David Reichert 1 Neil Rabinowitz 1 Andre Barreto 1 Thomas Degris 1 We applied
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation
Lecture 3: Policy Evaluation Without Knowing How the World Works / Model Free Policy Evaluation CS234: RL Emma Brunskill Winter 2018 Material builds on structure from David SIlver s Lecture 4: Model-Free
Deep Reinforcement Learning
Martin Matyášek Artificial Intelligence Center Czech Technical University in Prague October 27, 2016 Martin Matyášek VPD, 2016 1 / 50 Reinforcement Learning in a picture R. S. Sutton and A. G. Barto 2015
Reinforcement Learning. Spring 2018 Defining MDPs, Planning
Reinforcement Learning Spring 2018 Defining MDPs, Planning understandability 0 Slide 10 time You are here Markov Process Where you will go depends only on where you are Markov Process: Information state
Driving a car with low dimensional input features
December 2016 Driving a car with low dimensional input features Jean-Claude Manoli jcma@stanford.edu Abstract The aim of this project is to train a Deep Q-Network to drive a car on a race track using only
Be able to define the following terms and answer basic questions about them:
CS440/ECE448 Section Q Fall 2017 Final Review Be able to define the following terms and answer basic questions about them: Probability o Random variables, axioms of probability o Joint, marginal, conditional
Reinforcement Learning with Function Approximation. Joseph Christian G. Noel
Reinforcement Learning with Function Approximation Joseph Christian G. Noel November 2011 Abstract Reinforcement learning (RL) is a key problem in the field of Artificial Intelligence. The main goal is
Lecture 3: The Reinforcement Learning Problem
Lecture 3: The Reinforcement Learning Problem Objectives of this lecture: describe the RL problem we will be studying for the remainder of the course present idealized form of the RL problem for which