Learning to Query, Reason, and Answer Questions On Ambiguous Texts
|
|
|
- Norman Bell
- 5 years ago
- Views:
Transcription
1 Learning to Query, Reason, and Answer Questions On Ambiguous Texts Xiaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Joseph P. Bigus, Murray Campbell, Ban Kawas, Kartik Talamadupula, Gerald Tesauro, Satinder Singh University of Michigan, IBM Research, UMass Amherst ICLR 2017 Presenter: Jack Lanchantin ICLR 2017 Presenter: Jack Lanchantin 1
2 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 2
3 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 3
4 Motivation Human conversation is incomplete, ambiguous and full of extraneous detail Conversational agents must be able to reason in the presence of missing or unclear info ICLR 2017 Presenter: Jack Lanchantin 4
5 Previous Datasets babi & children s book: answer questions about short stories Task-oriented dialog systems: answer questions to find restaurants or movies (slot filling) ICLR 2017 Presenter: Jack Lanchantin 5
6 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 6
7 QRAQ (Query, Reason, and Answer Questions) Dataset 1 Simulator provides a story and a question to the agent, with some of the entities replaced by variables ICLR 2017 Presenter: Jack Lanchantin 7
8 QRAQ (Query, Reason, and Answer Questions) Dataset 1 Simulator provides a story and a question to the agent, with some of the entities replaced by variables 2 The agent must be able to decide whether it has enough info to answer the question ICLR 2017 Presenter: Jack Lanchantin 7
9 QRAQ (Query, Reason, and Answer Questions) Dataset 1 Simulator provides a story and a question to the agent, with some of the entities replaced by variables 2 The agent must be able to decide whether it has enough info to answer the question 3 If the agent cannot answer the question by reasoning alone, it must learn to query the simulator for a variable value ICLR 2017 Presenter: Jack Lanchantin 7
10 QRAQ Problem C 1, C 2,... : Context E 1, E 2,... : Events Q: Question GT : Ground Truth ICLR 2017 Presenter: Jack Lanchantin 8
11 QRAQ Query Graph ICLR 2017 Presenter: Jack Lanchantin 9
12 QRAQ Query Graph ICLR 2017 Presenter: Jack Lanchantin 10
13 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 11
14 Control Loop 1 Initialization: Question vector c and memory matrix S 0 are initialized ICLR 2017 Presenter: Jack Lanchantin 12
15 Control Loop 1 Initialization: Question vector c and memory matrix S 0 are initialized 2 Action Selection: Policy π(a S t, c) maps memory matrix S t and question c, into a distribution over actions. Action a could be either a query for a variable or a final answer ICLR 2017 Presenter: Jack Lanchantin 12
16 Control Loop 1 Initialization: Question vector c and memory matrix S 0 are initialized 2 Action Selection: Policy π(a S t, c) maps memory matrix S t and question c, into a distribution over actions. Action a could be either a query for a variable or a final answer 3 Variable Query and Memory Update: If action is a query, simulator provides the true value v t for the variable in action a t. All occurrences of variable in action a t in the memory S t are replaced with the true value v t : S t+1 = S t [a t v t ] ICLR 2017 Presenter: Jack Lanchantin 12
17 Control Loop 1 Initialization: Question vector c and memory matrix S 0 are initialized 2 Action Selection: Policy π(a S t, c) maps memory matrix S t and question c, into a distribution over actions. Action a could be either a query for a variable or a final answer 3 Variable Query and Memory Update: If action is a query, simulator provides the true value v t for the variable in action a t. All occurrences of variable in action a t in the memory S t are replaced with the true value v t : S t+1 = S t [a t v t ] 4 Final Answer Generation and Termination: If the action is an answer, task terminates and a reward is generated based on correctness ICLR 2017 Presenter: Jack Lanchantin 12 iaoxiao Guo, Tim Klinger, Clemens Rosenbaum, Learning Joseph to Query, P. Bigus, Reason, Murray and Campbell, Answer Questions Ban Kawas, OnKartik Ambiguous Talamadupula, Texts Gerald Tesauro, / 29
18 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 13
19 baserl: End-to-End Memory Net Based Policy Learner Maps the memory matrix, S, and the challenge question representation, c, into an action distribution. S ij is dictionary index of j th word in the i th sentence c i is the dictionary index of the i th word in the question m i = j l j A[S ij ] (1) q = j l j A[c j ] (2) A R d N, A[k] returns the k-th column vector (sentence), d is embedding dimension, N is the dictionary size, lj k = (1 j/j)(k/d)(1 2j/J), with J being the number of words in the sentences ICLR 2017 Presenter: Jack Lanchantin 14
20 baserl: End-to-End Memory Net Based Policy Learner m i = j l j A[S ij ] (3) q = j l j A[c j ] (4) Output vector from reading {m i } after the k th hop is u k (u 0 = q): u k = tanh(h(o k + u k 1 )) (5) o k = pi k m i (6) i pi k = softmax(uk 1 T m i) (7) (8) ICLR 2017 Presenter: Jack Lanchantin 15
21 baserl: End-to-End Memory Net Based Policy Learner Query Network Output. Since the problems have at most one variable per sentence, the distribution can be converted into the distribution over sentences: π i Q = softmax(ut K m i) (9) Answer Network Output. The final output of the policy module is a distribution over potential answers: π A = softmax(wu K + b) (10) ICLR 2017 Presenter: Jack Lanchantin 16
22 baserl imprl baserl: Final action-distribution output only conditions on the last hop output imprl: computes the final action-distribution-output over all memory hop outputs ICLR 2017 Presenter: Jack Lanchantin 17
23 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 18
24 imprl: Improved End-to-End Memory Net Based Policy Learner πq i = softmax(ut m i ) (11) π A = softmax(wu + b) (12) u = z j u j j (13) z j = softmax(q T u j ) (14) ICLR 2017 Presenter: Jack Lanchantin 19
25 imprl: Improved End-to-End Memory Net Based Policy Learner ICLR 2017 Presenter: Jack Lanchantin 20
26 Outline 1 Introduction Background This paper: QRAQ 2 QRAQ 3 Model Control Loop baserl imprl Policy Gradient & Reward Function 4 Data, Training, and Results ICLR 2017 Presenter: Jack Lanchantin 21
27 Policy Gradient & Reward Function +Reward when the action is the correct answer -Reward when the action is a wrong answer -Reward when the action is a query to a variable Objective function is to optimize the expected cumulative reward over M training problem instances M m=1 E{ t r m t } (15) GPOMDP (Weaver & Tao (2001)) is used to calculate the policy gradient ICLR 2017 Presenter: Jack Lanchantin 22
28 Data 4 Types 107,000 QRAQ problems in each type 100,000 training 2,000 testing 5,000 validation ICLR 2017 Presenter: Jack Lanchantin 23
29 Data (Loc): Context and events describe locations and movements of people in rooms. Questions are about the location of a specific person (+obj): Adds objects to (Loc) (+alias): Adds aliases to (Loc) Some of them are defined in the context. (+par): Substitutes sentences with semantically equivalent paraphrases in (Loc) ICLR 2017 Presenter: Jack Lanchantin 24
30 Curriculum Learning First encourage the agent to query variables by assigning positive rewards for querying any variable After convergence under this initial reward function, switch to the true reward function that assigns a negative reward for querying a variable to reduce the number of unnecessary queries +1 for correct final answers, -5 for wrong final answers, query reward +/ ICLR 2017 Presenter: Jack Lanchantin 25
31 Supervised Learning (SL) Baseline Upper-bound on achievable reinforcement learning performance Relevant variables and (when appropriate) the correct final answers are provided at each turn, and the cross entropy is used to optimize ICLR 2017 Presenter: Jack Lanchantin 26
32 Accuracy Metrics Trajectory: a sequence of variable queries followed by an answer 1 Answer-accuracy: proportion of trajectories in which the agent answered correctly 2 Trajectory-accuracy: proportion of trajectories in which the agent queried only relevant variables then answered correctly 3 Trajectory-completeness: proportion of trajectories in which the agent queried all and only relevant variables before answering correctly 4 Query accuracy: the proportion of correct queries among all queries made in any trajectory ICLR 2017 Presenter: Jack Lanchantin 27
33 Results ICLR 2017 Presenter: Jack Lanchantin 28
34 Conclusion New dataset, QRAQ, for reasoning under insufficient information First to formulate these types of QA problems in the RL format ICLR 2017 Presenter: Jack Lanchantin 29
Reasoning with Memory Augmented Neural Networks for Language Comprehension, ICLR17
Reasoning with Memory Augmented Neural Networks for Language Comprehension, ICLR17 UMass Medical School ICLR 2017 Presenter: Jack Lanchantin ICLR 2017 Presenter: Jack Lanchantin 1 Outline 1 Intro 2 Hypothesis
Lecture 23: Reinforcement Learning
Lecture 23: Reinforcement Learning MDPs revisited Model-based learning Monte Carlo value function estimation Temporal-difference (TD) learning Exploration November 23, 2006 1 COMP-424 Lecture 23 Recall:
Reinforcement Learning and NLP
1 Reinforcement Learning and NLP Kapil Thadani kapil@cs.columbia.edu RESEARCH Outline 2 Model-free RL Markov decision processes (MDPs) Derivative-free optimization Policy gradients Variance reduction Value
REINFORCEMENT LEARNING
REINFORCEMENT LEARNING Larry Page: Where s Google going next? DeepMind's DQN playing Breakout Contents Introduction to Reinforcement Learning Deep Q-Learning INTRODUCTION TO REINFORCEMENT LEARNING Contents
brainlinksystem.com $25+ / hr AI Decision Tree Learning Part I Outline Learning 11/9/2010 Carnegie Mellon
I Decision Tree Learning Part I brainlinksystem.com $25+ / hr Illah Nourbakhsh s version Chapter 8, Russell and Norvig Thanks to all past instructors Carnegie Mellon Outline Learning and philosophy Induction
Tracking the World State with Recurrent Entity Networks
Tracking the World State with Recurrent Entity Networks Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, Yann LeCun Task At each timestep, get information (in the form of a sentence) about the
CS599 Lecture 1 Introduction To RL
CS599 Lecture 1 Introduction To RL Reinforcement Learning Introduction Learning from rewards Policies Value Functions Rewards Models of the Environment Exploitation vs. Exploration Dynamic Programming
Deep Reinforcement Learning. Scratching the surface
Deep Reinforcement Learning Scratching the surface Deep Reinforcement Learning Scenario of Reinforcement Learning Observation State Agent Action Change the environment Don t do that Reward Environment
Supplementary Material: Control of Memory, Active Perception, and Action in Minecraft
Supplementary Material: Control of Memory, Active Perception, and Action in Minecraft Junhyuk Oh Valliappa Chockalingam Satinder Singh Honglak Lee Computer Science & Engineering, University of Michigan
Online Videos FERPA. Sign waiver or sit on the sides or in the back. Off camera question time before and after lecture. Questions?
Online Videos FERPA Sign waiver or sit on the sides or in the back Off camera question time before and after lecture Questions? Lecture 1, Slide 1 CS224d Deep NLP Lecture 4: Word Window Classification
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer.
This question has three parts, each of which can be answered concisely, but be prepared to explain and justify your concise answer. 1. Suppose you have a policy and its action-value function, q, then you
Temporal Difference Learning & Policy Iteration
Temporal Difference Learning & Policy Iteration Advanced Topics in Reinforcement Learning Seminar WS 15/16 ±0 ±0 +1 by Tobias Joppen 03.11.2015 Fachbereich Informatik Knowledge Engineering Group Prof.
Machine Learning I Continuous Reinforcement Learning
Machine Learning I Continuous Reinforcement Learning Thomas Rückstieß Technische Universität München January 7/8, 2010 RL Problem Statement (reminder) state s t+1 ENVIRONMENT reward r t+1 new step r t
Topic Models and Applications to Short Documents
Topic Models and Applications to Short Documents Dieu-Thu Le Email: dieuthu.le@unitn.it Trento University April 6, 2011 1 / 43 Outline Introduction Latent Dirichlet Allocation Gibbs Sampling Short Text
Experiments on the Consciousness Prior
Yoshua Bengio and William Fedus UNIVERSITÉ DE MONTRÉAL, MILA Abstract Experiments are proposed to explore a novel prior for representation learning, which can be combined with other priors in order to
Grundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Reinforcement learning Daniel Hennes 4.12.2017 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Reinforcement learning Model based and
Reinforcement Learning as Variational Inference: Two Recent Approaches
Reinforcement Learning as Variational Inference: Two Recent Approaches Rohith Kuditipudi Duke University 11 August 2017 Outline 1 Background 2 Stein Variational Policy Gradient 3 Soft Q-Learning 4 Closing
Natural Language Processing. Slides from Andreas Vlachos, Chris Manning, Mihai Surdeanu
Natural Language Processing Slides from Andreas Vlachos, Chris Manning, Mihai Surdeanu Projects Project descriptions due today! Last class Sequence to sequence models Attention Pointer networks Today Weak
Today s s Lecture. Applicability of Neural Networks. Back-propagation. Review of Neural Networks. Lecture 20: Learning -4. Markov-Decision Processes
Today s s Lecture Lecture 20: Learning -4 Review of Neural Networks Markov-Decision Processes Victor Lesser CMPSCI 683 Fall 2004 Reinforcement learning 2 Back-propagation Applicability of Neural Networks
Open Theoretical Questions in Reinforcement Learning
Open Theoretical Questions in Reinforcement Learning Richard S. Sutton AT&T Labs, Florham Park, NJ 07932, USA, sutton@research.att.com, www.cs.umass.edu/~rich Reinforcement learning (RL) concerns the problem
Reinforcement Learning. Yishay Mansour Tel-Aviv University
Reinforcement Learning Yishay Mansour Tel-Aviv University 1 Reinforcement Learning: Course Information Classes: Wednesday Lecture 10-13 Yishay Mansour Recitations:14-15/15-16 Eliya Nachmani Adam Polyak
Introduction to Reinforcement Learning. CMPT 882 Mar. 18
Introduction to Reinforcement Learning CMPT 882 Mar. 18 Outline for the week Basic ideas in RL Value functions and value iteration Policy evaluation and policy improvement Model-free RL Monte-Carlo and
Visual meta-learning for planning and control
Visual meta-learning for planning and control Seminar on Current Works in Computer Vision @ Chair of Pattern Recognition and Image Processing. Samuel Roth Winter Semester 2018/19 Albert-Ludwigs-Universität
Final Exam December 12, 2017
Introduction to Artificial Intelligence CSE 473, Autumn 2017 Dieter Fox Final Exam December 12, 2017 Directions This exam has 7 problems with 111 points shown in the table below, and you have 110 minutes
Final Exam, Fall 2002
15-781 Final Exam, Fall 22 1. Write your name and your andrew email address below. Name: Andrew ID: 2. There should be 17 pages in this exam (excluding this cover sheet). 3. If you need more room to work
Deep Reinforcement Learning. STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017
Deep Reinforcement Learning STAT946 Deep Learning Guest Lecture by Pascal Poupart University of Waterloo October 19, 2017 Outline Introduction to Reinforcement Learning AlphaGo (Deep RL for Computer Go)
Deep Reinforcement Learning: Policy Gradients and Q-Learning
Deep Reinforcement Learning: Policy Gradients and Q-Learning John Schulman Bay Area Deep Learning School September 24, 2016 Introduction and Overview Aim of This Talk What is deep RL, and should I use
Towards Universal Sentence Embeddings
Towards Universal Sentence Embeddings Towards Universal Paraphrastic Sentence Embeddings J. Wieting, M. Bansal, K. Gimpel and K. Livescu, ICLR 2016 A Simple But Tough-To-Beat Baseline For Sentence Embeddings
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning
REINFORCE Framework for Stochastic Policy Optimization and its use in Deep Learning Ronen Tamari The Hebrew University of Jerusalem Advanced Seminar in Deep Learning (#67679) February 28, 2016 Ronen Tamari
Part of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch. COMP-599 Oct 1, 2015
Part of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch COMP-599 Oct 1, 2015 Announcements Research skills workshop today 3pm-4:30pm Schulich Library room 313 Start thinking about
Final Exam December 12, 2017
Introduction to Artificial Intelligence CSE 473, Autumn 2017 Dieter Fox Final Exam December 12, 2017 Directions This exam has 7 problems with 111 points shown in the table below, and you have 110 minutes
What s so Hard about Natural Language Understanding?
What s so Hard about Natural Language Understanding? Alan Ritter Computer Science and Engineering The Ohio State University Collaborators: Jiwei Li, Dan Jurafsky (Stanford) Bill Dolan, Michel Galley, Jianfeng
INF 5860 Machine learning for image classification. Lecture 14: Reinforcement learning May 9, 2018
Machine learning for image classification Lecture 14: Reinforcement learning May 9, 2018 Page 3 Outline Motivation Introduction to reinforcement learning (RL) Value function based methods (Q-learning)
Exercises, II part Exercises, II part
Inference: 12 Jul 2012 Consider the following Joint Probability Table for the three binary random variables A, B, C. Compute the following queries: 1 P(C A=T,B=T) 2 P(C A=T) P(A, B, C) A B C 0.108 T T
Reinforcement Learning via Policy Optimization
Reinforcement Learning via Policy Optimization Hanxiao Liu November 22, 2017 1 / 27 Reinforcement Learning Policy a π(s) 2 / 27 Example - Mario 3 / 27 Example - ChatBot 4 / 27 Applications - Video Games
CS230: Lecture 9 Deep Reinforcement Learning
CS230: Lecture 9 Deep Reinforcement Learning Kian Katanforoosh Menti code: 21 90 15 Today s outline I. Motivation II. Recycling is good: an introduction to RL III. Deep Q-Learning IV. Application of Deep
MIDTERM: CS 6375 INSTRUCTOR: VIBHAV GOGATE October,
MIDTERM: CS 6375 INSTRUCTOR: VIBHAV GOGATE October, 23 2013 The exam is closed book. You are allowed a one-page cheat sheet. Answer the questions in the spaces provided on the question sheets. If you run
Reinforcement Learning. Machine Learning, Fall 2010
Reinforcement Learning Machine Learning, Fall 2010 1 Administrativia This week: finish RL, most likely start graphical models LA2: due on Thursday LA3: comes out on Thursday TA Office hours: Today 1:30-2:30
A Tutorial on Learning with Bayesian Networks
A utorial on Learning with Bayesian Networks David Heckerman Presented by: Krishna V Chengavalli April 21 2003 Outline Introduction Different Approaches Bayesian Networks Learning Probabilities and Structure
Basics of reinforcement learning
Basics of reinforcement learning Lucian Buşoniu TMLSS, 20 July 2018 Main idea of reinforcement learning (RL) Learn a sequential decision policy to optimize the cumulative performance of an unknown system
Machine Learning. Machine Learning: Jordan Boyd-Graber University of Maryland REINFORCEMENT LEARNING. Slides adapted from Tom Mitchell and Peter Abeel
Machine Learning Machine Learning: Jordan Boyd-Graber University of Maryland REINFORCEMENT LEARNING Slides adapted from Tom Mitchell and Peter Abeel Machine Learning: Jordan Boyd-Graber UMD Machine Learning
Today s Outline. Recap: MDPs. Bellman Equations. Q-Value Iteration. Bellman Backup 5/7/2012. CSE 473: Artificial Intelligence Reinforcement Learning
CSE 473: Artificial Intelligence Reinforcement Learning Dan Weld Today s Outline Reinforcement Learning Q-value iteration Q-learning Exploration / exploitation Linear function approximation Many slides
Driving Semantic Parsing from the World s Response
Driving Semantic Parsing from the World s Response James Clarke, Dan Goldwasser, Ming-Wei Chang, Dan Roth Cognitive Computation Group University of Illinois at Urbana-Champaign CoNLL 2010 Clarke, Goldwasser,
Fast Adaptive Algorithm for Robust Evaluation of Quality of Experience
Fast Adaptive Algorithm for Robust Evaluation of Quality of Experience Qianqian Xu, Ming Yan, Yuan Yao October 2014 1 Motivation Mean Opinion Score vs. Paired Comparisons Crowdsourcing Ranking on Internet
CS885 Reinforcement Learning Lecture 7a: May 23, 2018
CS885 Reinforcement Learning Lecture 7a: May 23, 2018 Policy Gradient Methods [SutBar] Sec. 13.1-13.3, 13.7 [SigBuf] Sec. 5.1-5.2, [RusNor] Sec. 21.5 CS885 Spring 2018 Pascal Poupart 1 Outline Stochastic
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning Lei Lei Ruoxuan Xiong December 16, 2017 1 Introduction Deep Neural Network
Neural Map. Structured Memory for Deep RL. Emilio Parisotto
Neural Map Structured Memory for Deep RL Emilio Parisotto eparisot@andrew.cmu.edu PhD Student Machine Learning Department Carnegie Mellon University Supervised Learning Most deep learning problems are
The convergence limit of the temporal difference learning
The convergence limit of the temporal difference learning Ryosuke Nomura the University of Tokyo September 3, 2013 1 Outline Reinforcement Learning Convergence limit Construction of the feature vector
Markov Models and Reinforcement Learning. Stephen G. Ware CSCI 4525 / 5525
Markov Models and Reinforcement Learning Stephen G. Ware CSCI 4525 / 5525 Camera Vacuum World (CVW) 2 discrete rooms with cameras that detect dirt. A mobile robot with a vacuum. The goal is to ensure both
Introduction of Reinforcement Learning
Introduction of Reinforcement Learning Deep Reinforcement Learning Reference Textbook: Reinforcement Learning: An Introduction http://incompleteideas.net/sutton/book/the-book.html Lectures of David Silver
The exam is closed book, closed calculator, and closed notes except your one-page crib sheet.
CS 188 Fall 2015 Introduction to Artificial Intelligence Final You have approximately 2 hours and 50 minutes. The exam is closed book, closed calculator, and closed notes except your one-page crib sheet.
Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity- Representativeness Reward
Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity- Representativeness Reward Kaiyang Zhou, Yu Qiao, Tao Xiang AAAI 2018 What is video summarization? Goal: to automatically
Reinforcement Learning
Reinforcement Learning March May, 2013 Schedule Update Introduction 03/13/2015 (10:15-12:15) Sala conferenze MDPs 03/18/2015 (10:15-12:15) Sala conferenze Solving MDPs 03/20/2015 (10:15-12:15) Aula Alpha
Midterm for CSC321, Intro to Neural Networks Winter 2018, night section Tuesday, March 6, 6:10-7pm
Midterm for CSC321, Intro to Neural Networks Winter 2018, night section Tuesday, March 6, 6:10-7pm Name: Student number: This is a closed-book test. It is marked out of 15 marks. Please answer ALL of the
Reinforcement Learning
Reinforcement Learning Ron Parr CompSci 7 Department of Computer Science Duke University With thanks to Kris Hauser for some content RL Highlights Everybody likes to learn from experience Use ML techniques
Time to the Half Hour
LESSON.7 Time to the Half Hour FOCUS COHERENCE RIGOR LESSON AT A GLANCE F C R Focus: Common Core State Standards.MD.B. Tell and write time in hours and half-hours using analog and digital clocks. MATHEMATICAL
CS 598 Statistical Reinforcement Learning. Nan Jiang
CS 598 Statistical Reinforcement Learning Nan Jiang Overview What s this course about? A grad-level seminar course on theory of RL 3 What s this course about? A grad-level seminar course on theory of RL
Reinforcement Learning
Reinforcement Learning Temporal Difference Learning Temporal difference learning, TD prediction, Q-learning, elibigility traces. (many slides from Marc Toussaint) Vien Ngo Marc Toussaint University of
Final. Introduction to Artificial Intelligence. CS 188 Spring You have approximately 2 hours and 50 minutes.
CS 188 Spring 2014 Introduction to Artificial Intelligence Final You have approximately 2 hours and 50 minutes. The exam is closed book, closed notes except your two-page crib sheet. Mark your answers
Prof. Dr. Ann Nowé. Artificial Intelligence Lab ai.vub.ac.be
REINFORCEMENT LEARNING AN INTRODUCTION Prof. Dr. Ann Nowé Artificial Intelligence Lab ai.vub.ac.be REINFORCEMENT LEARNING WHAT IS IT? What is it? Learning from interaction Learning about, from, and while
Administration. CSCI567 Machine Learning (Fall 2018) Outline. Outline. HW5 is available, due on 11/18. Practice final will also be available soon.
Administration CSCI567 Machine Learning Fall 2018 Prof. Haipeng Luo U of Southern California Nov 7, 2018 HW5 is available, due on 11/18. Practice final will also be available soon. Remaining weeks: 11/14,
A few applications of the SVD
A few applications of the SVD Many methods require to approximate the original data (matrix) by a low rank matrix before attempting to solve the original problem Regularization methods require the solution
Deep reinforcement learning. Dialogue Systems Group, Cambridge University Engineering Department
Deep reinforcement learning Milica Gašić Dialogue Systems Group, Cambridge University Engineering Department 1 / 25 In this lecture... Introduction to deep reinforcement learning Value-based Deep RL Deep
In Advances in Neural Information Processing Systems 6. J. D. Cowan, G. Tesauro and. Convergence of Indirect Adaptive. Andrew G.
In Advances in Neural Information Processing Systems 6. J. D. Cowan, G. Tesauro and J. Alspector, (Eds.). Morgan Kaufmann Publishers, San Fancisco, CA. 1994. Convergence of Indirect Adaptive Asynchronous
Information Extraction from Text
Information Extraction from Text Jing Jiang Chapter 2 from Mining Text Data (2012) Presented by Andrew Landgraf, September 13, 2013 1 What is Information Extraction? Goal is to discover structured information
Reinforcement learning an introduction
Reinforcement learning an introduction Prof. Dr. Ann Nowé Computational Modeling Group AIlab ai.vub.ac.be November 2013 Reinforcement Learning What is it? Learning from interaction Learning about, from,
Lecture 1: March 7, 2018
Reinforcement Learning Spring Semester, 2017/8 Lecture 1: March 7, 2018 Lecturer: Yishay Mansour Scribe: ym DISCLAIMER: Based on Learning and Planning in Dynamical Systems by Shie Mannor c, all rights
Reinforcement Learning
Reinforcement Learning Function approximation Daniel Hennes 19.06.2017 University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Eligibility traces n-step TD returns Forward and backward view Function
Reinforcement Learning
Reinforcement Learning Model-Based Reinforcement Learning Model-based, PAC-MDP, sample complexity, exploration/exploitation, RMAX, E3, Bayes-optimal, Bayesian RL, model learning Vien Ngo MLR, University
Trust Region Policy Optimization
Trust Region Policy Optimization Yixin Lin Duke University yixin.lin@duke.edu March 28, 2017 Yixin Lin (Duke) TRPO March 28, 2017 1 / 21 Overview 1 Preliminaries Markov Decision Processes Policy iteration
Deep Reinforcement Learning SISL. Jeremy Morton (jmorton2) November 7, Stanford Intelligent Systems Laboratory
Deep Reinforcement Learning Jeremy Morton (jmorton2) November 7, 2016 SISL Stanford Intelligent Systems Laboratory Overview 2 1 Motivation 2 Neural Networks 3 Deep Reinforcement Learning 4 Deep Learning
Natural Language Processing
Natural Language Processing Word vectors Many slides borrowed from Richard Socher and Chris Manning Lecture plan Word representations Word vectors (embeddings) skip-gram algorithm Relation to matrix factorization
A brief introduction to Conditional Random Fields
A brief introduction to Conditional Random Fields Mark Johnson Macquarie University April, 2005, updated October 2010 1 Talk outline Graphical models Maximum likelihood and maximum conditional likelihood
Reinforcement Learning
Reinforcement Learning Markov decision process & Dynamic programming Evaluative feedback, value function, Bellman equation, optimality, Markov property, Markov decision process, dynamic programming, value
Midterm, Fall 2003
5-78 Midterm, Fall 2003 YOUR ANDREW USERID IN CAPITAL LETTERS: YOUR NAME: There are 9 questions. The ninth may be more time-consuming and is worth only three points, so do not attempt 9 unless you are
PROBABILISTIC KNOWLEDGE GRAPH CONSTRUCTION: COMPOSITIONAL AND INCREMENTAL APPROACHES. Dongwoo Kim with Lexing Xie and Cheng Soon Ong CIKM 2016
PROBABILISTIC KNOWLEDGE GRAPH CONSTRUCTION: COMPOSITIONAL AND INCREMENTAL APPROACHES Dongwoo Kim with Lexing Xie and Cheng Soon Ong CIKM 2016 1 WHAT IS KNOWLEDGE GRAPH (KG)? KG widely used in various tasks
Learning Control for Air Hockey Striking using Deep Reinforcement Learning
Learning Control for Air Hockey Striking using Deep Reinforcement Learning Ayal Taitler, Nahum Shimkin Faculty of Electrical Engineering Technion - Israel Institute of Technology May 8, 2017 A. Taitler,
Rutgers Business School, Introduction to Probability, 26:960:575. Homework 1. Prepared by Daniel Pirutinsky. Feburary 1st, 2016
Rutgers usiness School, Introduction to Probability, 6:960:575 Homework Prepared by Daniel Pirutinsky Feburary st, 06 Introduction Suppose we have two boxes, labeled and. Each box contains some amount
Name: UW CSE 473 Final Exam, Fall 2014
P1 P6 Instructions Please answer clearly and succinctly. If an explanation is requested, think carefully before writing. Points may be removed for rambling answers. If a question is unclear or ambiguous,
Reinforcement Learning Active Learning
Reinforcement Learning Active Learning Alan Fern * Based in part on slides by Daniel Weld 1 Active Reinforcement Learning So far, we ve assumed agent has a policy We just learned how good it is Now, suppose
Monte Carlo is important in practice. CSE 190: Reinforcement Learning: An Introduction. Chapter 6: Temporal Difference Learning.
Monte Carlo is important in practice CSE 190: Reinforcement Learning: An Introduction Chapter 6: emporal Difference Learning When there are just a few possibilitieo value, out of a large state space, Monte
Q-learning. Tambet Matiisen
Q-learning Tambet Matiisen (based on chapter 11.3 of online book Artificial Intelligence, foundations of computational agents by David Poole and Alan Mackworth) Stochastic gradient descent Experience
ENGAGE. Daily Routines Common Core. Essential Question
LESSON 7. Time to the Hour and Half Hour FOCUS COHERENCE RIGOR LESSON AT A GLANCE F C R Focus: Common Core State Standards Learning Objective.MD.C.7 Tell and write time from analog and digital clocks to
Backpropagation Through
Backpropagation Through Backpropagation Through Backpropagation Through Will Grathwohl Dami Choi Yuhuai Wu Geoff Roeder David Duvenaud Where do we see this guy? L( ) =E p(b ) [f(b)] Just about everywhere!
Embedding-Based Techniques MATRICES, TENSORS, AND NEURAL NETWORKS
Embedding-Based Techniques MATRICES, TENSORS, AND NEURAL NETWORKS Probabilistic Models: Downsides Limitation to Logical Relations Embeddings Representation restricted by manual design Clustering? Assymetric
Development of a Deep Recurrent Neural Network Controller for Flight Applications
Development of a Deep Recurrent Neural Network Controller for Flight Applications American Control Conference (ACC) May 26, 2017 Scott A. Nivison Pramod P. Khargonekar Department of Electrical and Computer
Natural Language Processing with Deep Learning CS224N/Ling284
Natural Language Processing with Deep Learning CS224N/Ling284 Lecture 4: Word Window Classification and Neural Networks Richard Socher Organization Main midterm: Feb 13 Alternative midterm: Friday Feb
Reinforcement Learning. Up until now we have been
Reinforcement Learning Slides by Rich Sutton Mods by Dan Lizotte Refer to Reinforcement Learning: An Introduction by Sutton and Barto Alpaydin Chapter 16 Up until now we have been Supervised Learning Classifying,
CS 570: Machine Learning Seminar. Fall 2016
CS 570: Machine Learning Seminar Fall 2016 Class Information Class web page: http://web.cecs.pdx.edu/~mm/mlseminar2016-2017/fall2016/ Class mailing list: cs570@cs.pdx.edu My office hours: T,Th, 2-3pm or
On and Off-Policy Relational Reinforcement Learning
On and Off-Policy Relational Reinforcement Learning Christophe Rodrigues, Pierre Gérard, and Céline Rouveirol LIPN, UMR CNRS 73, Institut Galilée - Université Paris-Nord first.last@lipn.univ-paris13.fr
CMU Lecture 12: Reinforcement Learning. Teacher: Gianni A. Di Caro
CMU 15-781 Lecture 12: Reinforcement Learning Teacher: Gianni A. Di Caro REINFORCEMENT LEARNING Transition Model? State Action Reward model? Agent Goal: Maximize expected sum of future rewards 2 MDP PLANNING
MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti
1 MARKOV DECISION PROCESSES (MDP) AND REINFORCEMENT LEARNING (RL) Versione originale delle slide fornita dal Prof. Francesco Lo Presti Historical background 2 Original motivation: animal learning Early
Reinforcement Learning for NLP
Reinforcement Learning for NLP Advanced Machine Learning for NLP Jordan Boyd-Graber REINFORCEMENT OVERVIEW, POLICY GRADIENT Adapted from slides by David Silver, Pieter Abbeel, and John Schulman Advanced
Robert D. Borchert GIS Technician
QA/QC: AM/FM: A Checklist Confirmed for fit Quality Methods and Control Actions Robert D. Borchert GIS Technician This just goes to show that QA/QC is important. Robert D. Borchert GIS Technician Did you
Recurrent Neural Networks Deep Learning Lecture 5. Efstratios Gavves
Recurrent Neural Networks Deep Learning Lecture 5 Efstratios Gavves Sequential Data So far, all tasks assumed stationary data Neither all data, nor all tasks are stationary though Sequential Data: Text
MS&E338 Reinforcement Learning Lecture 1 - April 2, Introduction
MS&E338 Reinforcement Learning Lecture 1 - April 2, 2018 Introduction Lecturer: Ben Van Roy Scribe: Gabriel Maher 1 Reinforcement Learning Introduction In reinforcement learning (RL) we consider an agent
Chapter 7 R&N ICS 271 Fall 2017 Kalev Kask
Set 6: Knowledge Representation: The Propositional Calculus Chapter 7 R&N ICS 271 Fall 2017 Kalev Kask Outline Representing knowledge using logic Agent that reason logically A knowledge based agent Representing
Anticipating Visual Representations from Unlabeled Data. Carl Vondrick, Hamed Pirsiavash, Antonio Torralba
Anticipating Visual Representations from Unlabeled Data Carl Vondrick, Hamed Pirsiavash, Antonio Torralba Overview Problem Key Insight Methods Experiments Problem: Predict future actions and objects Image
Regularization Introduction to Machine Learning. Matt Gormley Lecture 10 Feb. 19, 2018
1-61 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Regularization Matt Gormley Lecture 1 Feb. 19, 218 1 Reminders Homework 4: Logistic
Recurrent Autoregressive Networks for Online Multi-Object Tracking. Presented By: Ishan Gupta
Recurrent Autoregressive Networks for Online Multi-Object Tracking Presented By: Ishan Gupta Outline Multi Object Tracking Recurrent Autoregressive Networks (RANs) RANs for Online Tracking Other State