Conditional Random Fields for Sequential Supervised Learning
|
|
- Aron Rich
- 5 years ago
- Views:
Transcription
1 Conditional Random Fields for Sequential Supervised Learning Thomas G. Dietterich Adam Ashenfelter Department of Computer Science Oregon State University Corvallis, Oregon Supported by NSF IIS
2 Outline Goal: Off-the-Shelf Sequential Supervised Learning Candidate Methods Training CRFs by Gradient Boosting Concluding Remarks
3 Many Application Problems Require Sequential Learning Part-of-speech Tagging Information Extraction from the Web Text-to-Speech Mapping
4 Part-of-Speech Tagging Given an English sentence, can we assign a part of speech to each word? Do you want fries with that? <verb pron verb noun prep pron>
5 Information Extraction from the Web <dl><dt><b>srinivasan Seshan</b> (Carnegie Mellon University) <dt><a href= ><i>making Virtual Worlds Real</i></a><dt>Tuesday, June 4, 2002<dd>2:00 PM, 322 Sieg<dd>Research Seminar * * * name name * * affiliation affiliation affiliation * * * * title title title title * * * date date date date * time time * location location * event-type event-type
6 Text-to-Speech Mapping photograph =>
7 Sequential Supervised Learning (SSL) Given: A set of training examples of the form (X i,y i ), where X i = x i,1,, x i,t i and Y i = y i,1,, y i,t i are sequences of length T i Find: A function f for predicting new sequences: Y = f(x).
8 Examples as Sequential Supervised Learning Domain Input X i Output Y i Part-of-speech sequence of sequence of Tagging words parts of speech Information Extraction Test-to-speech Mapping sequence of tokens sequence of letters sequence of field labels {name, } sequence phonemes
9 Goal: Off-the-Shelf Learning Methods for SSL No existing machine learning, data mining, and statistical packages supports SSL No existing method meets all of the requirements needed for an off-theshelf method
10 Requirements for Off-the-Shelf Methods Accuracy Model both x y and y t y t+1 relationships Support rich X y t features Avoid label bias problem Computational efficiency Easy to use No parameter tuning required
11 Two Kinds of Relationships y 1 y 2 y 3 x 1 x 2 x 3 Vertical relationship between the x t s and y t s Example: Friday is usually a date Horizontal relationships among the y t s Example: name is usually followed by affiliation SSL can (and should) exploit both kinds of information
12 Example of y y relationships Consider the text-to-speech problem: photograph => /f-ot@graf-/ photography =>/f-@tagr@f-i/ The letter y changes the pronunciation of all vowels in the word! x y relationships are not sufficient: o is pronounced as /O/, /@/, and /A/ need context to tell which is correct
13 Rich X y Relationships Generative models such as HMMs model each x t as being generated by a single y t Can t incorporate the context around x t Example: Decide how to pronounce h based on surrounding letters th, ph, sh, ch. Can t include global features Example: Sentence begins with question word
14 Existing Methods Sliding windows Recurrent sliding windows Hidden Markov models Maximum entropy Markov models Input/Output Markov models Conditional Random Fields Maximum Margin Markov Networks
15 Sliding Windows Do you want fries with that Do you verb Do you want pron you want fries verb want fries with noun fries with that prep with that pron
16 Properties of Sliding Windows Converts SSL to ordinary supervised learning Only captures the relationship between (part of) X and y t. Does not explicitly model relations among the y t s Assumes each window is independent
17 Recurrent Sliding Windows Do you want fries with that Do you verb Do you want verb pron you want fries pron verb want fries with verb noun fries with that noun prep with that prep pron
18 Recurrent Sliding Windows Key Idea: Include y t as input feature when computing y t+1. During training: Use the correct value of y t Or train iteratively (especially recurrent neural networks) During evaluation: Use the predicted value of y t
19 Properties of Recurrent Sliding Windows Captures relationship among the y s, but only in one direction! Results on text-to-speech: Method Direction Words Letters sliding window none 12.5% 69.6% recurrent s. w. left-right 17.0% 67.9% recurrent s. w. right-left 24.4% 74.2%
20 Evaluation of Methods Issue SW RSW HMM MEMM IOHMM CRF x t y t y t y t+1 NO Partly YES YES YES YES X y t rich? YES YES NO YES YES YES label bias ok? YES YES YES NO NO YES efficient? YES YES YES YES? NO NO
21 Conditional Random Fields y 1 y 2 y 3 x 1 x 2 x 3 The y t s form a Markov Random Field conditioned on X: P(Y X) Lafferty, McCallum, & Pereira (2001)
22 Markov Random Fields Graph G = (V,E) Each vertex v V represents a random variable y v. Each edge represents a direct probabilistic dependency. P(Y) = 1/Z exp [ c c (c(y))] c indexes the cliques in the graph c is a potential function c(y) selects the random variables participating in clique c.
23 A Simple MRF y 1 y 2 y 3 Cliques: singletons: {y 1 }, {y 2 }, {y 3 } pairs (edges); {y 1,y 2 }, {y 2,y 3 } P( y 1,y 2,y 3 ) = 1/Z exp[ 1 (y 1 ) + 2 (y 2 ) + 3 (y 3 ) + 12 (y 1,y 2 ) + 23 (y 2,y 3 )]
24 CRF Potential Functions are Conditioned on X y 1 y 2 y 3 x 1 x 2 x 3 t (y t,x) t,t+1 (y t,y t+1,x)
25 CRF Potentials are Log Linear Models t (y t,x) = b b g b (y t,x) t,t+1 (y t,y t+1,x) = a a f a (y t,y t+1,x) where g b and f a are user-defined boolean functions ( features ) Example: g 23 = [x t = o and y t = /@/]
26 Training CRFs Let = { 1, 2,, 1, 2, } be all of our parameters Let F be our CRF, so F (Y,X) = P(Y X) Define the loss function L(Y,F (Y,X)) to be the Negative Log Likelihood L(Y,F (Y,X)) = log F (Y,X) Goal: Find to minimize loss (maximize likelihood)
27 Algorithms Iterative Scaling Gradient Descent Functional Gradient Descent Gradient tree boosting
28 Gradient Descent Search From calculus we know that the minimum loss will be where d L(Y,F (Y,X)) d = L(Y,F (Y,X)) = 0 Method: := L(Y,F (Y,X))
29 Gradient Descent with Set of Training Examples We have N training examples (X i,y i ) Negative log likelihood of all N examples is the sum of the neg log likelihoods of each example The gradient of the negative log likelihood is the sum of the gradients of the neg log likelihoods of each example.
30 Gradients from Each Example example gradient (X 1,Y 1 ) L(Y 1,F (Y 1,X 1 )) (X 2,Y 2 ) L(Y 2,F (Y 2,X 2 )) (X 3,Y 3 ) L(Y 3,F (Y 3,X 3 )) (X 4,Y 4 ) L(Y 4,F (Y 4,X 4 )) := i L(Y i, F (Y i,x i ))
31 Problem: Gradient Descent is Very Slow Lafferty et al. employed modified iterative scaling but reported that it was very slow. We (and others) implemented conjugate gradient search, which is faster, but not fast enough For text-to-speech: 16 parallel processors, 40 hours per line search.
32 Functional Gradient Descent (Breiman, et al.) Standard gradient descent: final = M where m = m-1 i L(Y i, F m-1 (Y i,x i )) Functional Gradient Descent: F final = F M where m = h m, and h m is a function that approximates F i L(Y i,f m-1 (Y i,x i ))
33 Functional Gradient Descent (2) example functional gradient functional gradient example (X 1,Y 1 ) F L(Y 1,F m-1 (Y 1,X 1 )) = g 1 (X 1,g 1 ) (X 2,Y 2 ) F L(Y 2,F m-1 (Y 2,X 2 )) = g 2 (X 2,g 2 ) (X 3,Y 3 ) F L(Y 3,F m-1 (Y 3,X 3 )) = g 3 (X 3,g 3 ) (X 4,Y 4 ) F L(Y 4,F m-1 (Y 4,X 4 )) = g 4 (X 4,g 4 ) Fit h to minimize i [h(x i ) g i ] 2
34 Friedman s Gradient Boosting Algorithm F 0 = argmin i L(Y i, ) For m = 1,, M do g i := F L(Y i,f m-1 (Y i,x i )), i = 1,, N fit regression tree h := argmin f i [f(x i ) g i ] 2 m = argmin i L(Y i, F m-1 (Y i,x i ) + h(x i )) F m = F m-1 + m h m
35 Regression Trees x 1 = o yes no x 2 = t x 1 = a yes no yes no h= 0.3 h= -2.1 h= 0.8 Very fast and effective algorithms
36 Application to CRF Training Recall CRF model: (y t-1,y t,x) = a a f a (y t-1,y t,x) (y t,x) = b b g b (y t,x)] Represent (y t-1,y t,x) + (y t,x) by a set of K functions (one per class label): (l,k,x) + (k,x) = F k (l,x), k = 1,, K where F k (l,x) = m m h k,m (l,x) Each h k,m is a regression tree that tests the features {f a, g b } of the CRF The values in the leaves of the tree become the weights a and b
37 Sum of Regression Trees is Equivalent to CRF Circled Path is equivalent to expression of the form a f a a = f a = s 1 & s 4 & s 18
38 Resulting CRF Model P(Y X) = 1/Z * exp[ t F yt (y t-1,x)]
39 Forward-Backward Algorithm: Recursive Computation of Z Let (k,1) = exp F k (,X) (k,t) = k [exp F k (k,x)] (k,t 1) (k,t) = 1 (k,t) = k [exp F k (k,x)] (k,t+1) Z = k (k,t) = (,0)
40 Functional Gradient Computation Let w t (X i ) be the window of X i used by the features at time t. We get one training example for each k, l, i, and t: g k,l,i,t = log L(Y i, P(Y i X i )) F k (l,w t (X i )) Training example for F k : ( l,w t (X i ), g k,l,i,t )
41 Functional Gradient Computation (2) g k,l,i,t = log L(Y i, P(Y i X i )) F k (l,w t (X i )) = F k (l,w t (X i )) t F yt (y t-1,w t (X i )) log Z = I[y t-1 =l,y t =k] 1 Z F k (l,w t (X i )) Z
42 Functional Gradient Computation (3) 1 Z F k (l,w t (X i )) Z = 1 Z F k (l,w t (X i )) u ( v [exp F u (v,w t (X i ))] (v,t-1)) (u,t) = F k (l,w t (X i ) (l,t-1) (k,t) Z = P(y i,t =k, y i,t-1 =l X i )
43 Functional Gradient Computation (4) g k,l,i,t = log L(Y i, P(Y i X i )) F k (l,w t (X i )) = I[y i,t-1 =l, y i,t =k] P(y i,t =k, y i,t-1 =l X i ) This is our residual on the probability scale
44 Training Procedure Initialize F k = 0; k=1,,k For m = 1,, M For i = 1,, N Compute (k,t), (k,t), Z via forward/backward for (X i,y i ) Compute gradients for F k according to g k,l,i,t = I[y i,t = k, y i,t-1 = l] (l,t 1) [exp F k (l,x i )] (k,t)/z Fit regression trees h k,m to ( l,w t (X i ), g k,l,i,t ) pairs Update: F k := F k +h k,m
45 Initial Results: Training Times Gradient Boosting 1 processor: 100 iterations requires 6 hours (compared to 16*40*100 = 64,000 hours for conjugate gradient) However: Full Gradient Boosting algorithm was not implemented
46 Results: Whole words correct 5-letter window Viterbi beam width 20.
47 Whole Words: Window Sizes of 3 and 7
48 Predicting Single Letters
49 Why Gradient Boosting is More Effective Each step is large: Each iteration adds one regression tree to the potential function for each class Parameters are introduced only as necessary Combinations of features are constructed
50 Concluding Remarks Many machine learning applications can be formalized as Sequential Supervised Learning Similar issues arise in other complex learning problems (e.g., spatial and relational data) Many methods have been developed specifically for SSL, but none is perfect Gradient boosting may provide a general, offthe-shelf way of fitting CRFs.
Sequential Supervised Learning
Sequential Supervised Learning Many Application Problems Require Sequential Learning Part-of of-speech Tagging Information Extraction from the Web Text-to to-speech Mapping Part-of of-speech Tagging Given
More informationConditional Random Fields and beyond DANIEL KHASHABI CS 546 UIUC, 2013
Conditional Random Fields and beyond DANIEL KHASHABI CS 546 UIUC, 2013 Outline Modeling Inference Training Applications Outline Modeling Problem definition Discriminative vs. Generative Chain CRF General
More informationSequence labeling. Taking collective a set of interrelated instances x 1,, x T and jointly labeling them
HMM, MEMM and CRF 40-957 Special opics in Artificial Intelligence: Probabilistic Graphical Models Sharif University of echnology Soleymani Spring 2014 Sequence labeling aking collective a set of interrelated
More information10 : HMM and CRF. 1 Case Study: Supervised Part-of-Speech Tagging
10-708: Probabilistic Graphical Models 10-708, Spring 2018 10 : HMM and CRF Lecturer: Kayhan Batmanghelich Scribes: Ben Lengerich, Michael Kleyman 1 Case Study: Supervised Part-of-Speech Tagging We will
More informationGraphical models for part of speech tagging
Indian Institute of Technology, Bombay and Research Division, India Research Lab Graphical models for part of speech tagging Different Models for POS tagging HMM Maximum Entropy Markov Models Conditional
More informationLog-Linear Models, MEMMs, and CRFs
Log-Linear Models, MEMMs, and CRFs Michael Collins 1 Notation Throughout this note I ll use underline to denote vectors. For example, w R d will be a vector with components w 1, w 2,... w d. We use expx
More informationStatistical NLP for the Web Log Linear Models, MEMM, Conditional Random Fields
Statistical NLP for the Web Log Linear Models, MEMM, Conditional Random Fields Sameer Maskey Week 13, Nov 28, 2012 1 Announcements Next lecture is the last lecture Wrap up of the semester 2 Final Project
More informationSequence Modelling with Features: Linear-Chain Conditional Random Fields. COMP-599 Oct 6, 2015
Sequence Modelling with Features: Linear-Chain Conditional Random Fields COMP-599 Oct 6, 2015 Announcement A2 is out. Due Oct 20 at 1pm. 2 Outline Hidden Markov models: shortcomings Generative vs. discriminative
More informationMachine Learning for Structured Prediction
Machine Learning for Structured Prediction Grzegorz Chrupa la National Centre for Language Technology School of Computing Dublin City University NCLT Seminar Grzegorz Chrupa la (DCU) Machine Learning for
More informationConditional Random Field
Introduction Linear-Chain General Specific Implementations Conclusions Corso di Elaborazione del Linguaggio Naturale Pisa, May, 2011 Introduction Linear-Chain General Specific Implementations Conclusions
More informationUndirected Graphical Models
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Properties Properties 3 Generative vs. Conditional
More informationlecture 6: modeling sequences (final part)
Natural Language Processing 1 lecture 6: modeling sequences (final part) Ivan Titov Institute for Logic, Language and Computation Outline After a recap: } Few more words about unsupervised estimation of
More informationProbabilistic Models for Sequence Labeling
Probabilistic Models for Sequence Labeling Besnik Fetahu June 9, 2011 Besnik Fetahu () Probabilistic Models for Sequence Labeling June 9, 2011 1 / 26 Background & Motivation Problem introduction Generative
More informationInformation Extraction from Text
Information Extraction from Text Jing Jiang Chapter 2 from Mining Text Data (2012) Presented by Andrew Landgraf, September 13, 2013 1 What is Information Extraction? Goal is to discover structured information
More informationHidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 22 April 2, 2018 1 Reminders Homework
More informationLogistic Regression: Online, Lazy, Kernelized, Sequential, etc.
Logistic Regression: Online, Lazy, Kernelized, Sequential, etc. Harsha Veeramachaneni Thomson Reuter Research and Development April 1, 2010 Harsha Veeramachaneni (TR R&D) Logistic Regression April 1, 2010
More informationHidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 19 Nov. 5, 2018 1 Reminders Homework
More informationA brief introduction to Conditional Random Fields
A brief introduction to Conditional Random Fields Mark Johnson Macquarie University April, 2005, updated October 2010 1 Talk outline Graphical models Maximum likelihood and maximum conditional likelihood
More informationConditional Random Fields: An Introduction
University of Pennsylvania ScholarlyCommons Technical Reports (CIS) Department of Computer & Information Science 2-24-2004 Conditional Random Fields: An Introduction Hanna M. Wallach University of Pennsylvania
More informationLecture 13: Structured Prediction
Lecture 13: Structured Prediction Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/nlp16 CS6501: NLP 1 Quiz 2 v Lectures 9-13 v Lecture 12: before page
More informationMaximum Entropy Markov Models
Wi nøt trei a høliday in Sweden this yër? September 19th 26 Background Preliminary CRF work. Published in 2. Authors: McCallum, Freitag and Pereira. Key concepts: Maximum entropy / Overlapping features.
More informationStatistical Machine Learning Theory. From Multi-class Classification to Structured Output Prediction. Hisashi Kashima.
http://goo.gl/jv7vj9 Course website KYOTO UNIVERSITY Statistical Machine Learning Theory From Multi-class Classification to Structured Output Prediction Hisashi Kashima kashima@i.kyoto-u.ac.jp DEPARTMENT
More informationStructure Learning in Sequential Data
Structure Learning in Sequential Data Liam Stewart liam@cs.toronto.edu Richard Zemel zemel@cs.toronto.edu 2005.09.19 Motivation. Cau, R. Kuiper, and W.-P. de Roever. Formalising Dijkstra's development
More informationCOMP90051 Statistical Machine Learning
COMP90051 Statistical Machine Learning Semester 2, 2017 Lecturer: Trevor Cohn 24. Hidden Markov Models & message passing Looking back Representation of joint distributions Conditional/marginal independence
More informationStatistical Machine Learning Theory. From Multi-class Classification to Structured Output Prediction. Hisashi Kashima.
http://goo.gl/xilnmn Course website KYOTO UNIVERSITY Statistical Machine Learning Theory From Multi-class Classification to Structured Output Prediction Hisashi Kashima kashima@i.kyoto-u.ac.jp DEPARTMENT
More informationLecture 13: Discriminative Sequence Models (MEMM and Struct. Perceptron)
Lecture 13: Discriminative Sequence Models (MEMM and Struct. Perceptron) Intro to NLP, CS585, Fall 2014 http://people.cs.umass.edu/~brenocon/inlp2014/ Brendan O Connor (http://brenocon.com) 1 Models for
More information2.2 Structured Prediction
The hinge loss (also called the margin loss), which is optimized by the SVM, is a ramp function that has slope 1 when yf(x) < 1 and is zero otherwise. Two other loss functions squared loss and exponential
More informationIncorporating Boosted Regression Trees into Ecological Latent Variable Models
Incorporating Boosted Regression Trees into Ecological Latent Variable Models Rebecca A. Hutchinson, Li-Ping Liu, Thomas G. Dietterich School of EECS, Oregon State University Motivation Species Distribution
More informationProbabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Max-margin learning of GM Eric Xing Lecture 28, Apr 28, 2014 b r a c e Reading: 1 Classical Predictive Models Input and output space: Predictive
More informationCOMP 551 Applied Machine Learning Lecture 3: Linear regression (cont d)
COMP 551 Applied Machine Learning Lecture 3: Linear regression (cont d) Instructor: Herke van Hoof (herke.vanhoof@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless
More informationExpectation Maximization (EM)
Expectation Maximization (EM) The EM algorithm is used to train models involving latent variables using training data in which the latent variables are not observed (unlabeled data). This is to be contrasted
More informationHidden Markov Models in Language Processing
Hidden Markov Models in Language Processing Dustin Hillard Lecture notes courtesy of Prof. Mari Ostendorf Outline Review of Markov models What is an HMM? Examples General idea of hidden variables: implications
More informationA.I. in health informatics lecture 8 structured learning. kevin small & byron wallace
A.I. in health informatics lecture 8 structured learning kevin small & byron wallace today models for structured learning: HMMs and CRFs structured learning is particularly useful in biomedical applications:
More informationPart of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch. COMP-599 Oct 1, 2015
Part of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch COMP-599 Oct 1, 2015 Announcements Research skills workshop today 3pm-4:30pm Schulich Library room 313 Start thinking about
More informationOnline Learning and Sequential Decision Making
Online Learning and Sequential Decision Making Emilie Kaufmann CNRS & CRIStAL, Inria SequeL, emilie.kaufmann@univ-lille.fr Research School, ENS Lyon, Novembre 12-13th 2018 Emilie Kaufmann Online Learning
More informationCSE 490 U Natural Language Processing Spring 2016
CSE 490 U Natural Language Processing Spring 2016 Feature Rich Models Yejin Choi - University of Washington [Many slides from Dan Klein, Luke Zettlemoyer] Structure in the output variable(s)? What is the
More informationLogistic Regression & Neural Networks
Logistic Regression & Neural Networks CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Logistic Regression Perceptron & Probabilities What if we want a probability
More informationMACHINE LEARNING FOR NATURAL LANGUAGE PROCESSING
MACHINE LEARNING FOR NATURAL LANGUAGE PROCESSING Outline Some Sample NLP Task [Noah Smith] Structured Prediction For NLP Structured Prediction Methods Conditional Random Fields Structured Perceptron Discussion
More informationNatural Language Processing
Natural Language Processing Global linear models Based on slides from Michael Collins Globally-normalized models Why do we decompose to a sequence of decisions? Can we directly estimate the probability
More informationPartially Directed Graphs and Conditional Random Fields. Sargur Srihari
Partially Directed Graphs and Conditional Random Fields Sargur srihari@cedar.buffalo.edu 1 Topics Conditional Random Fields Gibbs distribution and CRF Directed and Undirected Independencies View as combination
More informationMachine Learning for NLP
Machine Learning for NLP Uppsala University Department of Linguistics and Philology Slides borrowed from Ryan McDonald, Google Research Machine Learning for NLP 1(50) Introduction Linear Classifiers Classifiers
More informationMidterm sample questions
Midterm sample questions CS 585, Brendan O Connor and David Belanger October 12, 2014 1 Topics on the midterm Language concepts Translation issues: word order, multiword translations Human evaluation Parts
More informationA Gentle Introduction to Gradient Boosting. Cheng Li College of Computer and Information Science Northeastern University
A Gentle Introduction to Gradient Boosting Cheng Li chengli@ccs.neu.edu College of Computer and Information Science Northeastern University Gradient Boosting a powerful machine learning algorithm it can
More informationSTA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 11 Project
More informationBayesian Machine Learning - Lecture 7
Bayesian Machine Learning - Lecture 7 Guido Sanguinetti Institute for Adaptive and Neural Computation School of Informatics University of Edinburgh gsanguin@inf.ed.ac.uk March 4, 2015 Today s lecture 1
More informationHidden Markov Models. Aarti Singh Slides courtesy: Eric Xing. Machine Learning / Nov 8, 2010
Hidden Markov Models Aarti Singh Slides courtesy: Eric Xing Machine Learning 10-701/15-781 Nov 8, 2010 i.i.d to sequential data So far we assumed independent, identically distributed data Sequential data
More informationIntelligent Systems (AI-2)
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 19 Oct, 24, 2016 Slide Sources Raymond J. Mooney University of Texas at Austin D. Koller, Stanford CS - Probabilistic Graphical Models D. Page,
More informationRegularization Introduction to Machine Learning. Matt Gormley Lecture 10 Feb. 19, 2018
1-61 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Regularization Matt Gormley Lecture 1 Feb. 19, 218 1 Reminders Homework 4: Logistic
More informationBridging the two cultures: Latent variable statistical modeling with boosted regression trees
Bridging the two cultures: Latent variable statistical modeling with boosted regression trees Thomas G. Dietterich and Rebecca Hutchinson Oregon State University Corvallis, Oregon, USA 1 A Species Distribution
More informationSTA 414/2104: Machine Learning
STA 414/2104: Machine Learning Russ Salakhutdinov Department of Computer Science! Department of Statistics! rsalakhu@cs.toronto.edu! http://www.cs.toronto.edu/~rsalakhu/ Lecture 9 Sequential Data So far
More informationNeural Networks in Structured Prediction. November 17, 2015
Neural Networks in Structured Prediction November 17, 2015 HWs and Paper Last homework is going to be posted soon Neural net NER tagging model This is a new structured model Paper - Thursday after Thanksgiving
More informationCRF for human beings
CRF for human beings Arne Skjærholt LNS seminar CRF for human beings LNS seminar 1 / 29 Let G = (V, E) be a graph such that Y = (Y v ) v V, so that Y is indexed by the vertices of G. Then (X, Y) is a conditional
More informationIntroduction to Machine Learning CMU-10701
Introduction to Machine Learning CMU-10701 Hidden Markov Models Barnabás Póczos & Aarti Singh Slides courtesy: Eric Xing i.i.d to sequential data So far we assumed independent, identically distributed
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
More informationBayesian Networks Introduction to Machine Learning. Matt Gormley Lecture 24 April 9, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Bayesian Networks Matt Gormley Lecture 24 April 9, 2018 1 Homework 7: HMMs Reminders
More informationLecture 15. Probabilistic Models on Graph
Lecture 15. Probabilistic Models on Graph Prof. Alan Yuille Spring 2014 1 Introduction We discuss how to define probabilistic models that use richly structured probability distributions and describe how
More informationMachine Learning Lecture 7
Course Outline Machine Learning Lecture 7 Fundamentals (2 weeks) Bayes Decision Theory Probability Density Estimation Statistical Learning Theory 23.05.2016 Discriminative Approaches (5 weeks) Linear Discriminant
More informationCSCI-567: Machine Learning (Spring 2019)
CSCI-567: Machine Learning (Spring 2019) Prof. Victor Adamchik U of Southern California Mar. 19, 2019 March 19, 2019 1 / 43 Administration March 19, 2019 2 / 43 Administration TA3 is due this week March
More informationHidden Markov Models. By Parisa Abedi. Slides courtesy: Eric Xing
Hidden Markov Models By Parisa Abedi Slides courtesy: Eric Xing i.i.d to sequential data So far we assumed independent, identically distributed data Sequential (non i.i.d.) data Time-series data E.g. Speech
More informationBrief Introduction of Machine Learning Techniques for Content Analysis
1 Brief Introduction of Machine Learning Techniques for Content Analysis Wei-Ta Chu 2008/11/20 Outline 2 Overview Gaussian Mixture Model (GMM) Hidden Markov Model (HMM) Support Vector Machine (SVM) Overview
More informationCSC242: Intro to AI. Lecture 21
CSC242: Intro to AI Lecture 21 Administrivia Project 4 (homeworks 18 & 19) due Mon Apr 16 11:59PM Posters Apr 24 and 26 You need an idea! You need to present it nicely on 2-wide by 4-high landscape pages
More informationMachine Learning & Data Mining Caltech CS/CNS/EE 155 Hidden Markov Models Last Updated: Feb 7th, 2017
1 Introduction Let x = (x 1,..., x M ) denote a sequence (e.g. a sequence of words), and let y = (y 1,..., y M ) denote a corresponding hidden sequence that we believe explains or influences x somehow
More informationDynamic Approaches: The Hidden Markov Model
Dynamic Approaches: The Hidden Markov Model Davide Bacciu Dipartimento di Informatica Università di Pisa bacciu@di.unipi.it Machine Learning: Neural Networks and Advanced Models (AA2) Inference as Message
More informationLecture 9: PGM Learning
13 Oct 2014 Intro. to Stats. Machine Learning COMP SCI 4401/7401 Table of Contents I Learning parameters in MRFs 1 Learning parameters in MRFs Inference and Learning Given parameters (of potentials) and
More informationMachine Learning, Midterm Exam
10-601 Machine Learning, Midterm Exam Instructors: Tom Mitchell, Ziv Bar-Joseph Wednesday 12 th December, 2012 There are 9 questions, for a total of 100 points. This exam has 20 pages, make sure you have
More informationBias-Variance Tradeoff
What s learning, revisited Overfitting Generative versus Discriminative Logistic Regression Machine Learning 10701/15781 Carlos Guestrin Carnegie Mellon University September 19 th, 2007 Bias-Variance Tradeoff
More informationChapter 4 Dynamic Bayesian Networks Fall Jin Gu, Michael Zhang
Chapter 4 Dynamic Bayesian Networks 2016 Fall Jin Gu, Michael Zhang Reviews: BN Representation Basic steps for BN representations Define variables Define the preliminary relations between variables Check
More informationHidden Markov Models, I. Examples. Steven R. Dunbar. Toy Models. Standard Mathematical Models. Realistic Hidden Markov Models.
, I. Toy Markov, I. February 17, 2017 1 / 39 Outline, I. Toy Markov 1 Toy 2 3 Markov 2 / 39 , I. Toy Markov A good stack of examples, as large as possible, is indispensable for a thorough understanding
More informationFINAL: CS 6375 (Machine Learning) Fall 2014
FINAL: CS 6375 (Machine Learning) Fall 2014 The exam is closed book. You are allowed a one-page cheat sheet. Answer the questions in the spaces provided on the question sheets. If you run out of room for
More informationMIDTERM: CS 6375 INSTRUCTOR: VIBHAV GOGATE October,
MIDTERM: CS 6375 INSTRUCTOR: VIBHAV GOGATE October, 23 2013 The exam is closed book. You are allowed a one-page cheat sheet. Answer the questions in the spaces provided on the question sheets. If you run
More informationProbabilistic Graphical Models: MRFs and CRFs. CSE628: Natural Language Processing Guest Lecturer: Veselin Stoyanov
Probabilistic Graphical Models: MRFs and CRFs CSE628: Natural Language Processing Guest Lecturer: Veselin Stoyanov Why PGMs? PGMs can model joint probabilities of many events. many techniques commonly
More informationPattern Recognition and Machine Learning
Christopher M. Bishop Pattern Recognition and Machine Learning ÖSpri inger Contents Preface Mathematical notation Contents vii xi xiii 1 Introduction 1 1.1 Example: Polynomial Curve Fitting 4 1.2 Probability
More informationMachine Learning for NLP
Machine Learning for NLP Linear Models Joakim Nivre Uppsala University Department of Linguistics and Philology Slides adapted from Ryan McDonald, Google Research Machine Learning for NLP 1(26) Outline
More informationHidden Markov Models
Hidden Markov Models Slides mostly from Mitch Marcus and Eric Fosler (with lots of modifications). Have you seen HMMs? Have you seen Kalman filters? Have you seen dynamic programming? HMMs are dynamic
More informationCSE 447/547 Natural Language Processing Winter 2018
CSE 447/547 Natural Language Processing Winter 2018 Feature Rich Models (Log Linear Models) Yejin Choi University of Washington [Many slides from Dan Klein, Luke Zettlemoyer] Announcements HW #3 Due Feb
More informationGeneralized Linear Classifiers in NLP
Generalized Linear Classifiers in NLP (or Discriminative Generalized Linear Feature-Based Classifiers) Graduate School of Language Technology, Sweden 2009 Ryan McDonald Google Inc., New York, USA E-mail:
More informationNeural Architectures for Image, Language, and Speech Processing
Neural Architectures for Image, Language, and Speech Processing Karl Stratos June 26, 2018 1 / 31 Overview Feedforward Networks Need for Specialized Architectures Convolutional Neural Networks (CNNs) Recurrent
More informationMachine Learning, Fall 2011: Homework 5
10-601 Machine Learning, Fall 2011: Homework 5 Machine Learning Department Carnegie Mellon University Due:???????????, 5pm Instructions There are???? questions on this assignment. The?????? question involves
More information6.047 / Computational Biology: Genomes, Networks, Evolution Fall 2008
MIT OpenCourseWare http://ocw.mit.edu 6.047 / 6.878 Computational Biology: Genomes, etworks, Evolution Fall 2008 For information about citing these materials or our Terms of Use, visit: http://ocw.mit.edu/terms.
More informationCOMS 4771 Probabilistic Reasoning via Graphical Models. Nakul Verma
COMS 4771 Probabilistic Reasoning via Graphical Models Nakul Verma Last time Dimensionality Reduction Linear vs non-linear Dimensionality Reduction Principal Component Analysis (PCA) Non-linear methods
More informationIntelligent Systems (AI-2)
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 19 Oct, 23, 2015 Slide Sources Raymond J. Mooney University of Texas at Austin D. Koller, Stanford CS - Probabilistic Graphical Models D. Page,
More informationHidden Markov Models
CS769 Spring 2010 Advanced Natural Language Processing Hidden Markov Models Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu 1 Part-of-Speech Tagging The goal of Part-of-Speech (POS) tagging is to label each
More informationGenerative v. Discriminative classifiers Intuition
Logistic Regression Machine Learning 070/578 Carlos Guestrin Carnegie Mellon University September 24 th, 2007 Generative v. Discriminative classifiers Intuition Want to Learn: h:x a Y X features Y target
More informationRandom Field Models for Applications in Computer Vision
Random Field Models for Applications in Computer Vision Nazre Batool Post-doctorate Fellow, Team AYIN, INRIA Sophia Antipolis Outline Graphical Models Generative vs. Discriminative Classifiers Markov Random
More informationEmpirical Methods in Natural Language Processing Lecture 11 Part-of-speech tagging and HMMs
Empirical Methods in Natural Language Processing Lecture 11 Part-of-speech tagging and HMMs (based on slides by Sharon Goldwater and Philipp Koehn) 21 February 2018 Nathan Schneider ENLP Lecture 11 21
More informationLab 12: Structured Prediction
December 4, 2014 Lecture plan structured perceptron application: confused messages application: dependency parsing structured SVM Class review: from modelization to classification What does learning mean?
More informationLearning from Examples
Learning from Examples Data fitting Decision trees Cross validation Computational learning theory Linear classifiers Neural networks Nonparametric methods: nearest neighbor Support vector machines Ensemble
More informationIntroduction to Machine Learning Midterm Exam Solutions
10-701 Introduction to Machine Learning Midterm Exam Solutions Instructors: Eric Xing, Ziv Bar-Joseph 17 November, 2015 There are 11 questions, for a total of 100 points. This exam is open book, open notes,
More informationMore on HMMs and other sequence models. Intro to NLP - ETHZ - 18/03/2013
More on HMMs and other sequence models Intro to NLP - ETHZ - 18/03/2013 Summary Parts of speech tagging HMMs: Unsupervised parameter estimation Forward Backward algorithm Bayesian variants Discriminative
More informationIntroduction to Logistic Regression and Support Vector Machine
Introduction to Logistic Regression and Support Vector Machine guest lecturer: Ming-Wei Chang CS 446 Fall, 2009 () / 25 Fall, 2009 / 25 Before we start () 2 / 25 Fall, 2009 2 / 25 Before we start Feel
More informationJessica Wehner. Summer Fellow Bioengineering and Bioinformatics Summer Institute University of Pittsburgh 29 May 2008
Journal Club Jessica Wehner Summer Fellow Bioengineering and Bioinformatics Summer Institute University of Pittsburgh 29 May 2008 Comparison of Probabilistic Combination Methods for Protein Secondary Structure
More informationLecture 5 Neural models for NLP
CS546: Machine Learning in NLP (Spring 2018) http://courses.engr.illinois.edu/cs546/ Lecture 5 Neural models for NLP Julia Hockenmaier juliahmr@illinois.edu 3324 Siebel Center Office hours: Tue/Thu 2pm-3pm
More informationECE521 Lectures 9 Fully Connected Neural Networks
ECE521 Lectures 9 Fully Connected Neural Networks Outline Multi-class classification Learning multi-layer neural networks 2 Measuring distance in probability space We learnt that the squared L2 distance
More informationMax Margin-Classifier
Max Margin-Classifier Oliver Schulte - CMPT 726 Bishop PRML Ch. 7 Outline Maximum Margin Criterion Math Maximizing the Margin Non-Separable Data Kernels and Non-linear Mappings Where does the maximization
More informationLow Bias Bagged Support Vector Machines
Low Bias Bagged Support Vector Machines Giorgio Valentini Dipartimento di Scienze dell Informazione Università degli Studi di Milano, Italy valentini@dsi.unimi.it Thomas G. Dietterich Department of Computer
More informationA gentle introduction to Hidden Markov Models
A gentle introduction to Hidden Markov Models Mark Johnson Brown University November 2009 1 / 27 Outline What is sequence labeling? Markov models Hidden Markov models Finding the most likely state sequence
More informationProbabilistic Graphical Models
Probabilistic Graphical Models Lecture 11 CRFs, Exponential Family CS/CNS/EE 155 Andreas Krause Announcements Homework 2 due today Project milestones due next Monday (Nov 9) About half the work should
More information10-701/15-781, Machine Learning: Homework 4
10-701/15-781, Machine Learning: Homewor 4 Aarti Singh Carnegie Mellon University ˆ The assignment is due at 10:30 am beginning of class on Mon, Nov 15, 2010. ˆ Separate you answers into five parts, one
More informationWe Live in Exciting Times. CSCI-567: Machine Learning (Spring 2019) Outline. Outline. ACM (an international computing research society) has named
We Live in Exciting Times ACM (an international computing research society) has named CSCI-567: Machine Learning (Spring 2019) Prof. Victor Adamchik U of Southern California Apr. 2, 2019 Yoshua Bengio,
More informationNeural Networks. Intro to AI Bert Huang Virginia Tech
Neural Networks Intro to AI Bert Huang Virginia Tech Outline Biological inspiration for artificial neural networks Linear vs. nonlinear functions Learning with neural networks: back propagation https://en.wikipedia.org/wiki/neuron#/media/file:chemical_synapse_schema_cropped.jpg
More information