CSE 490 U Natural Language Processing Spring 2016
|
|
- Archibald Holt
- 5 years ago
- Views:
Transcription
1 CSE 490 U atural Language Processing Spring 2016 Parts of Speech Yejin Choi [Slides adapted from an Klein, Luke Zettlemoyer]
2 Overview POS Tagging Feature Rich Techniques Maximum Entropy Markov Models (MEMMs) Structured Perceptron Conditional Random Fields (CRFs)
3 Parts-of-Speech (English) One basic kind of linguistic structure: syntactic word classes Open class (lexical) words ouns erbs Adjectives yellow Proper Common Main Adverbs slowly IBM cat / cats see Italy snow registered umbers more 122,312 Closed class (functional) Modals one eterminers the some can Prepositions to with Conjunctions and or had Particles off up Pronouns he its more
4 Penn Treebank POS: 36 possible tags, 34 pages of tagging guidelines. CC conjunction, coordinating and both but either or C numeral, cardinal mid-1890 nine-thirty 0.5 one T determiner a all an every no that the EX existential there there FW foreign word gemeinschaft hund ich jeux I preposition or conjunction, subordinating among whether out on by if adjective or numeral, ordinal third ill-mannered regrettable R adjective, comparative braver cheaper taller S adjective, superlative bravest cheapest tallest M modal auxiliary can may might will would noun, common, singular or mass cabbage thermostat investment subhumanity P noun, proper, singular Motown Cougar Yvette Liverpool PS noun, proper, plural Americans Materials States S noun, common, plural undergraduates bric-a-brac averages POS genitive marker ' 's PRP pronoun, personal hers himself it we them PRP pronoun, possessive her his mine my our ours their thy your RB adverb occasionally maddeningly adventurously RBR adverb, comparative further gloomier heavier less-perfectly RBS adverb, superlative best biggest nearest worst RP particle aboard away back by on open through ftp://ftp.cis.upenn.edu/pub/treebank/doc/tagguide.ps.gz
5 PRP pronoun, personal hers himself it we them PRP pronoun, possessive her his mine my our ours their thy your RB adverb occasionally maddeningly adventurously RBR adverb, comparative further gloomier heavier less-perfectly RBS adverb, superlative best biggest nearest worst RP particle aboard away back by on open through TO "to" as preposition or infinitive marker to UH interjection huh howdy uh whammo shucks heck B verb, base form ask bring fire see take B verb, past tense pleaded swiped registered saw BG verb, present participle or gerund stirring focusing approaching erasing B verb, past participle dilapidated imitated reunifed unsettled BP verb, present tense, not 3rd person singular twist appear comprise mold postpone BZ verb, present tense, 3rd person singular bases reconstructs marks uses WT WH-determiner that what whatever which whichever WP WH-pronoun that what whatever which who whom WP WH-pronoun, possessive whose WRB Wh-adverb however whenever where why ftp://ftp.cis.upenn.edu/pub/treebank/doc/tagguide.ps.gz
6 Part-of-Speech Ambiguity Words can have multiple parts of speech B B B BZ BP BZ P S S C Fed raises interest rates 0.5 percent Two basic sources of constraint: Grammatical environment Identity of the current word Many more possible features: Suffixes, capitalization, name databases (gazetteers), etc
7 Why POS Tagging? Useful in and of itself (more than you d think) Text-to-speech: record, lead Lemmatization: saw[v] see, saw[n] saw Quick-and-dirty P-chunk detection: grep { }* { S} Useful as a pre-processing step for parsing Less tag ambiguity means fewer parses However, some tag choices are better decided by parsers I T P B B RP S The Georgia branch had taken on loan commitments T I B S B The average of interbank offered rates plummeted
8 Baselines and Upper Bounds Choose the most common tag 90.3% with a bad unknown word model 93.7% with a good one oise in the data Many errors in the training and test corpora Probably about 2% guaranteed error chief executive officer chief executive officer chief executive officer from noise (on this data) chief executive officer
9 Ambiguity in POS Tagging Particle (RP) vs. preposition (I) He talked over the deal. He talked over the telephone. past tense (B) vs. past participle (B) The horse walked past the barn. The horse walked past the barn fell. noun vs. adjective? The executive decision. noun vs. present participle Fishing can be fun
10 Ambiguity in POS Tagging Like can be a verb or a preposition I like/bp candy. Time flies like/i an arrow. Around can be a preposition, particle, or adverb I bought it at the shop around/i the corner. I never got around/rp to getting a car. A new Prius costs around/rb 25K. 10
11 Overview: Accuracies Roadmap of (known / unknown) accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% TnT (Brants, 2000): A carefully smoothed trigram tagger Suffix trees for emissions 96.7% on WS text (SOA is ~97.5%) Most errors on unknown words Upper bound: ~98%
12 Common Errors Common errors [from Toutanova & Manning 00] / official knowledge B RP/I T made up the story RB B/B S recently sold shares
13 What about better features? Choose the most common tag 90.3% with a bad unknown word model 93.7% with a good one What about looking at a word and its environment, but no sequence information? Add in previous / next word the Previous / next word shapes X X Occurrence pattern features [X: x X occurs] Crude entity detection.. (Inc. Co.) Phrasal verb in sentence? put Conjunctions of these things x 2 s 3 x 3 x 4 Uses lots of features: > 200K
14 Overview: Accuracies Roadmap of (known / unknown) accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% TnT (HMM++): 96.2% / 86.0% Maxent P(s i x): 96.8% / 86.8% Q: What does this say about sequence models? Q: How do we add more features to our sequence models? Upper bound: ~98%
15 MEMM Taggers One step up: also condition on previous tags p(s 1...s m x 1...x m )= Train up p(s i s i-1,x 1...x m ) as a discrete log-linear (maxent) model, then use to score sequences This is referred to as an MEMM tagger [Ratnaparkhi 96] Beam search effective! (Why?) my p(s i s 1...s i 1,x 1...x m ) i=1 What s the advantage of beam size 1? = my p(s i s i 1,x 1...x m ) i=1 p(s i s i 1,x 1...x m )= exp (w (x 1...x m,i,s i 1,s i )) P s 0 exp (w (x 1...x m,i,s i 1,s 0 ))
16 The HMM State Lattice / Trellis (repeat slide) START Fed raises interest rates STOP e(fed ) e(raises ) e(interest ) e(rates ) q( ) e(stop )
17 The MEMM State Lattice / Trellis x = START Fed raises interest rates STOP p(,x)
18 ecoding ecoding maxent taggers: ust like decoding HMMs iterbi, beam search, posterior decoding iterbi algorithm (HMMs): efine π(i,s i ) to be the max score of a sequence of length i ending in tag s i (i, s i ) = max e(x i s i )q(s i s i 1 ) (i 1,s i 1 ) s i 1 iterbi algorithm (Maxent): Can use same algorithm for MEMMs, just need to redefine π(i,s i )! (i, s i ) = max s i 1 p(s i s i 1,x 1...x m ) (i 1,s i 1 )
19 Overview: Accuracies Roadmap of (known / unknown) accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% TnT (HMM++): 96.2% / 86.0% Maxent P(s i x): 96.8% / 86.8% MEMM tagger: 96.9% / 86.9% Upper bound: ~98%
20 Global iscriminative Taggers ewer, higher-powered discriminative sequence models CRFs (also perceptrons, M3s) o not decompose training into independent local regions Can be deathly slow to train require repeated inference on training set ifferences can vary in importance, depending on task However: one issue worth knowing about in local models Label bias and other explaining away effects MEMM taggers local scores can be near one without having both good transitions and emissions This means that often evidence doesn t flow properly Why isn t this a big deal for POS tagging? Also: in decoding, condition on predicted, not gold, histories
21 Linear Models: Perceptron [Collins 02] The perceptron algorithm Iteratively processes the training set, reacting to training errors Can be thought of as trying to drive down training error The (online) perceptron algorithm: Start with zero weights isit training instances (x i,y i ) one by one Make a prediction y =argmax y If correct (y*==y i ): no change, goto next example! If wrong: adjust weights w (x i,y) Sentence: x=x 1 x m w = w + (x i,y i ) (x i,y ) Challenge: How to compute argmax efficiently? Tag Sequence: y=s 1 s m
22 Linear Perceptron ecoding s = arg max s Features must be local, for x=x 1 x m, and s=s 1 s m (x, s) = mx j=1 (x, j, s j 1,s j ) w (x, s)
23 The MEMM State Lattice / Trellis (repeat) x = START Fed raises interest rates STOP p(,x) x x x x
24 The Perceptron State Lattice / Trellis x = START Fed raises interest rates STOP wφ(x,3,,)
25 Linear Perceptron ecoding Features must be local, for x=x 1 x m, and s=s 1 s m (x, s) = mx j=1 efine π(i,s i ) to be the max score of a sequence of length i ending in tag s i iterbi algorithm (HMMs): s = arg max s (x, j, s j 1,s j ) w (x, s) (i, s i ) = max s i 1 w (x, i, s i i,s i )+ (i 1,s i 1 ) (i, s i ) = max s i 1 e(x i s i )q(s i s i 1 ) (i 1,s i 1 ) iterbi algorithm (Maxent): (i, s i ) = max s i 1 p(s i s i 1,x 1...x m ) (i 1,s i 1 )
26 Overview: Accuracies Roadmap of (known / unknown) accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% TnT (HMM++): 96.2% / 86.0% Maxent P(s i x): 96.8% / 86.8% MEMM tagger: 96.9% / 86.9% Perceptron 96.7% /?? Upper bound: ~98%
27 Conditional Random Fields (CRFs) Maximum entropy (logistic regression) Sentence: x=x 1 x m Tag Sequence: y=s 1 s m p(y x; w) = [Lafferty, McCallum, Pereira 01] exp (w (x, y)) P y 0 exp (w (x, y 0 )) Learning: maximize the (log) conditional likelihood of training data{(x i,y i )} j L(w) = nx i=1 j(x i,y i )! X p(y x i ; w) j (x i,y) y w j Computational Challenges? Most likely tag sequence, normalization constant, gradient
28 CRFs ecoding s = arg max s p(s x; w) Features must be local, for x=x 1 x m, and s=s 1 s m p(s x; w) = exp (w (x, s)) P s 0 exp (w (x, s 0 )) (x, s) = mx j=1 (x, j, s j 1,s j ) arg max s exp (w (x, s)) Ps exp (w (x, s 0 )) 0 = arg max s exp (w (x, s)) = arg max s w (x, s) Same as Linear Perceptron!!! (i, s i ) = max s i 1 (x, i, s i i,s i )+ (i 1,s i 1 )
29 CRFs: Computing ormalization* p(s x; w) = X s 0 exp w (x, s 0 ) exp (w (x, s)) P s exp (w (x, s 0 )) 0 = X X s 0 j = X Y exp (w (x, j, s j 1,s j )) s 0 j 0 (x, s) = w (x, j, s j 1,s j ) A j=1 1 efine norm(i,s i ) to sum of scores for sequences ending in position i norm(i, y i )= X s i 1 exp (w (x, i, s i 1,s i )) norm(i 1,s i 1 ) mx (x, j, s j 1,s j ) Forward Algorithm! Remember HMM case: (i, y i )= X y i 1 e(x i y i )q(y i y i 1 ) (i 1,y i 1 ) Could also use backward?
30 p(s x; w) = CRFs: j L(w) = exp (w (x, s)) P s exp (w (x, s 0 )) 0 nx i=1 X p(s x i ; w) s = mx X j=1 a,b eed forward and backward messages See notes for full details! j(x i,s i )! X p(s x i ; w) j (x i,s) s j (x i,s) = X s X s:s j 1 =a,s b =b (x, s) = p(s x i ; w) mx j=1 mx j=1 p(s x i ; w) k (x i,j,s j 1,s j ) (x, j, s j 1,s j ) w j k(x i,j,s j 1,s j )
31 Overview: Accuracies Roadmap of (known / unknown) accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% TnT (HMM++): 96.2% / 86.0% Maxent P(s i x): 96.8% / 86.8% MEMM tagger: 96.9% / 86.9% Perceptron 96.7% /?? CRF (untuned) 95.7% / 76.2% Upper bound: ~98%
32 Cyclic etwork [Toutanova et al 03] Train two MEMMs, multiple together to score And be very careful Tune regularization Try lots of different features See paper for full details (a) Left-to-Right CMM (b) Right-to-Left CMM (c) Bidirectional ependency etwork
33 Overview: Accuracies Roadmap of (known / unknown) accuracies: Most freq tag: ~90% / ~50% Trigram HMM: ~95% / ~55% TnT (HMM++): 96.2% / 86.0% Maxent P(s i x): 96.8% / 86.8% MEMM tagger: 96.9% / 86.9% Perceptron 96.7% /?? CRF (untuned) 95.7% / 76.2% Cyclic tagger: 97.2% / 89.0% Upper bound: ~98%
34 omain Effects Accuracies degrade outside of domain Up to triple error rate Usually make the most errors on the things you care about in the domain (e.g. protein names) Open questions How to effectively exploit unlabeled data from a new domain (what could we gain?) How to best incorporate domain lexica in a principled way (e.g. UMLS specialist lexicon, ontologies)
Parts-of-Speech (English) Statistical NLP Spring Part-of-Speech Ambiguity. Why POS Tagging? Classic Solution: HMMs. Lecture 6: POS / Phrase MT
Statistical LP Spring 2011 Lecture 6: POS / Phrase MT an Klein UC Berkeley Parts-of-Speech (English) One basic kind of linguistic structure: syntactic word classes Open class (lexical) words ouns Proper
More informationStatistical NLP Spring Parts-of-Speech (English)
Statistical LP Spring 2009 Lecture 6: Parts-of-Speech an Klein UC Berkeley Parts-of-Speech (English) One basic kind of linguistic structure: syntactic word classes Open class (lexical) words ouns erbs
More informationCSE 517 Natural Language Processing Winter2015
CSE 517 Natural Language Processing Winter2015 Feature Rich Models Sameer Singh Guest lecture for Yejin Choi - University of Washington [Slides from Jason Eisner, Dan Klein, Luke ZeLlemoyer] Outline POS
More informationNatural Language Processing
Natural Language Processing Part of Speech Tagging Dan Klein UC Berkeley 1 Parts of Speech 2 Parts of Speech (English) One basic kind of linguistic structure: syntactic word classes Open class (lexical)
More informationSequence Prediction and Part-of-speech Tagging
CS5740: atural Language Processing Spring 2017 Sequence Prediction and Part-of-speech Tagging Instructor: Yoav Artzi Slides adapted from Dan Klein, Dan Jurafsky, Chris Manning, Michael Collins, Luke Zettlemoyer,
More informationCSE 490 U Natural Language Processing Spring 2016
CSE 490 U Natural Language Processing Spring 2016 Feature Rich Models Yejin Choi - University of Washington [Many slides from Dan Klein, Luke Zettlemoyer] Structure in the output variable(s)? What is the
More informationLECTURER: BURCU CAN Spring
LECTURER: BURCU CAN 2018-2019 Spring Open class (lexical) words Nouns Verbs Adjectives yellow Proper Common Main Adverbs slowly IBM Italy cat / cats snow see registered Numbers more 122,312 Closed class
More informationCSE 447/547 Natural Language Processing Winter 2018
CSE 447/547 Natural Language Processing Winter 2018 Feature Rich Models (Log Linear Models) Yejin Choi University of Washington [Many slides from Dan Klein, Luke Zettlemoyer] Announcements HW #3 Due Feb
More information10/17/04. Today s Main Points
Part-of-speech Tagging & Hidden Markov Model Intro Lecture #10 Introduction to Natural Language Processing CMPSCI 585, Fall 2004 University of Massachusetts Amherst Andrew McCallum Today s Main Points
More informationHidden Markov Models
CS 2750: Machine Learning Hidden Markov Models Prof. Adriana Kovashka University of Pittsburgh March 21, 2016 All slides are from Ray Mooney Motivating Example: Part Of Speech Tagging Annotate each word
More informationHidden Markov Models (HMMs)
Hidden Markov Models HMMs Raymond J. Mooney University of Texas at Austin 1 Part Of Speech Tagging Annotate each word in a sentence with a part-of-speech marker. Lowest level of syntactic analysis. John
More informationEmpirical Methods in Natural Language Processing Lecture 11 Part-of-speech tagging and HMMs
Empirical Methods in Natural Language Processing Lecture 11 Part-of-speech tagging and HMMs (based on slides by Sharon Goldwater and Philipp Koehn) 21 February 2018 Nathan Schneider ENLP Lecture 11 21
More informationTime Zones - KET Grammar
Inventory of grammatical areas Verbs Regular and irregular forms Pages 104-105 (Unit 1 Modals can (ability; requests; permission) could (ability; polite requests) Page 3 (Getting Started) Pages 45-47,
More informationPart of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch. COMP-599 Oct 1, 2015
Part of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch COMP-599 Oct 1, 2015 Announcements Research skills workshop today 3pm-4:30pm Schulich Library room 313 Start thinking about
More informationGraphical models for part of speech tagging
Indian Institute of Technology, Bombay and Research Division, India Research Lab Graphical models for part of speech tagging Different Models for POS tagging HMM Maximum Entropy Markov Models Conditional
More informationCS838-1 Advanced NLP: Hidden Markov Models
CS838-1 Advanced NLP: Hidden Markov Models Xiaojin Zhu 2007 Send comments to jerryzhu@cs.wisc.edu 1 Part of Speech Tagging Tag each word in a sentence with its part-of-speech, e.g., The/AT representative/nn
More informationLog-Linear Models, MEMMs, and CRFs
Log-Linear Models, MEMMs, and CRFs Michael Collins 1 Notation Throughout this note I ll use underline to denote vectors. For example, w R d will be a vector with components w 1, w 2,... w d. We use expx
More informationConditional Random Fields and beyond DANIEL KHASHABI CS 546 UIUC, 2013
Conditional Random Fields and beyond DANIEL KHASHABI CS 546 UIUC, 2013 Outline Modeling Inference Training Applications Outline Modeling Problem definition Discriminative vs. Generative Chain CRF General
More informationNatural Language Processing Winter 2013
Natural Language Processing Winter 2013 Hidden Markov Models Luke Zettlemoyer - University of Washington [Many slides from Dan Klein and Michael Collins] Overview Hidden Markov Models Learning Supervised:
More informationPosterior vs. Parameter Sparsity in Latent Variable Models Supplementary Material
Posterior vs. Parameter Sparsity in Latent Variable Models Supplementary Material João V. Graça L 2 F INESC-ID Lisboa, Portugal Kuzman Ganchev Ben Taskar University of Pennsylvania Philadelphia, PA, USA
More informationLecture 13: Discriminative Sequence Models (MEMM and Struct. Perceptron)
Lecture 13: Discriminative Sequence Models (MEMM and Struct. Perceptron) Intro to NLP, CS585, Fall 2014 http://people.cs.umass.edu/~brenocon/inlp2014/ Brendan O Connor (http://brenocon.com) 1 Models for
More informationCSE 447 / 547 Natural Language Processing Winter 2018
CSE 447 / 547 Natural Language Processing Winter 2018 Hidden Markov Models Yejin Choi University of Washington [Many slides from Dan Klein, Michael Collins, Luke Zettlemoyer] Overview Hidden Markov Models
More informationINF4820: Algorithms for Artificial Intelligence and Natural Language Processing. Hidden Markov Models
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov Models Murhaf Fares & Stephan Oepen Language Technology Group (LTG) October 18, 2017 Recap: Probabilistic Language
More informationLecture 13: Structured Prediction
Lecture 13: Structured Prediction Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/nlp16 CS6501: NLP 1 Quiz 2 v Lectures 9-13 v Lecture 12: before page
More informationProbabilistic Models for Sequence Labeling
Probabilistic Models for Sequence Labeling Besnik Fetahu June 9, 2011 Besnik Fetahu () Probabilistic Models for Sequence Labeling June 9, 2011 1 / 26 Background & Motivation Problem introduction Generative
More informationSequence Modelling with Features: Linear-Chain Conditional Random Fields. COMP-599 Oct 6, 2015
Sequence Modelling with Features: Linear-Chain Conditional Random Fields COMP-599 Oct 6, 2015 Announcement A2 is out. Due Oct 20 at 1pm. 2 Outline Hidden Markov models: shortcomings Generative vs. discriminative
More informationStatistical NLP for the Web Log Linear Models, MEMM, Conditional Random Fields
Statistical NLP for the Web Log Linear Models, MEMM, Conditional Random Fields Sameer Maskey Week 13, Nov 28, 2012 1 Announcements Next lecture is the last lecture Wrap up of the semester 2 Final Project
More informationCS388: Natural Language Processing Lecture 4: Sequence Models I
CS388: Natural Language Processing Lecture 4: Sequence Models I Greg Durrett Mini 1 due today Administrivia Project 1 out today, due September 27 Viterbi algorithm, CRF NER system, extension Extension
More informationNatural Language Processing Prof. Pawan Goyal Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur
Natural Language Processing Prof. Pawan Goyal Department of Computer Science and Engineering Indian Institute of Technology, Kharagpur Lecture - 18 Maximum Entropy Models I Welcome back for the 3rd module
More informationINF4820: Algorithms for Artificial Intelligence and Natural Language Processing. Hidden Markov Models
INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Hidden Markov Models Murhaf Fares & Stephan Oepen Language Technology Group (LTG) October 27, 2016 Recap: Probabilistic Language
More informationNatural Language Processing (CSE 517): Sequence Models (II)
Natural Language Processing (CSE 517): Sequence Models (II) Noah Smith c 2016 University of Washington nasmith@cs.washington.edu February 3, 2016 1 / 40 Full Viterbi Procedure Input: x, θ, γ, π Output:
More informationACS Introduction to NLP Lecture 2: Part of Speech (POS) Tagging
ACS Introduction to NLP Lecture 2: Part of Speech (POS) Tagging Stephen Clark Natural Language and Information Processing (NLIP) Group sc609@cam.ac.uk The POS Tagging Problem 2 England NNP s POS fencers
More informationSequential Supervised Learning
Sequential Supervised Learning Many Application Problems Require Sequential Learning Part-of of-speech Tagging Information Extraction from the Web Text-to to-speech Mapping Part-of of-speech Tagging Given
More informationProbabilistic Graphical Models
CS 1675: Intro to Machine Learning Probabilistic Graphical Models Prof. Adriana Kovashka University of Pittsburgh November 27, 2018 Plan for This Lecture Motivation for probabilistic graphical models Directed
More informationConditional Random Fields for Sequential Supervised Learning
Conditional Random Fields for Sequential Supervised Learning Thomas G. Dietterich Adam Ashenfelter Department of Computer Science Oregon State University Corvallis, Oregon 97331 http://www.eecs.oregonstate.edu/~tgd
More informationSequence labeling. Taking collective a set of interrelated instances x 1,, x T and jointly labeling them
HMM, MEMM and CRF 40-957 Special opics in Artificial Intelligence: Probabilistic Graphical Models Sharif University of echnology Soleymani Spring 2014 Sequence labeling aking collective a set of interrelated
More informationNatural Language Processing (CSE 517): Sequence Models
Natural Language Processing (CSE 517): Sequence Models Noah Smith c 2018 University of Washington nasmith@cs.washington.edu April 27, 2018 1 / 60 High-Level View of Viterbi The decision about Y l is a
More informationNatural Language Processing
Natural Language Processing Global linear models Based on slides from Michael Collins Globally-normalized models Why do we decompose to a sequence of decisions? Can we directly estimate the probability
More informationMachine Learning for Structured Prediction
Machine Learning for Structured Prediction Grzegorz Chrupa la National Centre for Language Technology School of Computing Dublin City University NCLT Seminar Grzegorz Chrupa la (DCU) Machine Learning for
More informationMore on HMMs and other sequence models. Intro to NLP - ETHZ - 18/03/2013
More on HMMs and other sequence models Intro to NLP - ETHZ - 18/03/2013 Summary Parts of speech tagging HMMs: Unsupervised parameter estimation Forward Backward algorithm Bayesian variants Discriminative
More informationLecture 12: Algorithms for HMMs
Lecture 12: Algorithms for HMMs Nathan Schneider (some slides from Sharon Goldwater; thanks to Jonathan May for bug fixes) ENLP 26 February 2018 Recap: tagging POS tagging is a sequence labelling task.
More informationStatistical Methods for NLP
Statistical Methods for NLP Sequence Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Statistical Methods for NLP 1(21) Introduction Structured
More informationCOMP (Fall 2017) Natural Language Processing (with deep learning and connections to vision/robotics)
COMP 790.139 (Fall 2017) Natural Language Processing (with deep learning and connections to vision/robotics) Lecture 3: POS-Tagging, NER, Seq Labeling, Coreference Mohit Bansal (various slides adapted/borrowed
More informationNatural Language Processing
SFU NatLangLab Natural Language Processing Anoop Sarkar anoopsarkar.github.io/nlp-class Simon Fraser University October 9, 2018 0 Natural Language Processing Anoop Sarkar anoopsarkar.github.io/nlp-class
More informationNatural Language Processing CS Lecture 06. Razvan C. Bunescu School of Electrical Engineering and Computer Science
Natural Language Processing CS 6840 Lecture 06 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Statistical Parsing Define a probabilistic model of syntax P(T S):
More informationPart-of-Speech Tagging + Neural Networks CS 287
Part-of-Speech Tagging + Neural Networks CS 287 Quiz Last class we focused on hinge loss. L hinge = max{0, 1 (ŷ c ŷ c )} Consider now the squared hinge loss, (also called l 2 SVM) L hinge 2 = max{0, 1
More informationWhat s an HMM? Extraction with Finite State Machines e.g. Hidden Markov Models (HMMs) Hidden Markov Models (HMMs) for Information Extraction
Hidden Markov Models (HMMs) for Information Extraction Daniel S. Weld CSE 454 Extraction with Finite State Machines e.g. Hidden Markov Models (HMMs) standard sequence model in genomics, speech, NLP, What
More informationINF4820: Algorithms for Artificial Intelligence and Natural Language Processing. Language Models & Hidden Markov Models
1 University of Oslo : Department of Informatics INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Language Models & Hidden Markov Models Stephan Oepen & Erik Velldal Language
More informationA.I. in health informatics lecture 8 structured learning. kevin small & byron wallace
A.I. in health informatics lecture 8 structured learning kevin small & byron wallace today models for structured learning: HMMs and CRFs structured learning is particularly useful in biomedical applications:
More informationLecture 12: Algorithms for HMMs
Lecture 12: Algorithms for HMMs Nathan Schneider (some slides from Sharon Goldwater; thanks to Jonathan May for bug fixes) ENLP 17 October 2016 updated 9 September 2017 Recap: tagging POS tagging is a
More informationA Context-Free Grammar
Statistical Parsing A Context-Free Grammar S VP VP Vi VP Vt VP VP PP DT NN PP PP P Vi sleeps Vt saw NN man NN dog NN telescope DT the IN with IN in Ambiguity A sentence of reasonable length can easily
More informationBasic Text Analysis. Hidden Markov Models. Joakim Nivre. Uppsala University Department of Linguistics and Philology
Basic Text Analysis Hidden Markov Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakimnivre@lingfiluuse Basic Text Analysis 1(33) Hidden Markov Models Markov models are
More informationHidden Markov Models in Language Processing
Hidden Markov Models in Language Processing Dustin Hillard Lecture notes courtesy of Prof. Mari Ostendorf Outline Review of Markov models What is an HMM? Examples General idea of hidden variables: implications
More information10 : HMM and CRF. 1 Case Study: Supervised Part-of-Speech Tagging
10-708: Probabilistic Graphical Models 10-708, Spring 2018 10 : HMM and CRF Lecturer: Kayhan Batmanghelich Scribes: Ben Lengerich, Michael Kleyman 1 Case Study: Supervised Part-of-Speech Tagging We will
More informationStatistical Machine Learning Theory. From Multi-class Classification to Structured Output Prediction. Hisashi Kashima.
http://goo.gl/xilnmn Course website KYOTO UNIVERSITY Statistical Machine Learning Theory From Multi-class Classification to Structured Output Prediction Hisashi Kashima kashima@i.kyoto-u.ac.jp DEPARTMENT
More informationLecture 7: Sequence Labeling
http://courses.engr.illinois.edu/cs447 Lecture 7: Sequence Labeling Julia Hockenmaier juliahmr@illinois.edu 3324 Siebel Center Recap: Statistical POS tagging with HMMs (J. Hockenmaier) 2 Recap: Statistical
More informationPart-of-Speech Tagging
Part-of-Speech Tagging Informatics 2A: Lecture 17 Adam Lopez School of Informatics University of Edinburgh 27 October 2016 1 / 46 Last class We discussed the POS tag lexicon When do words belong to the
More informationHMM and Part of Speech Tagging. Adam Meyers New York University
HMM and Part of Speech Tagging Adam Meyers New York University Outline Parts of Speech Tagsets Rule-based POS Tagging HMM POS Tagging Transformation-based POS Tagging Part of Speech Tags Standards There
More informationTnT Part of Speech Tagger
TnT Part of Speech Tagger By Thorsten Brants Presented By Arghya Roy Chaudhuri Kevin Patel Satyam July 29, 2014 1 / 31 Outline 1 Why Then? Why Now? 2 Underlying Model Other technicalities 3 Evaluation
More informationHidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 19 Nov. 5, 2018 1 Reminders Homework
More informationStatistical Machine Learning Theory. From Multi-class Classification to Structured Output Prediction. Hisashi Kashima.
http://goo.gl/jv7vj9 Course website KYOTO UNIVERSITY Statistical Machine Learning Theory From Multi-class Classification to Structured Output Prediction Hisashi Kashima kashima@i.kyoto-u.ac.jp DEPARTMENT
More informationNatural Language Processing. Classification. Features. Some Definitions. Classification. Feature Vectors. Classification I. Dan Klein UC Berkeley
Natural Language Processing Classification Classification I Dan Klein UC Berkeley Classification Automatically make a decision about inputs Example: document category Example: image of digit digit Example:
More informationUndirected Graphical Models
Outline Hong Chang Institute of Computing Technology, Chinese Academy of Sciences Machine Learning Methods (Fall 2012) Outline Outline I 1 Introduction 2 Properties Properties 3 Generative vs. Conditional
More informationStructured Prediction
Structured Prediction Classification Algorithms Classify objects x X into labels y Y First there was binary: Y = {0, 1} Then multiclass: Y = {1,...,6} The next generation: Structured Labels Structured
More informationMidterm sample questions
Midterm sample questions CS 585, Brendan O Connor and David Belanger October 12, 2014 1 Topics on the midterm Language concepts Translation issues: word order, multiword translations Human evaluation Parts
More informationMaximum Entropy Markov Models
Wi nøt trei a høliday in Sweden this yër? September 19th 26 Background Preliminary CRF work. Published in 2. Authors: McCallum, Freitag and Pereira. Key concepts: Maximum entropy / Overlapping features.
More informationConditional Random Field
Introduction Linear-Chain General Specific Implementations Conclusions Corso di Elaborazione del Linguaggio Naturale Pisa, May, 2011 Introduction Linear-Chain General Specific Implementations Conclusions
More informationHidden Markov Models
Hidden Markov Models Slides mostly from Mitch Marcus and Eric Fosler (with lots of modifications). Have you seen HMMs? Have you seen Kalman filters? Have you seen dynamic programming? HMMs are dynamic
More informationIntelligent Systems (AI-2)
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 19 Oct, 24, 2016 Slide Sources Raymond J. Mooney University of Texas at Austin D. Koller, Stanford CS - Probabilistic Graphical Models D. Page,
More informationCSCI 5832 Natural Language Processing. Today 2/19. Statistical Sequence Classification. Lecture 9
CSCI 5832 Natural Language Processing Jim Martin Lecture 9 1 Today 2/19 Review HMMs for POS tagging Entropy intuition Statistical Sequence classifiers HMMs MaxEnt MEMMs 2 Statistical Sequence Classification
More informationText Mining. March 3, March 3, / 49
Text Mining March 3, 2017 March 3, 2017 1 / 49 Outline Language Identification Tokenisation Part-Of-Speech (POS) tagging Hidden Markov Models - Sequential Taggers Viterbi Algorithm March 3, 2017 2 / 49
More informationInformation Extraction from Text
Information Extraction from Text Jing Jiang Chapter 2 from Mining Text Data (2012) Presented by Andrew Landgraf, September 13, 2013 1 What is Information Extraction? Goal is to discover structured information
More informationApplied Natural Language Processing
Applied Natural Language Processing Info 256 Lecture 20: Sequence labeling (April 9, 2019) David Bamman, UC Berkeley POS tagging NNP Labeling the tag that s correct for the context. IN JJ FW SYM IN JJ
More informationPartially Directed Graphs and Conditional Random Fields. Sargur Srihari
Partially Directed Graphs and Conditional Random Fields Sargur srihari@cedar.buffalo.edu 1 Topics Conditional Random Fields Gibbs distribution and CRF Directed and Undirected Independencies View as combination
More informationStructured Output Prediction: Generative Models
Structured Output Prediction: Generative Models CS6780 Advanced Machine Learning Spring 2015 Thorsten Joachims Cornell University Reading: Murphy 17.3, 17.4, 17.5.1 Structured Output Prediction Supervised
More informationStatistical Methods for NLP
Statistical Methods for NLP Information Extraction, Hidden Markov Models Sameer Maskey Week 5, Oct 3, 2012 *many slides provided by Bhuvana Ramabhadran, Stanley Chen, Michael Picheny Speech Recognition
More informationA brief introduction to Conditional Random Fields
A brief introduction to Conditional Random Fields Mark Johnson Macquarie University April, 2005, updated October 2010 1 Talk outline Graphical models Maximum likelihood and maximum conditional likelihood
More informationMachine Learning for natural language processing
Machine Learning for natural language processing Classification: Maximum Entropy Models Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Summer 2016 1 / 24 Introduction Classification = supervised
More informationThe Noisy Channel Model and Markov Models
1/24 The Noisy Channel Model and Markov Models Mark Johnson September 3, 2014 2/24 The big ideas The story so far: machine learning classifiers learn a function that maps a data item X to a label Y handle
More informationIntelligent Systems (AI-2)
Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 19 Oct, 23, 2015 Slide Sources Raymond J. Mooney University of Texas at Austin D. Koller, Stanford CS - Probabilistic Graphical Models D. Page,
More informationConditional Random Fields
Conditional Random Fields Micha Elsner February 14, 2013 2 Sums of logs Issue: computing α forward probabilities can undeflow Normally we d fix this using logs But α requires a sum of probabilities Not
More informationHidden Markov Models
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Hidden Markov Models Matt Gormley Lecture 22 April 2, 2018 1 Reminders Homework
More informationHidden Markov Models
CS769 Spring 2010 Advanced Natural Language Processing Hidden Markov Models Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu 1 Part-of-Speech Tagging The goal of Part-of-Speech (POS) tagging is to label each
More informationFeatures of Statistical Parsers
Features of tatistical Parsers Preliminary results Mark Johnson Brown University TTI, October 2003 Joint work with Michael Collins (MIT) upported by NF grants LI 9720368 and II0095940 1 Talk outline tatistical
More informationToday s Agenda. Need to cover lots of background material. Now on to the Map Reduce stuff. Rough conceptual sketch of unsupervised training using EM
Today s Agenda Need to cover lots of background material l Introduction to Statistical Models l Hidden Markov Models l Part of Speech Tagging l Applying HMMs to POS tagging l Expectation-Maximization (EM)
More informationConditional Random Fields: An Introduction
University of Pennsylvania ScholarlyCommons Technical Reports (CIS) Department of Computer & Information Science 2-24-2004 Conditional Random Fields: An Introduction Hanna M. Wallach University of Pennsylvania
More informationA Support Vector Method for Multivariate Performance Measures
A Support Vector Method for Multivariate Performance Measures Thorsten Joachims Cornell University Department of Computer Science Thanks to Rich Caruana, Alexandru Niculescu-Mizil, Pierre Dupont, Jérôme
More informationMaschinelle Sprachverarbeitung
Maschinelle Sprachverarbeitung Parsing with Probabilistic Context-Free Grammar Ulf Leser Content of this Lecture Phrase-Structure Parse Trees Probabilistic Context-Free Grammars Parsing with PCFG Other
More informationNatural Language Processing
SFU NatLangLab Natural Language Processing Anoop Sarkar anoopsarkar.github.io/nlp-class Simon Fraser University September 27, 2018 0 Natural Language Processing Anoop Sarkar anoopsarkar.github.io/nlp-class
More informationConstituency Parsing
CS5740: Natural Language Processing Spring 2017 Constituency Parsing Instructor: Yoav Artzi Slides adapted from Dan Klein, Dan Jurafsky, Chris Manning, Michael Collins, Luke Zettlemoyer, Yejin Choi, and
More informationMaschinelle Sprachverarbeitung
Maschinelle Sprachverarbeitung Parsing with Probabilistic Context-Free Grammar Ulf Leser Content of this Lecture Phrase-Structure Parse Trees Probabilistic Context-Free Grammars Parsing with PCFG Other
More informationAlgorithms for NLP. Classification II. Taylor Berg-Kirkpatrick CMU Slides: Dan Klein UC Berkeley
Algorithms for NLP Classification II Taylor Berg-Kirkpatrick CMU Slides: Dan Klein UC Berkeley Minimize Training Error? A loss function declares how costly each mistake is E.g. 0 loss for correct label,
More informationPOS-Tagging. Fabian M. Suchanek
POS-Tagging Fabian M. Suchanek 100 Def: POS A Part-of-Speech (also: POS, POS-tag, word class, lexical class, lexical category) is a set of words with the same grammatical role. Alizée wrote a really great
More informationConditional Language Modeling. Chris Dyer
Conditional Language Modeling Chris Dyer Unconditional LMs A language model assigns probabilities to sequences of words,. w =(w 1,w 2,...,w`) It is convenient to decompose this probability using the chain
More informationDegree in Mathematics
Degree in Mathematics Title: Introduction to Natural Language Understanding and Chatbots. Author: Víctor Cristino Marcos Advisor: Jordi Saludes Department: Matemàtiques (749) Academic year: 2017-2018 Introduction
More informationEcient Higher-Order CRFs for Morphological Tagging
Ecient Higher-Order CRFs for Morphological Tagging Thomas Müller, Helmut Schmid and Hinrich Schütze Center for Information and Language Processing University of Munich Outline 1 Contributions 2 Motivation
More informationLECTURER: BURCU CAN Spring
LECTURER: BURCU CAN 2017-2018 Spring Regular Language Hidden Markov Model (HMM) Context Free Language Context Sensitive Language Probabilistic Context Free Grammar (PCFG) Unrestricted Language PCFGs can
More informationNatural Language Processing 1. lecture 7: constituent parsing. Ivan Titov. Institute for Logic, Language and Computation
atural Language Processing 1 lecture 7: constituent parsing Ivan Titov Institute for Logic, Language and Computation Outline Syntax: intro, CFGs, PCFGs PCFGs: Estimation CFGs: Parsing PCFGs: Parsing Parsing
More informationMACHINE LEARNING FOR NATURAL LANGUAGE PROCESSING
MACHINE LEARNING FOR NATURAL LANGUAGE PROCESSING Outline Some Sample NLP Task [Noah Smith] Structured Prediction For NLP Structured Prediction Methods Conditional Random Fields Structured Perceptron Discussion
More informationCPSC 340: Machine Learning and Data Mining
CPSC 340: Machine Learning and Data Mining Linear Classifiers: multi-class Original version of these slides by Mark Schmidt, with modifications by Mike Gelbart. 1 Admin Assignment 4: Due in a week Midterm:
More information