Probabilistic Language Modeling
|
|
- James Greer
- 5 years ago
- Views:
Transcription
1 Predicting String Probabilities Probabilistic Language Modeling Which string is more likely? (Which string is more grammatical?) Grill doctoral candidates. Regina Barzilay EECS Department MIT November 15, 2004 Grill doctoral updates. (example from Lee 1997) Methods for assigning probabilities to strings are called Language Models. Probabilistic Language Modeling 2/36 Last time Corpora processing Zipf s law Data sparseness Today: Probabilistic language Modeling Motivation Speech recognition, spelling correction, optical character recognition and other applications Let E be physical evidence, and we need to determine whether the string W is the message encoded by E Use Bayes Rule: P (W E) = P LM (W )P (E W ), P (E) where P LM (W ) is language model probability P LM (W ) provides the information necessary for disambiguation (esp. when the physical evidence is not sufficient for disambiguation) Probabilistic Language Modeling 1/36 Probabilistic Language Modeling 3/36
2 How to Compute it? The Language Modeling Problem Naive approach: Use the maximum likelihood estimates (MLE): the number of times that the string occurs in the corpus S, normalized by the corpus size P MLE ( Grill doctorate candidates ) = count( Grill doctorate candidates ) For unseen events, P MLE = 0 Dreadful behavior in the presence of Data Sparseness S Start with some vocabulary: ν = {the, a, doctorate, candidate, Professors, grill, cook, ask,...} Get a training sample of ν : Grill doctorate candidate. Cook Professors. Ask Professors.... Assumption: training sample is drawn from some underlying distribution P Probabilistic Language Modeling 4/36 Probabilistic Language Modeling 6/36 Two Famous Sentences The Language Modeling Problem It is fair to assume that neither sentence Colorless green ideas sleep furiously nor Furiously sleep ideas green colorless... has ever occurred... Hence, in any statistical model... these sentences will be ruled out on identical grounds as equally remote from English. Yet (1), though nonsensical, is grammatical, while (2) is not. [Chomsky 1957] Goal: learn a probability distribution P as close to P as possible x ν P (x) = 1 P (x) 0 for all x ν P (candidates) = 10 5 P (ask candidates) = 10 8 Probabilistic Language Modeling 5/36 Probabilistic Language Modeling 7/36
3 Deriving Language Model Assign probability to a sequence w 1 w 2... w n Apply chain rule: P (w 1 w 2... wn) = P (w 1 S) P (w 2 S, w 1 ) P (w 3 S, w 1, w 2 )... P (E S, w 1, w 2,..., wn) History-based model: we predict following things from past things How much context do we need to take into account? A Computational Model of Language A useful conceptual and practical device: coin-flipping models A sentence is generated by a randomized algorithm The generator can be one of several states Flip coins to choose the next state Flip other coins to decide which letter or word to output Shannon: The states will correspond to the residue of influence from preceding letters Probabilistic Language Modeling 8/36 Probabilistic Language Modeling 10/36 Markov Assumption For arbitrary long contexts P (w i w i n... w i 1 ) difficult to estimate Markov Assumption: w i depends only on n preceding words Trigrams (second order): P (w i ST ART, w 1, w 2,..., w i 1 ) = P (w i w i 1, w i 1 ) P (w 1 w 2... wn) = P (w 1 S) P (w 2 S, w 1 ) P (w 3 w 1, w 2 )... P (E w n 1, wn) Word-Based Approximations First-order approximation To him swallowed confess hear both. which. OF save on trail for are ay device and rote life have Every enter now severally so, let Hill he late speaks; or! a more to leg less first you enter Third-order approximation King Henry. What! I will go seek the traitor Gloucester. Exeunt some of the watch. A great banquet serv s in; Will you tell me how I am? It cannot be but so. Probabilistic Language Modeling 9/36 Probabilistic Language Modeling 11/36
4 Evaluating a Language Model We have n test string: S 1, S 2,..., S n Consider the probability under our model n i=1 P (S i), or log probability: Perplexity: log n P (S i ) = i=1 n log P (S i ) i=1 P erplexity = 2 x, where x = 1 W n i=1 log P (S i) and W is the total number of words in the test data. Perplexity estimate of human performance (Shannon, 1951) Shannon game humans guess next letter in text PP=142(1.3 bits/letter), uncased, open vocabulary estimate of trigram language model (Brown et al. 1992) PP=790(1.75 bits/letter), cased, open vocabulary Probabilistic Language Modeling 12/36 Probabilistic Language Modeling 14/36 Perplexity Perplexity is a measure of effective branching factor We have a vocabulary ν of size N, and model predicts for all the words in ν. What about Perplexity? P (w) = 1 N Maximum Likelihood Estimate MLE makes training data as probable as possible P ML (w i w i 1, w i 2 ) = Count(w i 2, w i 1, w i ) Count(w i 2, w i 1 ) For vocabulary of size N, we will have N 3 parameters in the model P erplexity = 2 x, where x = log 1 N P erplexity = N For N = 1, 000, we have to estimate 1, = 10 9 parameters Problem: how to deal with unseen words? Probabilistic Language Modeling 13/36 Probabilistic Language Modeling 15/36
5 Sparsity Add-One (Laplace) Smoothing Simplest discounting technique: The aggregate probability of unseen events constitutes a large fraction of the test data Brown et al (1992): considered a 350 million word corpus of English, 14% of trigrams are unseen P (w i w i 1 ) = where ν is a vocabulary size C(w1, w2) + 1 c(w1) + ν, Bayesian estimator assuming a uniform unit prior on events Problem: Too much probability mass to unseen events Probabilistic Language Modeling 16/36 Probabilistic Language Modeling 18/36 Today Example How to estimate probability of unseen elements? Assume ν = 10, 000, and S = 1, 000, 000 Discounting Laplace Good-Turing P MLE (ball kick a) = Count(kick a ball) Count(kick a) = 9 10 = 0.9 Linear Interpolation Katz Back-Off P +1 (ball kick a) = Count(kick a ball) + 1 Count(kick a) + ν = , 000 = Probabilistic Language Modeling 17/36 Probabilistic Language Modeling 19/36
6 Weaknesses of Laplace For Sparse distribution, Laplace s Law gives too much of the probability space to unseen events Worst at predicting the actual probabilities of bigrams than other methods More reasonable to use add-ɛ smoothing (Lidstone s Law) Good-Turing Discounting: Intuition Goal: estimate how often word with r counts in training data occurs in test set of equal size. We use deleted estimation: delete one word at a time if test word occurs r + 1 times in complete data set: it occurs r times in training set add one count to words with r counts total count placed to bucket for r-count words is: n r+1 (r + 1) (avg-count of r count words) = n r+1 (r+1) n r Probabilistic Language Modeling 20/36 Probabilistic Language Modeling 22/36 Good-Turing Discounting How likely are you to see a new word type in the future? Use things you ve seen once to estimate the probability of unseen things n r number of elements with r frequency and r > 0 n 0 size of the total lexicon minus the size of observed lexicon Good-Turing Discounting (cont.) In Good-Turing, the total probability assigned to all the unobserved events is equal to n 1 /N, where N is the size of the training set. It is the same as a relative frequency formula would assign to singleton events. Modified count for elements with frequency r r = (r + 1) n r+1 n r Probabilistic Language Modeling 21/36 Probabilistic Language Modeling 23/36
7 Example: Good-Turing Interpolation Training sample of 22,000,000 (Church&Gale 1991) r N r heldout r 0 74,671,100, ,018, , , , , One way of solving the sparseness in a trigram model is to mix that model with bigram and unigram models that suffer less from data sparseness The weights can be set using the Expectation-Maximization Algorithm or another numerical optimization technique 6 48, Probabilistic Language Modeling 24/36 Probabilistic Language Modeling 26/36 The Bias-Variance Trade-Off (Unsmoothed) trigram estimate Linear Interpolation P ML (w i w i 2, w i 1 ) = Count(w i 2w i 1 w i ) Count(w i 2, w i 1 ) (Unsmoothed) bigram estimate P ML (w i w i 1 ) = Count(w i 1w i ) Count(w i 1 ) (Unsmoothed) unigram estimate P ML (w i ) = Count(w i) Count(w j j) P (w i w i 2, w i 1 ) = λ 1 P ML (w i w i 2, w i 1 ) +λ 2 P ML (w i w i 1 ) +λ 3 P ML (w i ) where λ 1 + λ 2 + λ 3 = 1, and λ i 0 for all i. How close are these different estimates to the true probability P (w i w i 2, w i 1 )? Probabilistic Language Modeling 25/36 Probabilistic Language Modeling 27/36
8 This estimate defines distribution: w ν = P (w i w i 2, w i 1 ) w ν = λ 1 Linear Interpolation [λ 1 P ML (w i w i 2, w i 1 ) + λ 2 P ML (w i w i 1 ) + λ 3 P ML (w i )] λ 3 w ν w ν P ML (w i w i 2, w i 1 ) + λ 2 P ML (w i ) = λ 1 + λ 2 + λ 3 = 1 w ν P ML (w i w i 1 ) + An Iterative Method Initialization: Pick arbitrary/random values for λ 1, λ 2, λ 3 Step 1: Calculate the following quantities: c 1 = c 2 = c 3 = w 1,w 2,w 3 ν w 1,w 2,w 3 ν w 1,w 2,w 3 ν Step 2: Reestimate: λ i = Count 2 (w 1, w 2, w 3 )λ 1 P ML (w 3 w 2, w 1 ) λ 1 P ML (w i w i 2, w i 1 ) + λ 2 P ML (w i w i 1 ) + λ 3 P ML (w i ) Count 2 (w 1, w 2, w 3 )λ 2 P ML (w 3 w 2 ) λ 1 P ML (w i w i 2, w i 1 ) + λ 2 P ML (w i w i 1 ) + λ 3 P ML (w i ) Count 2 (w 1, w 2, w 3 )λ 3 P ML (w 3 ) λ 1 P ML (w i w i 2, w i 1 ) + λ 2 P ML (w i w i 1 ) + λ 3 P ML (w i ) c i c 1 +c 2 +c 3 Step 3: If λ i have not converged, go to Step 1. Probabilistic Language Modeling 28/36 Probabilistic Language Modeling 30/36 Parameter Estimation Hold out part of training set as validation data Define Count 2 (w 1, w 2, w 3 ) to be the number of times the trigram (w 1, w 2, w 3 ) is seen in validation set Choose λ i to maximize: L(λ 1, λ 2, λ 3 ) = w 1,w 2,w 3 ν such that λ 1 + λ 2 + λ 3 = 1, and λ i for all i. Count 2 (w 1, w 2, w 3 ) log P (w 3 w 1, w 2 ) Allowing the λ s to vary Partition histories (for instance, based on frequencies) Φ(w i 2, w i 1 ) = Condition λ s on partitions where λ Φ(w i 2,w i 1 ) 1 λ Φ(w i 2,w i 1 ) i P (w i w i 2, w i 1 ) = λ Φ(w i 2,w i 1 ) 1 { 1 if Count(wi 2, w i 1 ) = 0 +λ Φ(w i 2,w i 1 ) 2 +λ Φ(w i 2,w i 1 ) 3 0 for all i. 2 if 1 Count(w i 2, w i 1 ) 3 5 Otherwise + λ Φ(w i 2,w i 1 ) 2 P ML (w i w i 2, w i 1 ) P ML (w i w i 1 ) P ML (w i ) + λ Φ(w i 2,w i 1 ) 3 = 1, and Probabilistic Language Modeling 29/36 Probabilistic Language Modeling 31/36
9 Katz Back-Off Models (Bigrams) Define two sets Weaknesses of n-gram Models A(w i 1 ) = {w : Count(w i 1, w) > 0} B(w i 1 ) = {w : Count(w i 1, w) = 0} A bigram model P K (w i w i 1 ) = Count (w i 1,w) > 0 if w Count(w i 1 ) i A(w i 1 ) P α(w i 1 ) ML (w i ) if w i B(w i 1 ) w B(wi 1) P ML (w) Any ideas? Short-range Mid-range Long-range α(w i 1 ) = 1 w A(w i 1 ) Count (w i1, w) Count(w i 1 ) Probabilistic Language Modeling 32/36 Probabilistic Language Modeling 34/36 Count definitions More Refined Models Katz uses Good-Turing method for Count(x) < 5, and Count (x) = Count(x) for Count(x) 5 Kneser-Ney method: Count (x) = Count(x) D, whered = n 1 n 1 + n 2 n 1 is a number of elements with frequency 1 n 2 is a number of elements with frequency 2 Class-based models Structural models Topical and long-range models Probabilistic Language Modeling 33/36 Probabilistic Language Modeling 35/36
10 Summary Start with a vocabulary Select type of model Estimate Parameters Consider using CMU-Cambridge language modeling toolkit: prc14/toolkit.html Probabilistic Language Modeling 36/36
The Language Modeling Problem (Fall 2007) Smoothed Estimation, and Language Modeling. The Language Modeling Problem (Continued) Overview
The Language Modeling Problem We have some (finite) vocabulary, say V = {the, a, man, telescope, Beckham, two, } 6.864 (Fall 2007) Smoothed Estimation, and Language Modeling We have an (infinite) set of
More informationLanguage Modeling. Michael Collins, Columbia University
Language Modeling Michael Collins, Columbia University Overview The language modeling problem Trigram models Evaluating language models: perplexity Estimation techniques: Linear interpolation Discounting
More informationNatural Language Processing. Statistical Inference: n-grams
Natural Language Processing Statistical Inference: n-grams Updated 3/2009 Statistical Inference Statistical Inference consists of taking some data (generated in accordance with some unknown probability
More informationMachine Learning for natural language processing
Machine Learning for natural language processing N-grams and language models Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Summer 2016 1 / 25 Introduction Goals: Estimate the probability that a
More informationRecap: Language models. Foundations of Natural Language Processing Lecture 4 Language Models: Evaluation and Smoothing. Two types of evaluation in NLP
Recap: Language models Foundations of atural Language Processing Lecture 4 Language Models: Evaluation and Smoothing Alex Lascarides (Slides based on those from Alex Lascarides, Sharon Goldwater and Philipp
More informationN-gram Language Modeling
N-gram Language Modeling Outline: Statistical Language Model (LM) Intro General N-gram models Basic (non-parametric) n-grams Class LMs Mixtures Part I: Statistical Language Model (LM) Intro What is a statistical
More informationFoundations of Natural Language Processing Lecture 5 More smoothing and the Noisy Channel Model
Foundations of Natural Language Processing Lecture 5 More smoothing and the Noisy Channel Model Alex Lascarides (Slides based on those from Alex Lascarides, Sharon Goldwater and Philipop Koehn) 30 January
More informationN-grams. Motivation. Simple n-grams. Smoothing. Backoff. N-grams L545. Dept. of Linguistics, Indiana University Spring / 24
L545 Dept. of Linguistics, Indiana University Spring 2013 1 / 24 Morphosyntax We just finished talking about morphology (cf. words) And pretty soon we re going to discuss syntax (cf. sentences) In between,
More informationStatistical Methods for NLP
Statistical Methods for NLP Language Models, Graphical Models Sameer Maskey Week 13, April 13, 2010 Some slides provided by Stanley Chen and from Bishop Book Resources 1 Announcements Final Project Due,
More informationLanguage Modeling. Introduction to N-grams. Many Slides are adapted from slides by Dan Jurafsky
Language Modeling Introduction to N-grams Many Slides are adapted from slides by Dan Jurafsky Probabilistic Language Models Today s goal: assign a probability to a sentence Why? Machine Translation: P(high
More informationEmpirical Methods in Natural Language Processing Lecture 10a More smoothing and the Noisy Channel Model
Empirical Methods in Natural Language Processing Lecture 10a More smoothing and the Noisy Channel Model (most slides from Sharon Goldwater; some adapted from Philipp Koehn) 5 October 2016 Nathan Schneider
More informationLanguage Model. Introduction to N-grams
Language Model Introduction to N-grams Probabilistic Language Model Goal: assign a probability to a sentence Application: Machine Translation P(high winds tonight) > P(large winds tonight) Spelling Correction
More informationLanguage Models. Philipp Koehn. 11 September 2018
Language Models Philipp Koehn 11 September 2018 Language models 1 Language models answer the question: How likely is a string of English words good English? Help with reordering p LM (the house is small)
More informationN-gram Language Modeling Tutorial
N-gram Language Modeling Tutorial Dustin Hillard and Sarah Petersen Lecture notes courtesy of Prof. Mari Ostendorf Outline: Statistical Language Model (LM) Basics n-gram models Class LMs Cache LMs Mixtures
More informationLanguage Modeling. Introduction to N-grams. Many Slides are adapted from slides by Dan Jurafsky
Language Modeling Introduction to N-grams Many Slides are adapted from slides by Dan Jurafsky Probabilistic Language Models Today s goal: assign a probability to a sentence Why? Machine Translation: P(high
More informationLanguage Modelling: Smoothing and Model Complexity. COMP-599 Sept 14, 2016
Language Modelling: Smoothing and Model Complexity COMP-599 Sept 14, 2016 Announcements A1 has been released Due on Wednesday, September 28th Start code for Question 4: Includes some of the package import
More informationNatural Language Processing SoSe Language Modelling. (based on the slides of Dr. Saeedeh Momtazi)
Natural Language Processing SoSe 2015 Language Modelling Dr. Mariana Neves April 20th, 2015 (based on the slides of Dr. Saeedeh Momtazi) Outline 2 Motivation Estimation Evaluation Smoothing Outline 3 Motivation
More informationlanguage modeling: n-gram models
language modeling: n-gram models CS 585, Fall 2018 Introduction to Natural Language Processing http://people.cs.umass.edu/~miyyer/cs585/ Mohit Iyyer College of Information and Computer Sciences University
More informationDT2118 Speech and Speaker Recognition
DT2118 Speech and Speaker Recognition Language Modelling Giampiero Salvi KTH/CSC/TMH giampi@kth.se VT 2015 1 / 56 Outline Introduction Formal Language Theory Stochastic Language Models (SLM) N-gram Language
More informationNatural Language Processing SoSe Words and Language Model
Natural Language Processing SoSe 2016 Words and Language Model Dr. Mariana Neves May 2nd, 2016 Outline 2 Words Language Model Outline 3 Words Language Model Tokenization Separation of words in a sentence
More informationCMPT-825 Natural Language Processing
CMPT-825 Natural Language Processing Anoop Sarkar http://www.cs.sfu.ca/ anoop February 27, 2008 1 / 30 Cross-Entropy and Perplexity Smoothing n-gram Models Add-one Smoothing Additive Smoothing Good-Turing
More informationCS 6120/CS4120: Natural Language Processing
CS 6120/CS4120: Natural Language Processing Instructor: Prof. Lu Wang College of Computer and Information Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Today s Outline Probabilistic
More informationANLP Lecture 6 N-gram models and smoothing
ANLP Lecture 6 N-gram models and smoothing Sharon Goldwater (some slides from Philipp Koehn) 27 September 2018 Sharon Goldwater ANLP Lecture 6 27 September 2018 Recap: N-gram models We can model sentence
More informationLanguage Models. CS6200: Information Retrieval. Slides by: Jesse Anderton
Language Models CS6200: Information Retrieval Slides by: Jesse Anderton What s wrong with VSMs? Vector Space Models work reasonably well, but have a few problems: They are based on bag-of-words, so they
More informationLanguage Modeling. Introduction to N- grams
Language Modeling Introduction to N- grams Probabilistic Language Models Today s goal: assign a probability to a sentence Machine Translation: P(high winds tonite) > P(large winds tonite) Why? Spell Correction
More informationLanguage Modeling. Introduction to N- grams
Language Modeling Introduction to N- grams Probabilistic Language Models Today s goal: assign a probability to a sentence Machine Translation: P(high winds tonite) > P(large winds tonite) Why? Spell Correction
More informationSpeech Recognition Lecture 5: N-gram Language Models. Eugene Weinstein Google, NYU Courant Institute Slide Credit: Mehryar Mohri
Speech Recognition Lecture 5: N-gram Language Models Eugene Weinstein Google, NYU Courant Institute eugenew@cs.nyu.edu Slide Credit: Mehryar Mohri Components Acoustic and pronunciation model: Pr(o w) =
More informationCS 6120/CS4120: Natural Language Processing
CS 6120/CS4120: Natural Language Processing Instructor: Prof. Lu Wang College of Computer and Information Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Outline Probabilistic language
More information{ Jurafsky & Martin Ch. 6:! 6.6 incl.
N-grams Now Simple (Unsmoothed) N-grams Smoothing { Add-one Smoothing { Backo { Deleted Interpolation Reading: { Jurafsky & Martin Ch. 6:! 6.6 incl. 1 Word-prediction Applications Augmentative Communication
More informationACS Introduction to NLP Lecture 3: Language Modelling and Smoothing
ACS Introduction to NLP Lecture 3: Language Modelling and Smoothing Stephen Clark Natural Language and Information Processing (NLIP) Group sc609@cam.ac.uk Language Modelling 2 A language model is a probability
More informationGraphical Models. Mark Gales. Lent Machine Learning for Language Processing: Lecture 3. MPhil in Advanced Computer Science
Graphical Models Mark Gales Lent 2011 Machine Learning for Language Processing: Lecture 3 MPhil in Advanced Computer Science MPhil in Advanced Computer Science Graphical Models Graphical models have their
More informationLanguage Technology. Unit 1: Sequence Models. CUNY Graduate Center Spring Lectures 5-6: Language Models and Smoothing. required hard optional
Language Technology CUNY Graduate Center Spring 2013 Unit 1: Sequence Models Lectures 5-6: Language Models and Smoothing required hard optional Professor Liang Huang liang.huang.sh@gmail.com Python Review:
More informationCSA4050: Advanced Topics Natural Language Processing. Lecture Statistics III. Statistical Approaches to NLP
University of Malta BSc IT (Hons)Year IV CSA4050: Advanced Topics Natural Language Processing Lecture Statistics III Statistical Approaches to NLP Witten-Bell Discounting Unigrams Bigrams Dept Computer
More informationperplexity = 2 cross-entropy (5.1) cross-entropy = 1 N log 2 likelihood (5.2) likelihood = P(w 1 w N ) (5.3)
Chapter 5 Language Modeling 5.1 Introduction A language model is simply a model of what strings (of words) are more or less likely to be generated by a speaker of English (or some other language). More
More informationNatural Language Processing
Natural Language Processing Language models Based on slides from Michael Collins, Chris Manning and Richard Soccer Plan Problem definition Trigram models Evaluation Estimation Interpolation Discounting
More informationStatistical Natural Language Processing
Statistical Natural Language Processing N-gram Language Models Çağrı Çöltekin University of Tübingen Seminar für Sprachwissenschaft Summer Semester 2017 N-gram language models A language model answers
More informationLanguage Models. Data Science: Jordan Boyd-Graber University of Maryland SLIDES ADAPTED FROM PHILIP KOEHN
Language Models Data Science: Jordan Boyd-Graber University of Maryland SLIDES ADAPTED FROM PHILIP KOEHN Data Science: Jordan Boyd-Graber UMD Language Models 1 / 8 Language models Language models answer
More informationChapter 3: Basics of Language Modelling
Chapter 3: Basics of Language Modelling Motivation Language Models are used in Speech Recognition Machine Translation Natural Language Generation Query completion For research and development: need a simple
More informationLanguage Modeling. Introduc*on to N- grams. Many Slides are adapted from slides by Dan Jurafsky
Language Modeling Introduc*on to N- grams Many Slides are adapted from slides by Dan Jurafsky Probabilis1c Language Models Today s goal: assign a probability to a sentence Machine Transla*on: Why? P(high
More informationLanguage Modeling. Introduction to N-grams. Klinton Bicknell. (borrowing from: Dan Jurafsky and Jim Martin)
Language Modeling Introduction to N-grams Klinton Bicknell (borrowing from: Dan Jurafsky and Jim Martin) Probabilistic Language Models Today s goal: assign a probability to a sentence Why? Machine Translation:
More informationEmpirical Methods in Natural Language Processing Lecture 5 N-gram Language Models
Empirical Methods in Natural Language Processing Lecture 5 N-gram Language Models (most slides from Sharon Goldwater; some adapted from Alex Lascarides) 29 January 2017 Nathan Schneider ENLP Lecture 5
More informationNgram Review. CS 136 Lecture 10 Language Modeling. Thanks to Dan Jurafsky for these slides. October13, 2017 Professor Meteer
+ Ngram Review October13, 2017 Professor Meteer CS 136 Lecture 10 Language Modeling Thanks to Dan Jurafsky for these slides + ASR components n Feature Extraction, MFCCs, start of Acoustic n HMMs, the Forward
More informationAn Algorithm for Fast Calculation of Back-off N-gram Probabilities with Unigram Rescaling
An Algorithm for Fast Calculation of Back-off N-gram Probabilities with Unigram Rescaling Masaharu Kato, Tetsuo Kosaka, Akinori Ito and Shozo Makino Abstract Topic-based stochastic models such as the probabilistic
More informationWeek 13: Language Modeling II Smoothing in Language Modeling. Irina Sergienya
Week 13: Language Modeling II Smoothing in Language Modeling Irina Sergienya 07.07.2015 Couple of words first... There are much more smoothing techniques, [e.g. Katz back-off, Jelinek-Mercer,...] and techniques
More informationThe Noisy Channel Model and Markov Models
1/24 The Noisy Channel Model and Markov Models Mark Johnson September 3, 2014 2/24 The big ideas The story so far: machine learning classifiers learn a function that maps a data item X to a label Y handle
More informationNLP: N-Grams. Dan Garrette December 27, Predictive text (text messaging clients, search engines, etc)
NLP: N-Grams Dan Garrette dhg@cs.utexas.edu December 27, 2013 1 Language Modeling Tasks Language idenfication / Authorship identification Machine Translation Speech recognition Optical character recognition
More informationLanguage Processing with Perl and Prolog
Language Processing with Perl and Prolog Chapter 5: Counting Words Pierre Nugues Lund University Pierre.Nugues@cs.lth.se http://cs.lth.se/pierre_nugues/ Pierre Nugues Language Processing with Perl and
More informationNatural Language and Statistics
CS540 Intro to AI Natural Language and Statistics Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu 1 Common Preprocessing Steps Counting words alone gives interesting information. This is known as unigram word
More informationLecture 2: N-gram. Kai-Wei Chang University of Virginia Couse webpage:
Lecture 2: N-gram Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/nlp16 CS 6501: Natural Language Processing 1 This lecture Language Models What are
More informationLanguage Models. Hongning Wang
Language Models Hongning Wang CS@UVa Notion of Relevance Relevance (Rep(q), Rep(d)) Similarity P(r1 q,d) r {0,1} Probability of Relevance P(d q) or P(q d) Probabilistic inference Different rep & similarity
More informationAdministrivia. Lecture 5. Part I. Outline. The Big Picture IBM IBM IBM IBM
Administrivia Lecture 5 The Big Picture/Language Modeling Bhuvana Ramabhadran, Michael Picheny, Stanley F. Chen T.J. Watson Research Center Yorktown Heights, New York, USA {bhuvana,picheny,stanchen}@us.ibm.com
More informationLecture 5. The Big Picture/Language Modeling. Bhuvana Ramabhadran, Michael Picheny, Stanley F. Chen
Lecture 5 The Big Picture/Language Modeling Bhuvana Ramabhadran, Michael Picheny, Stanley F. Chen T.J. Watson Research Center Yorktown Heights, New York, USA {bhuvana,picheny,stanchen}@us.ibm.com 06 October
More informationSYNTHER A NEW M-GRAM POS TAGGER
SYNTHER A NEW M-GRAM POS TAGGER David Sündermann and Hermann Ney RWTH Aachen University of Technology, Computer Science Department Ahornstr. 55, 52056 Aachen, Germany {suendermann,ney}@cs.rwth-aachen.de
More informationNatural Language Processing (CSE 490U): Language Models
Natural Language Processing (CSE 490U): Language Models Noah Smith c 2017 University of Washington nasmith@cs.washington.edu January 6 9, 2017 1 / 67 Very Quick Review of Probability Event space (e.g.,
More informationCSCI 5832 Natural Language Processing. Today 2/19. Statistical Sequence Classification. Lecture 9
CSCI 5832 Natural Language Processing Jim Martin Lecture 9 1 Today 2/19 Review HMMs for POS tagging Entropy intuition Statistical Sequence classifiers HMMs MaxEnt MEMMs 2 Statistical Sequence Classification
More informationarxiv:cmp-lg/ v1 9 Jun 1997
arxiv:cmp-lg/9706007 v1 9 Jun 1997 Aggregate and mixed-order Markov models for statistical language processing Abstract We consider the use of language models whose size and accuracy are intermediate between
More informationMIA - Master on Artificial Intelligence
MIA - Master on Artificial Intelligence 1 Introduction Unsupervised & semi-supervised approaches Supervised Algorithms Maximum Likelihood Estimation Maximum Entropy Modeling Introduction 1 Introduction
More information(today we are assuming sentence segmentation) Wednesday, September 10, 14
(today we are assuming sentence segmentation) 1 Your TA: David Belanger http://people.cs.umass.edu/~belanger/ I am a third year PhD student advised by Professor Andrew McCallum. Before that, I was an Associate
More informationLanguage as a Stochastic Process
CS769 Spring 2010 Advanced Natural Language Processing Language as a Stochastic Process Lecturer: Xiaojin Zhu jerryzhu@cs.wisc.edu 1 Basic Statistics for NLP Pick an arbitrary letter x at random from any
More informationRanked Retrieval (2)
Text Technologies for Data Science INFR11145 Ranked Retrieval (2) Instructor: Walid Magdy 31-Oct-2017 Lecture Objectives Learn about Probabilistic models BM25 Learn about LM for IR 2 1 Recall: VSM & TFIDF
More informationCSCI 5832 Natural Language Processing. Today 1/31. Probability Basics. Lecture 6. Probability. Language Modeling (N-grams)
CSCI 5832 Natural Language Processing Jim Martin Lecture 6 1 Today 1/31 Probability Basic probability Conditional probability Bayes Rule Language Modeling (N-grams) N-gram Intro The Chain Rule Smoothing:
More informationSequences and Information
Sequences and Information Rahul Siddharthan The Institute of Mathematical Sciences, Chennai, India http://www.imsc.res.in/ rsidd/ Facets 16, 04/07/2016 This box says something By looking at the symbols
More informationLecture 4: Smoothing, Part-of-Speech Tagging. Ivan Titov Institute for Logic, Language and Computation Universiteit van Amsterdam
Lecture 4: Smoothing, Part-of-Speech Tagging Ivan Titov Institute for Logic, Language and Computation Universiteit van Amsterdam Language Models from Corpora We want a model of sentence probability P(w
More informationChapter 3: Basics of Language Modeling
Chapter 3: Basics of Language Modeling Section 3.1. Language Modeling in Automatic Speech Recognition (ASR) All graphs in this section are from the book by Schukat-Talamazzini unless indicated otherwise
More informationExploring Asymmetric Clustering for Statistical Language Modeling
Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL, Philadelphia, July 2002, pp. 83-90. Exploring Asymmetric Clustering for Statistical Language Modeling Jianfeng
More informationHidden Markov Models, I. Examples. Steven R. Dunbar. Toy Models. Standard Mathematical Models. Realistic Hidden Markov Models.
, I. Toy Markov, I. February 17, 2017 1 / 39 Outline, I. Toy Markov 1 Toy 2 3 Markov 2 / 39 , I. Toy Markov A good stack of examples, as large as possible, is indispensable for a thorough understanding
More informationHidden Markov Models in Language Processing
Hidden Markov Models in Language Processing Dustin Hillard Lecture notes courtesy of Prof. Mari Ostendorf Outline Review of Markov models What is an HMM? Examples General idea of hidden variables: implications
More informationDeep Learning Basics Lecture 10: Neural Language Models. Princeton University COS 495 Instructor: Yingyu Liang
Deep Learning Basics Lecture 10: Neural Language Models Princeton University COS 495 Instructor: Yingyu Liang Natural language Processing (NLP) The processing of the human languages by computers One of
More informationMidterm sample questions
Midterm sample questions CS 585, Brendan O Connor and David Belanger October 12, 2014 1 Topics on the midterm Language concepts Translation issues: word order, multiword translations Human evaluation Parts
More informationStatistical Machine Translation
Statistical Machine Translation Marcello Federico FBK-irst Trento, Italy Galileo Galilei PhD School University of Pisa Pisa, 7-19 May 2008 Part V: Language Modeling 1 Comparing ASR and statistical MT N-gram
More informationCross-Lingual Language Modeling for Automatic Speech Recogntion
GBO Presentation Cross-Lingual Language Modeling for Automatic Speech Recogntion November 14, 2003 Woosung Kim woosung@cs.jhu.edu Center for Language and Speech Processing Dept. of Computer Science The
More informationLanguage Modeling. Introduc*on to N- grams. Many Slides are adapted from slides by Dan Jurafsky
Language Modeling Introduc*on to N- grams Many Slides are adapted from slides by Dan Jurafsky Probabilis1c Language Models Today s goal: assign a probability to a sentence Machine Transla*on: Why? P(high
More informationMaximum Likelihood Estimation. only training data is available to design a classifier
Introduction to Pattern Recognition [ Part 5 ] Mahdi Vasighi Introduction Bayesian Decision Theory shows that we could design an optimal classifier if we knew: P( i ) : priors p(x i ) : class-conditional
More informationCSC401/2511 Spring CSC401/2511 Natural Language Computing Spring 2019 Lecture 5 Frank Rudzicz and Chloé Pou-Prom University of Toronto
CSC401/2511 Natural Language Computing Spring 2019 Lecture 5 Frank Rudzicz and Chloé Pou-Prom University of Toronto Revisiting PoS tagging Will/MD the/dt chair/nn chair/?? the/dt meeting/nn from/in that/dt
More informationarxiv: v1 [cs.cl] 5 Mar Introduction
Neural Machine Translation and Sequence-to-sequence Models: A Tutorial Graham Neubig Language Technologies Institute, Carnegie Mellon University arxiv:1703.01619v1 [cs.cl] 5 Mar 2017 1 Introduction This
More informationUniversität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Language Models. Tobias Scheffer
Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Language Models Tobias Scheffer Stochastic Language Models A stochastic language model is a probability distribution over words.
More informationTnT Part of Speech Tagger
TnT Part of Speech Tagger By Thorsten Brants Presented By Arghya Roy Chaudhuri Kevin Patel Satyam July 29, 2014 1 / 31 Outline 1 Why Then? Why Now? 2 Underlying Model Other technicalities 3 Evaluation
More informationHidden Markov Models. x 1 x 2 x 3 x N
Hidden Markov Models 1 1 1 1 K K K K x 1 x x 3 x N Example: The dishonest casino A casino has two dice: Fair die P(1) = P() = P(3) = P(4) = P(5) = P(6) = 1/6 Loaded die P(1) = P() = P(3) = P(4) = P(5)
More informationLanguage Modelling. Steve Renals. Automatic Speech Recognition ASR Lecture 11 6 March ASR Lecture 11 Language Modelling 1
Language Modelling Steve Renals Automatic Speech Recognition ASR Lecture 11 6 March 2017 ASR Lecture 11 Language Modelling 1 HMM Speech Recognition Recorded Speech Decoded Text (Transcription) Acoustic
More informationStatistical Methods for NLP
Statistical Methods for NLP Sequence Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Statistical Methods for NLP 1(21) Introduction Structured
More informationCS 224N HW:#3. (V N0 )δ N r p r + N 0. N r (r δ) + (V N 0)δ. N r r δ. + (V N 0)δ N = 1. 1 we must have the restriction: δ NN 0.
CS 224 HW:#3 ARIA HAGHIGHI SUID :# 05041774 1. Smoothing Probability Models (a). Let r be the number of words with r counts and p r be the probability for a word with r counts in the Absolute discounting
More informationImproved Decipherment of Homophonic Ciphers
Improved Decipherment of Homophonic Ciphers Malte Nuhn and Julian Schamper and Hermann Ney Human Language Technology and Pattern Recognition Computer Science Department, RWTH Aachen University, Aachen,
More informationConditional Language Modeling. Chris Dyer
Conditional Language Modeling Chris Dyer Unconditional LMs A language model assigns probabilities to sequences of words,. w =(w 1,w 2,...,w`) It is convenient to decompose this probability using the chain
More informationLecture 3: ASR: HMMs, Forward, Viterbi
Original slides by Dan Jurafsky CS 224S / LINGUIST 285 Spoken Language Processing Andrew Maas Stanford University Spring 2017 Lecture 3: ASR: HMMs, Forward, Viterbi Fun informative read on phonetics The
More informationHierarchical Bayesian Nonparametric Models of Language and Text
Hierarchical Bayesian Nonparametric Models of Language and Text Gatsby Computational Neuroscience Unit, UCL Joint work with Frank Wood *, Jan Gasthaus *, Cedric Archambeau, Lancelot James SIGIR Workshop
More informationHierarchical Bayesian Nonparametric Models of Language and Text
Hierarchical Bayesian Nonparametric Models of Language and Text Gatsby Computational Neuroscience Unit, UCL Joint work with Frank Wood *, Jan Gasthaus *, Cedric Archambeau, Lancelot James August 2010 Overview
More informationMore Smoothing, Tuning, and Evaluation
More Smoothing, Tuning, and Evaluation Nathan Schneider (slides adapted from Henry Thompson, Alex Lascarides, Chris Dyer, Noah Smith, et al.) ENLP 21 September 2016 1 Review: 2 Naïve Bayes Classifier w
More informationClassification & Information Theory Lecture #8
Classification & Information Theory Lecture #8 Introduction to Natural Language Processing CMPSCI 585, Fall 2007 University of Massachusetts Amherst Andrew McCallum Today s Main Points Automatically categorizing
More informationKneser-Ney smoothing explained
foldl home blog contact feed Kneser-Ney smoothing explained 18 January 2014 Language models are an essential element of natural language processing, central to tasks ranging from spellchecking to machine
More informationMicrosoft Corporation.
A Bit of Progress in Language Modeling Extended Version Joshua T. Goodman Machine Learning and Applied Statistics Group Microsoft Research One Microsoft Way Redmond, WA 98052 joshuago@microsoft.com August
More information8: Hidden Markov Models
8: Hidden Markov Models Machine Learning and Real-world Data Simone Teufel and Ann Copestake Computer Laboratory University of Cambridge Lent 2017 Last session: catchup 1 Research ideas from sentiment
More informationProbabilistic Spelling Correction CE-324: Modern Information Retrieval Sharif University of Technology
Probabilistic Spelling Correction CE-324: Modern Information Retrieval Sharif University of Technology M. Soleymani Fall 2016 Most slides have been adapted from: Profs. Manning, Nayak & Raghavan lectures
More informationUniversity of Illinois at Urbana-Champaign. Midterm Examination
University of Illinois at Urbana-Champaign Midterm Examination CS410 Introduction to Text Information Systems Professor ChengXiang Zhai TA: Azadeh Shakery Time: 2:00 3:15pm, Mar. 14, 2007 Place: Room 1105,
More informationIBM Model 1 for Machine Translation
IBM Model 1 for Machine Translation Micha Elsner March 28, 2014 2 Machine translation A key area of computational linguistics Bar-Hillel points out that human-like translation requires understanding of
More informationDesign and Implementation of Speech Recognition Systems
Design and Implementation of Speech Recognition Systems Spring 2013 Class 7: Templates to HMMs 13 Feb 2013 1 Recap Thus far, we have looked at dynamic programming for string matching, And derived DTW from
More informationBasic Text Analysis. Hidden Markov Models. Joakim Nivre. Uppsala University Department of Linguistics and Philology
Basic Text Analysis Hidden Markov Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakimnivre@lingfiluuse Basic Text Analysis 1(33) Hidden Markov Models Markov models are
More informationToday. Statistical Learning. Coin Flip. Coin Flip. Experiment 1: Heads. Experiment 1: Heads. Which coin will I use? Which coin will I use?
Today Statistical Learning Parameter Estimation: Maximum Likelihood (ML) Maximum A Posteriori (MAP) Bayesian Continuous case Learning Parameters for a Bayesian Network Naive Bayes Maximum Likelihood estimates
More informationPROBABILISTIC PASSWORD MODELING: PREDICTING PASSWORDS WITH MACHINE LEARNING
PROBABILISTIC PASSWORD MODELING: PREDICTING PASSWORDS WITH MACHINE LEARNING JAY DESTORIES MENTOR: ELIF YAMANGIL Abstract Many systems use passwords as the primary means of authentication. As the length
More informationCOS 424: Interacting with Data. Lecturer: Dave Blei Lecture #11 Scribe: Andrew Ferguson March 13, 2007
COS 424: Interacting with ata Lecturer: ave Blei Lecture #11 Scribe: Andrew Ferguson March 13, 2007 1 Graphical Models Wrap-up We began the lecture with some final words on graphical models. Choosing a
More informationCSC321 Lecture 7 Neural language models
CSC321 Lecture 7 Neural language models Roger Grosse and Nitish Srivastava February 1, 2015 Roger Grosse and Nitish Srivastava CSC321 Lecture 7 Neural language models February 1, 2015 1 / 19 Overview We
More information