Language Modeling. Michael Collins, Columbia University

Size: px
Start display at page:

Download "Language Modeling. Michael Collins, Columbia University"

Transcription

1 Language Modeling Michael Collins, Columbia University

2 Overview The language modeling problem Trigram models Evaluating language models: perplexity Estimation techniques: Linear interpolation Discounting methods

3 The Language Modeling Problem We have some (finite) vocabulary, say V = {the, a, man, telescope, Beckham, two,...} We have an (infinite) set of strings, V the STOP a STOP the fan STOP the fan saw Beckham STOP the fan saw saw STOP the fan saw Beckham play for Real Madrid STOP

4 The Language Modeling Problem (Continued) We have a training sample of example sentences in English

5 The Language Modeling Problem (Continued) We have a training sample of example sentences in English We need to learn a probability distribution p i.e., p is a function that satisfies x V p(x) = 1, p(x) 0 for all x V

6 The Language Modeling Problem (Continued) We have a training sample of example sentences in English We need to learn a probability distribution p i.e., p is a function that satisfies x V p(x) = 1, p(x) 0 for all x V p(the STOP) = p(the fan STOP) = 10 8 p(the fan saw Beckham STOP) = p(the fan saw saw STOP) = p(the fan saw Beckham play for Real Madrid STOP) =

7 Why on earth would we want to do this?! Speech recognition was the original motivation. (Related problems are optical character recognition, handwriting recognition.)

8 Why on earth would we want to do this?! Speech recognition was the original motivation. (Related problems are optical character recognition, handwriting recognition.) The estimation techniques developed for this problem will be VERY useful for other problems in NLP

9 A Naive Method We have N training sentences For any sentence x 1... x n, c(x 1... x n ) is the number of times the sentence is seen in our training data A naive estimate: p(x 1... x n ) = c(x 1... x n ) N

10 Overview The language modeling problem Trigram models Evaluating language models: perplexity Estimation techniques: Linear interpolation Discounting methods

11 Markov Processes Consider a sequence of random variables X 1, X 2,... X n. Each random variable can take any value in a finite set V. For now we assume the length n is fixed (e.g., n = 100). Our goal: model P (X 1 = x 1, X 2 = x 2,..., X n = x n )

12 First-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n )

13 First-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n ) n = P (X 1 = x 1 ) P (X i = x i X 1 = x 1,..., X i 1 = x i 1 ) i=2

14 First-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n ) n = P (X 1 = x 1 ) P (X i = x i X 1 = x 1,..., X i 1 = x i 1 ) = P (X 1 = x 1 ) i=2 n P (X i = x i X i 1 = x i 1 ) i=2

15 First-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n ) n = P (X 1 = x 1 ) P (X i = x i X 1 = x 1,..., X i 1 = x i 1 ) = P (X 1 = x 1 ) i=2 n P (X i = x i X i 1 = x i 1 ) i=2 The first-order Markov assumption: For any i {2... n}, for any x 1... x i, P (X i = x i X 1 = x 1... X i 1 = x i 1 ) = P (X i = x i X i 1 = x i 1 )

16 Second-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n )

17 Second-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n ) = P (X 1 = x 1 ) P (X 2 = x 2 X 1 = x 1 ) n P (X i = x i X i 2 = x i 2, X i 1 = x i 1 ) i=3

18 Second-Order Markov Processes P (X 1 = x 1, X 2 = x 2,... X n = x n ) = P (X 1 = x 1 ) P (X 2 = x 2 X 1 = x 1 ) n P (X i = x i X i 2 = x i 2, X i 1 = x i 1 ) = i=3 n P (X i = x i X i 2 = x i 2, X i 1 = x i 1 ) i=1 (For convenience we assume x 0 = x 1 = *, where * is a special start symbol.)

19 Modeling Variable Length Sequences We would like the length of the sequence, n, to also be a random variable A simple solution: always define X n = STOP where STOP is a special symbol

20 Modeling Variable Length Sequences We would like the length of the sequence, n, to also be a random variable A simple solution: always define X n = STOP where STOP is a special symbol Then use a Markov process as before: = P (X 1 = x 1, X 2 = x 2,... X n = x n ) n P (X i = x i X i 2 = x i 2, X i 1 = x i 1 ) i=1 (For convenience we assume x 0 = x 1 = *, where * is a special start symbol.)

21 Trigram Language Models A trigram language model consists of: 1. A finite set V 2. A parameter q(w u, v) for each trigram u, v, w such that w V {STOP}, and u, v V {*}.

22 Trigram Language Models A trigram language model consists of: 1. A finite set V 2. A parameter q(w u, v) for each trigram u, v, w such that w V {STOP}, and u, v V {*}. For any sentence x 1... x n where x i V for i = 1... (n 1), and x n = STOP, the probability of the sentence under the trigram language model is p(x 1... x n ) = n q(x i x i 2, x i 1 ) i=1 where we define x 0 = x 1 = *.

23 An Example For the sentence the dog barks STOP we would have p(the dog barks STOP) = q(the *, *) q(dog *, the) q(barks the, dog) q(stop dog, barks)

24 The Trigram Estimation Problem Remaining estimation problem: For example: q(w i w i 2, w i 1 ) q(laughs the, dog)

25 The Trigram Estimation Problem Remaining estimation problem: For example: q(w i w i 2, w i 1 ) q(laughs the, dog) A natural estimate (the maximum likelihood estimate ): q(w i w i 2, w i 1 ) = Count(w i 2, w i 1, w i ) Count(w i 2, w i 1 ) q(laughs the, dog) = Count(the, dog, laughs) Count(the, dog)

26 Sparse Data Problems A natural estimate (the maximum likelihood estimate ): q(w i w i 2, w i 1 ) = Count(w i 2, w i 1, w i ) Count(w i 2, w i 1 ) q(laughs the, dog) = Count(the, dog, laughs) Count(the, dog) Say our vocabulary size is N = V, then there are N 3 parameters in the model. e.g., N = 20, , = parameters

27 Overview The language modeling problem Trigram models Evaluating language models: perplexity Estimation techniques: Linear interpolation Discounting methods

28 Evaluating a Language Model: Perplexity We have some test data, m sentences s 1, s 2, s 3,..., s m

29 Evaluating a Language Model: Perplexity We have some test data, m sentences s 1, s 2, s 3,..., s m We could look at the probability under our model m i=1 p(s i). Or more conveniently, the log probability m log p(s i ) = i=1 m log p(s i ) i=1

30 Evaluating a Language Model: Perplexity We have some test data, m sentences s 1, s 2, s 3,..., s m We could look at the probability under our model m i=1 p(s i). Or more conveniently, the log probability m log p(s i ) = i=1 m log p(s i ) i=1 In fact the usual evaluation measure is perplexity Perplexity = 2 l where l = 1 M m log p(s i ) i=1 and M is the total number of words in the test data.

31 Some Intuition about Perplexity Say we have a vocabulary V, and N = V + 1 and model that predicts q(w u, v) = 1 N for all w V {STOP}, for all u, v V {*}. Easy to calculate the perplexity in this case: Perplexity = 2 l where l = log 1 N Perplexity = N Perplexity is a measure of effective branching factor

32 Typical Values of Perplexity Results from Goodman ( A bit of progress in language modeling ), where V = 50, 000 A trigram model: p(x 1... x n ) = n i=1 q(x i x i 2, x i 1 ). Perplexity = 74

33 Typical Values of Perplexity Results from Goodman ( A bit of progress in language modeling ), where V = 50, 000 A trigram model: p(x 1... x n ) = n i=1 q(x i x i 2, x i 1 ). Perplexity = 74 A bigram model: p(x 1... x n ) = n i=1 q(x i x i 1 ). Perplexity = 137

34 Typical Values of Perplexity Results from Goodman ( A bit of progress in language modeling ), where V = 50, 000 A trigram model: p(x 1... x n ) = n i=1 q(x i x i 2, x i 1 ). Perplexity = 74 A bigram model: p(x 1... x n ) = n i=1 q(x i x i 1 ). Perplexity = 137 A unigram model: p(x 1... x n ) = n i=1 q(x i). Perplexity = 955

35 Some History Shannon conducted experiments on entropy of English i.e., how good are people at the perplexity game? C. Shannon. Prediction and entropy of printed English. Bell Systems Technical Journal, 30:50 64, 1951.

36 Some History Chomsky (in Syntactic Structures (1957)): Second, the notion grammatical cannot be identified with meaningful or significant in any semantic sense. Sentences (1) and (2) are equally nonsensical, but any speaker of English will recognize that only the former is grammatical. (1) Colorless green ideas sleep furiously. (2) Furiously sleep ideas green colorless Third, the notion grammatical in English cannot be identified in any way with the notion high order of statistical approximation to English. It is fair to assume that neither sentence (1) nor (2) (nor indeed any part of these sentences) has ever occurred in an English discourse. Hence, in any statistical model for grammaticalness, these sentences will be ruled out on identical grounds as equally remote from English. Yet (1), though nonsensical, is grammatical, while (2) is not....

37 Overview The language modeling problem Trigram models Evaluating language models: perplexity Estimation techniques: Linear interpolation Discounting methods

38 Sparse Data Problems A natural estimate (the maximum likelihood estimate ): q(w i w i 2, w i 1 ) = Count(w i 2, w i 1, w i ) Count(w i 2, w i 1 ) q(laughs the, dog) = Count(the, dog, laughs) Count(the, dog) Say our vocabulary size is N = V, then there are N 3 parameters in the model. e.g., N = 20, , = parameters

39 The Bias-Variance Trade-Off Trigram maximum-likelihood estimate q ML (w i w i 2, w i 1 ) = Count(w i 2, w i 1, w i ) Count(w i 2, w i 1 ) Bigram maximum-likelihood estimate q ML (w i w i 1 ) = Count(w i 1, w i ) Count(w i 1 ) Unigram maximum-likelihood estimate q ML (w i ) = Count(w i) Count()

40 Linear Interpolation Take our estimate q(w i w i 2, w i 1 ) to be q(w i w i 2, w i 1 ) = λ 1 q ML (w i w i 2, w i 1 ) +λ 2 q ML (w i w i 1 ) +λ 3 q ML (w i ) where λ 1 + λ 2 + λ 3 = 1, and λ i 0 for all i.

41 Linear Interpolation (continued) Our estimate correctly defines a distribution (define V = V {STOP}): w V q(w u, v)

42 Linear Interpolation (continued) Our estimate correctly defines a distribution (define V = V {STOP}): w V q(w u, v) = w V [λ 1 q ML (w u, v) + λ 2 q ML (w v) + λ 3 q ML (w)]

43 Linear Interpolation (continued) Our estimate correctly defines a distribution (define V = V {STOP}): w V q(w u, v) = w V [λ 1 q ML (w u, v) + λ 2 q ML (w v) + λ 3 q ML (w)] = λ 1 w q ML (w u, v) + λ 2 w q ML (w v) + λ 3 w q ML (w)

44 Linear Interpolation (continued) Our estimate correctly defines a distribution (define V = V {STOP}): w V q(w u, v) = w V [λ 1 q ML (w u, v) + λ 2 q ML (w v) + λ 3 q ML (w)] = λ 1 w q ML (w u, v) + λ 2 w q ML (w v) + λ 3 w q ML (w) = λ 1 + λ 2 + λ 3

45 Linear Interpolation (continued) Our estimate correctly defines a distribution (define V = V {STOP}): w V q(w u, v) = w V [λ 1 q ML (w u, v) + λ 2 q ML (w v) + λ 3 q ML (w)] = λ 1 w q ML (w u, v) + λ 2 w q ML (w v) + λ 3 w q ML (w) = λ 1 + λ 2 + λ 3 = 1

46 Linear Interpolation (continued) Our estimate correctly defines a distribution (define V = V {STOP}): w V q(w u, v) = w V [λ 1 q ML (w u, v) + λ 2 q ML (w v) + λ 3 q ML (w)] = λ 1 w q ML (w u, v) + λ 2 w q ML (w v) + λ 3 w q ML (w) = λ 1 + λ 2 + λ 3 = 1 (Can show also that q(w u, v) 0 for all w V )

47 How to estimate the λ values? Hold out part of training set as validation data

48 How to estimate the λ values? Hold out part of training set as validation data Define c (w 1, w 2, w 3 ) to be the number of times the trigram (w 1, w 2, w 3 ) is seen in validation set

49 How to estimate the λ values? Hold out part of training set as validation data Define c (w 1, w 2, w 3 ) to be the number of times the trigram (w 1, w 2, w 3 ) is seen in validation set Choose λ 1, λ 2, λ 3 to maximize: L(λ 1, λ 2, λ 3 ) = c (w 1, w 2, w 3 ) log q(w 3 w 1, w 2 ) w 1,w 2,w 3 such that λ 1 + λ 2 + λ 3 = 1, and λ i 0 for all i, and where q(w i w i 2, w i 1 ) = λ 1 q ML (w i w i 2, w i 1 ) +λ 2 q ML (w i w i 1 ) +λ 3 q ML (w i )

50 Allowing the λ s to vary Take a function Π that partitions histories e.g., 1 If Count(w i 1, w i 2 ) = 0 2 If 1 Count(w Π(w i 2, w i 1 ) = i 1, w i 2 ) 2 3 If 3 Count(w i 1, w i 2 ) 5 4 Otherwise Introduce a dependence of the λ s on the partition: q(w i w i 2, w i 1 ) = λ Π(w i 2,w i 1 ) 1 q ML (w i w i 2, w i 1 ) +λ Π(w i 2,w i 1 ) 2 q ML (w i w i 1 ) +λ Π(w i 2,w i 1 ) 3 q ML (w i ) where λ Π(w i 2,w i 1 ) 1 + λ Π(w i 2,w i 1 ) 2 + λ Π(w i 2,w i 1 ) 3 = 1, and λ Π(w i 2,w i 1 ) i 0 for all i.

51 Overview The language modeling problem Trigram models Evaluating language models: perplexity Estimation techniques: Linear interpolation Discounting methods

52 Discounting Methods Say we ve seen the following counts: x Count(x) q ML (w i w i 1 ) the 48 the, dog 15 15/48 the, woman 11 11/48 the, man 10 10/48 the, park 5 5/48 the, job 2 2/48 the, telescope 1 1/48 the, manual 1 1/48 the, afternoon 1 1/48 the, country 1 1/48 the, street 1 1/48 The maximum-likelihood estimates are high (particularly for low count items)

53 Discounting Methods Now define discounted counts, Count (x) = Count(x) 0.5 New estimates: x Count(x) Count (x) the 48 Count (x) Count(the) the, dog /48 the, woman /48 the, man /48 the, park /48 the, job /48 the, telescope /48 the, manual /48 the, afternoon /48 the, country /48 the, street /48

54 Discounting Methods (Continued) We now have some missing probability mass : α(w i 1 ) = 1 w Count (w i 1, w) Count(w i 1 ) e.g., in our example, α(the) = /48 = 5/48

55 Katz Back-Off Models (Bigrams) For a bigram model, define two sets A(w i 1 ) = {w : Count(w i 1, w) > 0} B(w i 1 ) = {w : Count(w i 1, w) = 0} A bigram model Count (w i 1,w i ) If w i A(w i 1 ) Count(w i 1 ) q BO (w i w i 1 ) = α(w i 1 ) q ML (w i) If w w B(w i 1 ) q ML (w) i B(w i 1 ) where α(w i 1 ) = 1 w A(w i 1 ) Count (w i 1, w) Count(w i 1 )

56 Katz Back-Off Models (Trigrams) For a trigram model, first define two sets A(w i 2, w i 1 ) = {w : Count(w i 2, w i 1, w) > 0} B(w i 2, w i 1 ) = {w : Count(w i 2, w i 1, w) = 0} A trigram model is defined in terms of the bigram model: Count (w i 2,w i 1,w i ) Count(w i 2,w i 1 ) If w i A(w i 2, w i 1 ) q BO (w i w i 2, w i 1 ) = α(w i 2,w i 1 )q BO (w i w i 1 ) w B(w i 2,w i 1 ) q BO(w w i 1 ) If w i B(w i 2, w i 1 ) where α(w i 2, w i 1 ) = 1 w A(w i 2,w i 1 ) Count (w i 2, w i 1, w) Count(w i 2, w i 1 )

57 Summary Three steps in deriving the language model probabilities: 1. Expand p(w 1, w 2... w n ) using Chain rule. 2. Make Markov Independence Assumptions p(w i w 1, w 2... w i 2, w i 1 ) = p(w i w i 2, w i 1 ) 3. Smooth the estimates using low order counts. Other methods used to improve language models: Topic or long-range features. Syntactic models. It s generally hard to improve on trigram models though!!

The Language Modeling Problem (Fall 2007) Smoothed Estimation, and Language Modeling. The Language Modeling Problem (Continued) Overview

The Language Modeling Problem (Fall 2007) Smoothed Estimation, and Language Modeling. The Language Modeling Problem (Continued) Overview The Language Modeling Problem We have some (finite) vocabulary, say V = {the, a, man, telescope, Beckham, two, } 6.864 (Fall 2007) Smoothed Estimation, and Language Modeling We have an (infinite) set of

More information

Probabilistic Language Modeling

Probabilistic Language Modeling Predicting String Probabilities Probabilistic Language Modeling Which string is more likely? (Which string is more grammatical?) Grill doctoral candidates. Regina Barzilay EECS Department MIT November

More information

Natural Language Processing

Natural Language Processing Natural Language Processing Language models Based on slides from Michael Collins, Chris Manning and Richard Soccer Plan Problem definition Trigram models Evaluation Estimation Interpolation Discounting

More information

DT2118 Speech and Speaker Recognition

DT2118 Speech and Speaker Recognition DT2118 Speech and Speaker Recognition Language Modelling Giampiero Salvi KTH/CSC/TMH giampi@kth.se VT 2015 1 / 56 Outline Introduction Formal Language Theory Stochastic Language Models (SLM) N-gram Language

More information

CMPT-825 Natural Language Processing

CMPT-825 Natural Language Processing CMPT-825 Natural Language Processing Anoop Sarkar http://www.cs.sfu.ca/ anoop February 27, 2008 1 / 30 Cross-Entropy and Perplexity Smoothing n-gram Models Add-one Smoothing Additive Smoothing Good-Turing

More information

N-grams. Motivation. Simple n-grams. Smoothing. Backoff. N-grams L545. Dept. of Linguistics, Indiana University Spring / 24

N-grams. Motivation. Simple n-grams. Smoothing. Backoff. N-grams L545. Dept. of Linguistics, Indiana University Spring / 24 L545 Dept. of Linguistics, Indiana University Spring 2013 1 / 24 Morphosyntax We just finished talking about morphology (cf. words) And pretty soon we re going to discuss syntax (cf. sentences) In between,

More information

N-gram Language Modeling

N-gram Language Modeling N-gram Language Modeling Outline: Statistical Language Model (LM) Intro General N-gram models Basic (non-parametric) n-grams Class LMs Mixtures Part I: Statistical Language Model (LM) Intro What is a statistical

More information

Foundations of Natural Language Processing Lecture 5 More smoothing and the Noisy Channel Model

Foundations of Natural Language Processing Lecture 5 More smoothing and the Noisy Channel Model Foundations of Natural Language Processing Lecture 5 More smoothing and the Noisy Channel Model Alex Lascarides (Slides based on those from Alex Lascarides, Sharon Goldwater and Philipop Koehn) 30 January

More information

Natural Language Processing SoSe Words and Language Model

Natural Language Processing SoSe Words and Language Model Natural Language Processing SoSe 2016 Words and Language Model Dr. Mariana Neves May 2nd, 2016 Outline 2 Words Language Model Outline 3 Words Language Model Tokenization Separation of words in a sentence

More information

Language Modeling. Introduction to N-grams. Many Slides are adapted from slides by Dan Jurafsky

Language Modeling. Introduction to N-grams. Many Slides are adapted from slides by Dan Jurafsky Language Modeling Introduction to N-grams Many Slides are adapted from slides by Dan Jurafsky Probabilistic Language Models Today s goal: assign a probability to a sentence Why? Machine Translation: P(high

More information

Quiz 1, COMS Name: Good luck! 4705 Quiz 1 page 1 of 7

Quiz 1, COMS Name: Good luck! 4705 Quiz 1 page 1 of 7 Quiz 1, COMS 4705 Name: 10 30 30 20 Good luck! 4705 Quiz 1 page 1 of 7 Part #1 (10 points) Question 1 (10 points) We define a PCFG where non-terminal symbols are {S,, B}, the terminal symbols are {a, b},

More information

Natural Language Processing. Statistical Inference: n-grams

Natural Language Processing. Statistical Inference: n-grams Natural Language Processing Statistical Inference: n-grams Updated 3/2009 Statistical Inference Statistical Inference consists of taking some data (generated in accordance with some unknown probability

More information

Natural Language Processing SoSe Language Modelling. (based on the slides of Dr. Saeedeh Momtazi)

Natural Language Processing SoSe Language Modelling. (based on the slides of Dr. Saeedeh Momtazi) Natural Language Processing SoSe 2015 Language Modelling Dr. Mariana Neves April 20th, 2015 (based on the slides of Dr. Saeedeh Momtazi) Outline 2 Motivation Estimation Evaluation Smoothing Outline 3 Motivation

More information

Language Modeling. Introduction to N-grams. Many Slides are adapted from slides by Dan Jurafsky

Language Modeling. Introduction to N-grams. Many Slides are adapted from slides by Dan Jurafsky Language Modeling Introduction to N-grams Many Slides are adapted from slides by Dan Jurafsky Probabilistic Language Models Today s goal: assign a probability to a sentence Why? Machine Translation: P(high

More information

We have a sitting situation

We have a sitting situation We have a sitting situation 447 enrollment: 67 out of 64 547 enrollment: 10 out of 10 2 special approved cases for audits ---------------------------------------- 67 + 10 + 2 = 79 students in the class!

More information

Empirical Methods in Natural Language Processing Lecture 10a More smoothing and the Noisy Channel Model

Empirical Methods in Natural Language Processing Lecture 10a More smoothing and the Noisy Channel Model Empirical Methods in Natural Language Processing Lecture 10a More smoothing and the Noisy Channel Model (most slides from Sharon Goldwater; some adapted from Philipp Koehn) 5 October 2016 Nathan Schneider

More information

Language Models. Philipp Koehn. 11 September 2018

Language Models. Philipp Koehn. 11 September 2018 Language Models Philipp Koehn 11 September 2018 Language models 1 Language models answer the question: How likely is a string of English words good English? Help with reordering p LM (the house is small)

More information

Language Model. Introduction to N-grams

Language Model. Introduction to N-grams Language Model Introduction to N-grams Probabilistic Language Model Goal: assign a probability to a sentence Application: Machine Translation P(high winds tonight) > P(large winds tonight) Spelling Correction

More information

N-gram Language Modeling Tutorial

N-gram Language Modeling Tutorial N-gram Language Modeling Tutorial Dustin Hillard and Sarah Petersen Lecture notes courtesy of Prof. Mari Ostendorf Outline: Statistical Language Model (LM) Basics n-gram models Class LMs Cache LMs Mixtures

More information

perplexity = 2 cross-entropy (5.1) cross-entropy = 1 N log 2 likelihood (5.2) likelihood = P(w 1 w N ) (5.3)

perplexity = 2 cross-entropy (5.1) cross-entropy = 1 N log 2 likelihood (5.2) likelihood = P(w 1 w N ) (5.3) Chapter 5 Language Modeling 5.1 Introduction A language model is simply a model of what strings (of words) are more or less likely to be generated by a speaker of English (or some other language). More

More information

Recap: Language models. Foundations of Natural Language Processing Lecture 4 Language Models: Evaluation and Smoothing. Two types of evaluation in NLP

Recap: Language models. Foundations of Natural Language Processing Lecture 4 Language Models: Evaluation and Smoothing. Two types of evaluation in NLP Recap: Language models Foundations of atural Language Processing Lecture 4 Language Models: Evaluation and Smoothing Alex Lascarides (Slides based on those from Alex Lascarides, Sharon Goldwater and Philipp

More information

Language Modelling: Smoothing and Model Complexity. COMP-599 Sept 14, 2016

Language Modelling: Smoothing and Model Complexity. COMP-599 Sept 14, 2016 Language Modelling: Smoothing and Model Complexity COMP-599 Sept 14, 2016 Announcements A1 has been released Due on Wednesday, September 28th Start code for Question 4: Includes some of the package import

More information

Sequences and Information

Sequences and Information Sequences and Information Rahul Siddharthan The Institute of Mathematical Sciences, Chennai, India http://www.imsc.res.in/ rsidd/ Facets 16, 04/07/2016 This box says something By looking at the symbols

More information

The Noisy Channel Model and Markov Models

The Noisy Channel Model and Markov Models 1/24 The Noisy Channel Model and Markov Models Mark Johnson September 3, 2014 2/24 The big ideas The story so far: machine learning classifiers learn a function that maps a data item X to a label Y handle

More information

Speech Recognition Lecture 5: N-gram Language Models. Eugene Weinstein Google, NYU Courant Institute Slide Credit: Mehryar Mohri

Speech Recognition Lecture 5: N-gram Language Models. Eugene Weinstein Google, NYU Courant Institute Slide Credit: Mehryar Mohri Speech Recognition Lecture 5: N-gram Language Models Eugene Weinstein Google, NYU Courant Institute eugenew@cs.nyu.edu Slide Credit: Mehryar Mohri Components Acoustic and pronunciation model: Pr(o w) =

More information

Deep Learning Basics Lecture 10: Neural Language Models. Princeton University COS 495 Instructor: Yingyu Liang

Deep Learning Basics Lecture 10: Neural Language Models. Princeton University COS 495 Instructor: Yingyu Liang Deep Learning Basics Lecture 10: Neural Language Models Princeton University COS 495 Instructor: Yingyu Liang Natural language Processing (NLP) The processing of the human languages by computers One of

More information

(today we are assuming sentence segmentation) Wednesday, September 10, 14

(today we are assuming sentence segmentation) Wednesday, September 10, 14 (today we are assuming sentence segmentation) 1 Your TA: David Belanger http://people.cs.umass.edu/~belanger/ I am a third year PhD student advised by Professor Andrew McCallum. Before that, I was an Associate

More information

Probabilistic Context-Free Grammars. Michael Collins, Columbia University

Probabilistic Context-Free Grammars. Michael Collins, Columbia University Probabilistic Context-Free Grammars Michael Collins, Columbia University Overview Probabilistic Context-Free Grammars (PCFGs) The CKY Algorithm for parsing with PCFGs A Probabilistic Context-Free Grammar

More information

ACS Introduction to NLP Lecture 3: Language Modelling and Smoothing

ACS Introduction to NLP Lecture 3: Language Modelling and Smoothing ACS Introduction to NLP Lecture 3: Language Modelling and Smoothing Stephen Clark Natural Language and Information Processing (NLIP) Group sc609@cam.ac.uk Language Modelling 2 A language model is a probability

More information

Statistical Methods for NLP

Statistical Methods for NLP Statistical Methods for NLP Language Models, Graphical Models Sameer Maskey Week 13, April 13, 2010 Some slides provided by Stanley Chen and from Bishop Book Resources 1 Announcements Final Project Due,

More information

Machine Learning for natural language processing

Machine Learning for natural language processing Machine Learning for natural language processing N-grams and language models Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Summer 2016 1 / 25 Introduction Goals: Estimate the probability that a

More information

ANLP Lecture 6 N-gram models and smoothing

ANLP Lecture 6 N-gram models and smoothing ANLP Lecture 6 N-gram models and smoothing Sharon Goldwater (some slides from Philipp Koehn) 27 September 2018 Sharon Goldwater ANLP Lecture 6 27 September 2018 Recap: N-gram models We can model sentence

More information

CSEP 517 Natural Language Processing Autumn 2013

CSEP 517 Natural Language Processing Autumn 2013 CSEP 517 Natural Language Processing Autumn 2013 Language Models Luke Zettlemoyer Many slides from Dan Klein and Michael Collins Overview The language modeling problem N-gram language models Evaluation:

More information

Natural Language Processing (CSE 490U): Language Models

Natural Language Processing (CSE 490U): Language Models Natural Language Processing (CSE 490U): Language Models Noah Smith c 2017 University of Washington nasmith@cs.washington.edu January 6 9, 2017 1 / 67 Very Quick Review of Probability Event space (e.g.,

More information

Language Models. CS5740: Natural Language Processing Spring Instructor: Yoav Artzi

Language Models. CS5740: Natural Language Processing Spring Instructor: Yoav Artzi CS5740: Natural Language Processing Spring 2017 Language Models Instructor: Yoav Artzi Slides adapted from Dan Klein, Dan Jurafsky, Chris Manning, Michael Collins, Luke Zettlemoyer, and Yejin Choi Overview

More information

NLP: N-Grams. Dan Garrette December 27, Predictive text (text messaging clients, search engines, etc)

NLP: N-Grams. Dan Garrette December 27, Predictive text (text messaging clients, search engines, etc) NLP: N-Grams Dan Garrette dhg@cs.utexas.edu December 27, 2013 1 Language Modeling Tasks Language idenfication / Authorship identification Machine Translation Speech recognition Optical character recognition

More information

CSA4050: Advanced Topics Natural Language Processing. Lecture Statistics III. Statistical Approaches to NLP

CSA4050: Advanced Topics Natural Language Processing. Lecture Statistics III. Statistical Approaches to NLP University of Malta BSc IT (Hons)Year IV CSA4050: Advanced Topics Natural Language Processing Lecture Statistics III Statistical Approaches to NLP Witten-Bell Discounting Unigrams Bigrams Dept Computer

More information

Language Models. CS6200: Information Retrieval. Slides by: Jesse Anderton

Language Models. CS6200: Information Retrieval. Slides by: Jesse Anderton Language Models CS6200: Information Retrieval Slides by: Jesse Anderton What s wrong with VSMs? Vector Space Models work reasonably well, but have a few problems: They are based on bag-of-words, so they

More information

Language Models. Data Science: Jordan Boyd-Graber University of Maryland SLIDES ADAPTED FROM PHILIP KOEHN

Language Models. Data Science: Jordan Boyd-Graber University of Maryland SLIDES ADAPTED FROM PHILIP KOEHN Language Models Data Science: Jordan Boyd-Graber University of Maryland SLIDES ADAPTED FROM PHILIP KOEHN Data Science: Jordan Boyd-Graber UMD Language Models 1 / 8 Language models Language models answer

More information

Log-Linear Models, MEMMs, and CRFs

Log-Linear Models, MEMMs, and CRFs Log-Linear Models, MEMMs, and CRFs Michael Collins 1 Notation Throughout this note I ll use underline to denote vectors. For example, w R d will be a vector with components w 1, w 2,... w d. We use expx

More information

Probabilistic Context Free Grammars. Many slides from Michael Collins

Probabilistic Context Free Grammars. Many slides from Michael Collins Probabilistic Context Free Grammars Many slides from Michael Collins Overview I Probabilistic Context-Free Grammars (PCFGs) I The CKY Algorithm for parsing with PCFGs A Probabilistic Context-Free Grammar

More information

Chapter 3: Basics of Language Modelling

Chapter 3: Basics of Language Modelling Chapter 3: Basics of Language Modelling Motivation Language Models are used in Speech Recognition Machine Translation Natural Language Generation Query completion For research and development: need a simple

More information

language modeling: n-gram models

language modeling: n-gram models language modeling: n-gram models CS 585, Fall 2018 Introduction to Natural Language Processing http://people.cs.umass.edu/~miyyer/cs585/ Mohit Iyyer College of Information and Computer Sciences University

More information

Statistical Methods for NLP

Statistical Methods for NLP Statistical Methods for NLP Sequence Models Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Statistical Methods for NLP 1(21) Introduction Structured

More information

CS 6120/CS4120: Natural Language Processing

CS 6120/CS4120: Natural Language Processing CS 6120/CS4120: Natural Language Processing Instructor: Prof. Lu Wang College of Computer and Information Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Outline Probabilistic language

More information

Language Technology. Unit 1: Sequence Models. CUNY Graduate Center Spring Lectures 5-6: Language Models and Smoothing. required hard optional

Language Technology. Unit 1: Sequence Models. CUNY Graduate Center Spring Lectures 5-6: Language Models and Smoothing. required hard optional Language Technology CUNY Graduate Center Spring 2013 Unit 1: Sequence Models Lectures 5-6: Language Models and Smoothing required hard optional Professor Liang Huang liang.huang.sh@gmail.com Python Review:

More information

Language Processing with Perl and Prolog

Language Processing with Perl and Prolog Language Processing with Perl and Prolog Chapter 5: Counting Words Pierre Nugues Lund University Pierre.Nugues@cs.lth.se http://cs.lth.se/pierre_nugues/ Pierre Nugues Language Processing with Perl and

More information

Features of Statistical Parsers

Features of Statistical Parsers Features of tatistical Parsers Preliminary results Mark Johnson Brown University TTI, October 2003 Joint work with Michael Collins (MIT) upported by NF grants LI 9720368 and II0095940 1 Talk outline tatistical

More information

Algorithms for NLP. Language Modeling III. Taylor Berg-Kirkpatrick CMU Slides: Dan Klein UC Berkeley

Algorithms for NLP. Language Modeling III. Taylor Berg-Kirkpatrick CMU Slides: Dan Klein UC Berkeley Algorithms for NLP Language Modeling III Taylor Berg-Kirkpatrick CMU Slides: Dan Klein UC Berkeley Announcements Office hours on website but no OH for Taylor until next week. Efficient Hashing Closed address

More information

CSC321 Lecture 7 Neural language models

CSC321 Lecture 7 Neural language models CSC321 Lecture 7 Neural language models Roger Grosse and Nitish Srivastava February 1, 2015 Roger Grosse and Nitish Srivastava CSC321 Lecture 7 Neural language models February 1, 2015 1 / 19 Overview We

More information

Feedforward Neural Networks. Michael Collins, Columbia University

Feedforward Neural Networks. Michael Collins, Columbia University Feedforward Neural Networks Michael Collins, Columbia University Recap: Log-linear Models A log-linear model takes the following form: p(y x; v) = exp (v f(x, y)) y Y exp (v f(x, y )) f(x, y) is the representation

More information

Statistical Natural Language Processing

Statistical Natural Language Processing Statistical Natural Language Processing N-gram Language Models Çağrı Çöltekin University of Tübingen Seminar für Sprachwissenschaft Summer Semester 2017 N-gram language models A language model answers

More information

Hidden Markov Models in Language Processing

Hidden Markov Models in Language Processing Hidden Markov Models in Language Processing Dustin Hillard Lecture notes courtesy of Prof. Mari Ostendorf Outline Review of Markov models What is an HMM? Examples General idea of hidden variables: implications

More information

TnT Part of Speech Tagger

TnT Part of Speech Tagger TnT Part of Speech Tagger By Thorsten Brants Presented By Arghya Roy Chaudhuri Kevin Patel Satyam July 29, 2014 1 / 31 Outline 1 Why Then? Why Now? 2 Underlying Model Other technicalities 3 Evaluation

More information

Week 13: Language Modeling II Smoothing in Language Modeling. Irina Sergienya

Week 13: Language Modeling II Smoothing in Language Modeling. Irina Sergienya Week 13: Language Modeling II Smoothing in Language Modeling Irina Sergienya 07.07.2015 Couple of words first... There are much more smoothing techniques, [e.g. Katz back-off, Jelinek-Mercer,...] and techniques

More information

Lecture 4: Smoothing, Part-of-Speech Tagging. Ivan Titov Institute for Logic, Language and Computation Universiteit van Amsterdam

Lecture 4: Smoothing, Part-of-Speech Tagging. Ivan Titov Institute for Logic, Language and Computation Universiteit van Amsterdam Lecture 4: Smoothing, Part-of-Speech Tagging Ivan Titov Institute for Logic, Language and Computation Universiteit van Amsterdam Language Models from Corpora We want a model of sentence probability P(w

More information

Language Modeling. Introduc*on to N- grams. Many Slides are adapted from slides by Dan Jurafsky

Language Modeling. Introduc*on to N- grams. Many Slides are adapted from slides by Dan Jurafsky Language Modeling Introduc*on to N- grams Many Slides are adapted from slides by Dan Jurafsky Probabilis1c Language Models Today s goal: assign a probability to a sentence Machine Transla*on: Why? P(high

More information

Empirical Methods in Natural Language Processing Lecture 5 N-gram Language Models

Empirical Methods in Natural Language Processing Lecture 5 N-gram Language Models Empirical Methods in Natural Language Processing Lecture 5 N-gram Language Models (most slides from Sharon Goldwater; some adapted from Alex Lascarides) 29 January 2017 Nathan Schneider ENLP Lecture 5

More information

Graphical Models. Mark Gales. Lent Machine Learning for Language Processing: Lecture 3. MPhil in Advanced Computer Science

Graphical Models. Mark Gales. Lent Machine Learning for Language Processing: Lecture 3. MPhil in Advanced Computer Science Graphical Models Mark Gales Lent 2011 Machine Learning for Language Processing: Lecture 3 MPhil in Advanced Computer Science MPhil in Advanced Computer Science Graphical Models Graphical models have their

More information

ACS Introduction to NLP Lecture 2: Part of Speech (POS) Tagging

ACS Introduction to NLP Lecture 2: Part of Speech (POS) Tagging ACS Introduction to NLP Lecture 2: Part of Speech (POS) Tagging Stephen Clark Natural Language and Information Processing (NLIP) Group sc609@cam.ac.uk The POS Tagging Problem 2 England NNP s POS fencers

More information

Natural Language Processing

Natural Language Processing SFU NatLangLab Natural Language Processing Anoop Sarkar anoopsarkar.github.io/nlp-class Simon Fraser University October 20, 2017 0 Natural Language Processing Anoop Sarkar anoopsarkar.github.io/nlp-class

More information

{ Jurafsky & Martin Ch. 6:! 6.6 incl.

{ Jurafsky & Martin Ch. 6:! 6.6 incl. N-grams Now Simple (Unsmoothed) N-grams Smoothing { Add-one Smoothing { Backo { Deleted Interpolation Reading: { Jurafsky & Martin Ch. 6:! 6.6 incl. 1 Word-prediction Applications Augmentative Communication

More information

Conditional Language Modeling. Chris Dyer

Conditional Language Modeling. Chris Dyer Conditional Language Modeling Chris Dyer Unconditional LMs A language model assigns probabilities to sequences of words,. w =(w 1,w 2,...,w`) It is convenient to decompose this probability using the chain

More information

Hidden Markov Models, I. Examples. Steven R. Dunbar. Toy Models. Standard Mathematical Models. Realistic Hidden Markov Models.

Hidden Markov Models, I. Examples. Steven R. Dunbar. Toy Models. Standard Mathematical Models. Realistic Hidden Markov Models. , I. Toy Markov, I. February 17, 2017 1 / 39 Outline, I. Toy Markov 1 Toy 2 3 Markov 2 / 39 , I. Toy Markov A good stack of examples, as large as possible, is indispensable for a thorough understanding

More information

Ranked Retrieval (2)

Ranked Retrieval (2) Text Technologies for Data Science INFR11145 Ranked Retrieval (2) Instructor: Walid Magdy 31-Oct-2017 Lecture Objectives Learn about Probabilistic models BM25 Learn about LM for IR 2 1 Recall: VSM & TFIDF

More information

Ngram Review. CS 136 Lecture 10 Language Modeling. Thanks to Dan Jurafsky for these slides. October13, 2017 Professor Meteer

Ngram Review. CS 136 Lecture 10 Language Modeling. Thanks to Dan Jurafsky for these slides. October13, 2017 Professor Meteer + Ngram Review October13, 2017 Professor Meteer CS 136 Lecture 10 Language Modeling Thanks to Dan Jurafsky for these slides + ASR components n Feature Extraction, MFCCs, start of Acoustic n HMMs, the Forward

More information

Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Language Models. Tobias Scheffer

Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen. Language Models. Tobias Scheffer Universität Potsdam Institut für Informatik Lehrstuhl Maschinelles Lernen Language Models Tobias Scheffer Stochastic Language Models A stochastic language model is a probability distribution over words.

More information

Language Modeling. Introduction to N-grams. Klinton Bicknell. (borrowing from: Dan Jurafsky and Jim Martin)

Language Modeling. Introduction to N-grams. Klinton Bicknell. (borrowing from: Dan Jurafsky and Jim Martin) Language Modeling Introduction to N-grams Klinton Bicknell (borrowing from: Dan Jurafsky and Jim Martin) Probabilistic Language Models Today s goal: assign a probability to a sentence Why? Machine Translation:

More information

Deep Learning. Language Models and Word Embeddings. Christof Monz

Deep Learning. Language Models and Word Embeddings. Christof Monz Deep Learning Today s Class N-gram language modeling Feed-forward neural language model Architecture Final layer computations Word embeddings Continuous bag-of-words model Skip-gram Negative sampling 1

More information

FINITE STATE LANGUAGE MODELS SMOOTHED USING n-grams

FINITE STATE LANGUAGE MODELS SMOOTHED USING n-grams International Journal of Pattern Recognition and Artificial Intelligence Vol. 16, No. 3 (2002) 275 289 c World Scientific Publishing Company FINITE STATE LANGUAGE MODELS SMOOTHED USING n-grams DAVID LLORENS

More information

Cross-Lingual Language Modeling for Automatic Speech Recogntion

Cross-Lingual Language Modeling for Automatic Speech Recogntion GBO Presentation Cross-Lingual Language Modeling for Automatic Speech Recogntion November 14, 2003 Woosung Kim woosung@cs.jhu.edu Center for Language and Speech Processing Dept. of Computer Science The

More information

Midterm sample questions

Midterm sample questions Midterm sample questions CS 585, Brendan O Connor and David Belanger October 12, 2014 1 Topics on the midterm Language concepts Translation issues: word order, multiword translations Human evaluation Parts

More information

Chapter 3: Basics of Language Modeling

Chapter 3: Basics of Language Modeling Chapter 3: Basics of Language Modeling Section 3.1. Language Modeling in Automatic Speech Recognition (ASR) All graphs in this section are from the book by Schukat-Talamazzini unless indicated otherwise

More information

CSCI 5832 Natural Language Processing. Today 1/31. Probability Basics. Lecture 6. Probability. Language Modeling (N-grams)

CSCI 5832 Natural Language Processing. Today 1/31. Probability Basics. Lecture 6. Probability. Language Modeling (N-grams) CSCI 5832 Natural Language Processing Jim Martin Lecture 6 1 Today 1/31 Probability Basic probability Conditional probability Bayes Rule Language Modeling (N-grams) N-gram Intro The Chain Rule Smoothing:

More information

Lecture 12: Algorithms for HMMs

Lecture 12: Algorithms for HMMs Lecture 12: Algorithms for HMMs Nathan Schneider (some slides from Sharon Goldwater; thanks to Jonathan May for bug fixes) ENLP 26 February 2018 Recap: tagging POS tagging is a sequence labelling task.

More information

Info 2950, Lecture 24

Info 2950, Lecture 24 Info 2950, Lecture 24 2 May 2017 An additional "bonus" was added to problem 4 on Fri afternoon 28 Apr, and some additional typos plus slight rewording to resolve questions about problem 2 at end of afternoon.

More information

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing. Language Models & Hidden Markov Models

INF4820: Algorithms for Artificial Intelligence and Natural Language Processing. Language Models & Hidden Markov Models 1 University of Oslo : Department of Informatics INF4820: Algorithms for Artificial Intelligence and Natural Language Processing Language Models & Hidden Markov Models Stephan Oepen & Erik Velldal Language

More information

Statistical Machine Translation

Statistical Machine Translation Statistical Machine Translation Marcello Federico FBK-irst Trento, Italy Galileo Galilei PhD School University of Pisa Pisa, 7-19 May 2008 Part V: Language Modeling 1 Comparing ASR and statistical MT N-gram

More information

Natural Language Processing

Natural Language Processing Natural Language Processing Info 159/259 Lecture 7: Language models 2 (Sept 14, 2017) David Bamman, UC Berkeley Language Model Vocabulary V is a finite set of discrete symbols (e.g., words, characters);

More information

Advanced topics in language modeling: MaxEnt, features and marginal distritubion constraints

Advanced topics in language modeling: MaxEnt, features and marginal distritubion constraints Advanced topics in language modeling: MaxEnt, features and marginal distritubion constraints Brian Roark, Google NYU CSCI-GA.2585 Speech Recognition, Weds., April 15, 2015 (Don t forget to pay your taxes!)

More information

Lecture 12: Algorithms for HMMs

Lecture 12: Algorithms for HMMs Lecture 12: Algorithms for HMMs Nathan Schneider (some slides from Sharon Goldwater; thanks to Jonathan May for bug fixes) ENLP 17 October 2016 updated 9 September 2017 Recap: tagging POS tagging is a

More information

Learning Objectives. c D. Poole and A. Mackworth 2010 Artificial Intelligence, Lecture 7.2, Page 1

Learning Objectives. c D. Poole and A. Mackworth 2010 Artificial Intelligence, Lecture 7.2, Page 1 Learning Objectives At the end of the class you should be able to: identify a supervised learning problem characterize how the prediction is a function of the error measure avoid mixing the training and

More information

Log-linear models (part 1)

Log-linear models (part 1) Log-linear models (part 1) CS 690N, Spring 2018 Advanced Natural Language Processing http://people.cs.umass.edu/~brenocon/anlp2018/ Brendan O Connor College of Information and Computer Sciences University

More information

Lecture 2: N-gram. Kai-Wei Chang University of Virginia Couse webpage:

Lecture 2: N-gram. Kai-Wei Chang University of Virginia Couse webpage: Lecture 2: N-gram Kai-Wei Chang CS @ University of Virginia kw@kwchang.net Couse webpage: http://kwchang.net/teaching/nlp16 CS 6501: Natural Language Processing 1 This lecture Language Models What are

More information

Introduction to N-grams

Introduction to N-grams Lecture Notes, Artificial Intelligence Course, University of Birzeit, Palestine Spring Semester, 2014 (Advanced/) Artificial Intelligence Probabilistic Language Modeling Introduction to N-grams Dr. Mustafa

More information

Classification & Information Theory Lecture #8

Classification & Information Theory Lecture #8 Classification & Information Theory Lecture #8 Introduction to Natural Language Processing CMPSCI 585, Fall 2007 University of Massachusetts Amherst Andrew McCallum Today s Main Points Automatically categorizing

More information

P(t w) = arg maxp(t, w) (5.1) P(t,w) = P(t)P(w t). (5.2) The first term, P(t), can be described using a language model, for example, a bigram model:

P(t w) = arg maxp(t, w) (5.1) P(t,w) = P(t)P(w t). (5.2) The first term, P(t), can be described using a language model, for example, a bigram model: Chapter 5 Text Input 5.1 Problem In the last two chapters we looked at language models, and in your first homework you are building language models for English and Chinese to enable the computer to guess

More information

CS4705. Probability Review and Naïve Bayes. Slides from Dragomir Radev

CS4705. Probability Review and Naïve Bayes. Slides from Dragomir Radev CS4705 Probability Review and Naïve Bayes Slides from Dragomir Radev Classification using a Generative Approach Previously on NLP discriminative models P C D here is a line with all the social media posts

More information

To make a grammar probabilistic, we need to assign a probability to each context-free rewrite

To make a grammar probabilistic, we need to assign a probability to each context-free rewrite Notes on the Inside-Outside Algorithm To make a grammar probabilistic, we need to assign a probability to each context-free rewrite rule. But how should these probabilities be chosen? It is natural to

More information

CSCI 5822 Probabilistic Model of Human and Machine Learning. Mike Mozer University of Colorado

CSCI 5822 Probabilistic Model of Human and Machine Learning. Mike Mozer University of Colorado CSCI 5822 Probabilistic Model of Human and Machine Learning Mike Mozer University of Colorado Topics Language modeling Hierarchical processes Pitman-Yor processes Based on work of Teh (2006), A hierarchical

More information

CS 630 Basic Probability and Information Theory. Tim Campbell

CS 630 Basic Probability and Information Theory. Tim Campbell CS 630 Basic Probability and Information Theory Tim Campbell 21 January 2003 Probability Theory Probability Theory is the study of how best to predict outcomes of events. An experiment (or trial or event)

More information

Graphical models for part of speech tagging

Graphical models for part of speech tagging Indian Institute of Technology, Bombay and Research Division, India Research Lab Graphical models for part of speech tagging Different Models for POS tagging HMM Maximum Entropy Markov Models Conditional

More information

How to Use Probabilities. The Crash Course

How to Use Probabilities. The Crash Course How to Use Probabilities The Crash Course 1 Goals of this lecture Probability notation like p(y X): What does this expression mean? How can I manipulate it? How can I estimate its value in practice? Probability

More information

Probabilistic Context Free Grammars. Many slides from Michael Collins and Chris Manning

Probabilistic Context Free Grammars. Many slides from Michael Collins and Chris Manning Probabilistic Context Free Grammars Many slides from Michael Collins and Chris Manning Overview I Probabilistic Context-Free Grammars (PCFGs) I The CKY Algorithm for parsing with PCFGs A Probabilistic

More information

Grammars and introduction to machine learning. Computers Playing Jeopardy! Course Stony Brook University

Grammars and introduction to machine learning. Computers Playing Jeopardy! Course Stony Brook University Grammars and introduction to machine learning Computers Playing Jeopardy! Course Stony Brook University Last class: grammars and parsing in Prolog Noun -> roller Verb thrills VP Verb NP S NP VP NP S VP

More information

An Algorithm for Fast Calculation of Back-off N-gram Probabilities with Unigram Rescaling

An Algorithm for Fast Calculation of Back-off N-gram Probabilities with Unigram Rescaling An Algorithm for Fast Calculation of Back-off N-gram Probabilities with Unigram Rescaling Masaharu Kato, Tetsuo Kosaka, Akinori Ito and Shozo Makino Abstract Topic-based stochastic models such as the probabilistic

More information

From perceptrons to word embeddings. Simon Šuster University of Groningen

From perceptrons to word embeddings. Simon Šuster University of Groningen From perceptrons to word embeddings Simon Šuster University of Groningen Outline A basic computational unit Weighting some input to produce an output: classification Perceptron Classify tweets Written

More information

A fast and simple algorithm for training neural probabilistic language models

A fast and simple algorithm for training neural probabilistic language models A fast and simple algorithm for training neural probabilistic language models Andriy Mnih Joint work with Yee Whye Teh Gatsby Computational Neuroscience Unit University College London 25 January 2013 1

More information

CS 6120/CS4120: Natural Language Processing

CS 6120/CS4120: Natural Language Processing CS 6120/CS4120: Natural Language Processing Instructor: Prof. Lu Wang College of Computer and Information Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Today s Outline Probabilistic

More information

Language Modeling. Introduc*on to N- grams. Many Slides are adapted from slides by Dan Jurafsky

Language Modeling. Introduc*on to N- grams. Many Slides are adapted from slides by Dan Jurafsky Language Modeling Introduc*on to N- grams Many Slides are adapted from slides by Dan Jurafsky Probabilis1c Language Models Today s goal: assign a probability to a sentence Machine Transla*on: Why? P(high

More information