Graph-based Dependency Parsing. Ryan McDonald Google Research
|
|
|
- Edwin Simmons
- 6 years ago
- Views:
Transcription
1 Graph-based Dependency Parsing Ryan McDonald Google Research
2 Reader s Digest Graph-based Dependency Parsing Ryan McDonald Google Research ryanmcd@google.com
3 root ROOT Dependency Parsing SBJ will VC NMOD Tomash remain PP Mr as NP Mr Tomash will remain as a director emeritus a NMOD director emeritus NMOD
4 Definitions L = {l 1,l 2,...,l m } X = x 0 x 1...x n Y Arc label set Input sentence Dependency Graph/Tree
5 Definitions root L = {l 1,l 2,...,l m } X = x 0 x 1...x n Y Arc label set Input sentence Dependency Graph/Tree
6 Definitions L = {l 1,l 2,...,l m } X = x 0 x 1...x n Y Arc label set Input sentence Dependency Graph/Tree (i, j, k) Y indicates l k x i x j
7 Graph-based Parsing Factor the weight/score graphs by subgraphs w(y )= τ Y w τ τ is from a set of subgraphs of interest, e.g., arcs, adjacent arcs Product vs. Sum: Y = arg max Y τ Y w τ = arg max Y τ Y log w τ
8 Arc-factored Graph-based Parsing root saw John Mary
9 Arc-factored Graph-based Parsing root 10 9 John saw Mary Learn to weight arcs w(y )= a Y w a
10 Arc-factored Graph-based Parsing root 10 9 John saw Mary Learn to weight arcs w(y )= a Y w a Y = arg max Y a Y w a Inference/Parsing/Argmax
11 Arc-factored Graph-based Parsing root 10 9 John saw Mary Learn to weight arcs w(y )= a Y w a Y = arg max Y a Y w a Inference/Parsing/Argmax root John saw Mary
12 Arc-factored Projective Parsing W[i][j][h] = weight of best tree spanning words i to j rooted at word h k h h h w(a) w(b) w k hh max over k, l, h A B i l l+1 j k i j h A h B i l l+1 j
13 Arc-factored Projective Parsing W[i][j][h] = weight of best tree spanning words i to j rooted at word h Eisner 96 O( L n 5 ) O(n 3 + L n 2 ) k h h h w(a) w(b) w k hh max over k, l, h A B i l l+1 j k i j h A h B i l l+1 j
14 Arc-factored Non-projective Parsing Non-projective Parsing (McDonald et al 05) Inference: O( L n2 ) with Chu-Liu-Edmonds MST alg Greedy-Recursive algorithm root 10 9 John saw Mary Spanning trees Valid dependency graphs
15 Arc-factored Non-projective Parsing Non-projective Parsing (McDonald et al 05) Inference: O( L n2 ) with Chu-Liu-Edmonds MST alg Greedy-Recursive algorithm root 9 We win with non-projective algorithms!... err... John 10 saw Mary Spanning trees Valid dependency graphs
16 Arc-factored Non-projective Parsing Non-projective Parsing (McDonald et al 05) Inference: O( L n2 ) with Chu-Liu-Edmonds MST alg Greedy-Recursive algorithm root 9 We win with non-projective algorithms!... err Spanning trees Greedy/Recursive saw is not what we are used to John Mary Valid dependency graphs
17 Arc-factored models can be powerful But does not model linguistic reality Syntax is not context independent Beyond Arc-factored Models
18 Arc-factored models can be powerful But does not model linguistic reality Syntax is not context independent Beyond Arc-factored Models Arity Arity of a word = # of modifiers in graph Model arity through preference parameters
19 Figure 4: Vertical and Horizontal neighbourhood for Arc-factored models can be powerful But does not model linguistic reality Syntax is not context independent Beyond Arc-factored Models Arity Arity of a word = # of modifiers in graph Model arity through preference parameters Markovization Vertical/Horizontal Adjacent arcs
20 Projective -- Easy W[i][j][h][a] = weight of best tree spanning words i to j rooted at word h with arity a Arity terms k h,a-1 h h,a w(a) w a 1 h w(b) w k hh w a h A B i l l+1 j max over k, l, h k i j h A h,a-1 B i l l+1 j
21 Non-projective -- Hard McDonald and Satta 07 Arity (even just modified/not-modified) is NP-hard Markovization is NP-hard Can basically generalize to any non-local info Generalizes Nehaus and Boker 97 Arc-factored: non-projective easier Beyond arc-factored: non-projective harder
22 Non-projective Solutions In all cases we augment w(y) w(y )= (i,j,k) w k ij β Calculate w(y) using: Approximations (Jason s talk!) Exact ILP methods Chart-parsing Algorithms Re-ranking MCMC Arity/Markovization/etc
23 Annealing Approximations (McDonald & Pereira 06) Start with initial guess Make small changes to increase w(y) w(y )= (i,j,k) w k ij β
24 Annealing Approximations (McDonald & Pereira 06) Start with initial guess Make small changes to increase w(y) w(y )= (i,j,k) w k ij β Initial guess: arg max Y (i,j,k) w k ij Arc Factored
25 Annealing Approximations (McDonald & Pereira 06) Start with initial guess Make small changes to increase w(y) w(y )= (i,j,k) w k ij β Initial guess: arg max Y (i,j,k) w k ij Arc Factored Until convergence Find arc change to maximize Make the change to guess w(y )= (i,j,k) w k ij β
26 Annealing Approximations (McDonald & Pereira 06) Start with initial guess Make small changes to increase w(y) w(y )= (i,j,k) w k ij β Initial guess: arg max Y (i,j,k) w k ij Arc Factored Until convergence Find arc change to maximize Make the change to guess Good in w(y )= practice, wij k β but suffers from (i,j,k) local maxima
27 Integer Linear Programming (ILP) (Riedel and Clarke 06, Kubler et al 09, Martins, Smith and Xing 09) An ILP is an optimization problem with: A linear objective function A set of linear constraints ILPs are NP-hard in worst-case, but well understood w/ fast algorithms in practice Dependency parsing can be cast as an ILP Note: we will work in the log space Y = arg max Y Y (G X ) (i,j,k) log w k ij
28 Arc-factored Dependency Parsing as an ILP (from Kubler, MDonald and Nivre 2009) Define integer variables: a k ij {0, 1} a k ij = 1 iff (i, j, k) Y b ij {0, 1} b ij = 1 iff x i... x j Y
29 Arc-Factored Dependency Parsing as an ILP (from Kubler, McDonald and Nivre 2009) max a i,j,k a k ij log w k ij such that: i,k a k i0 =0 j : i,k a k ij =1 Constrain arc assignments to produce a tree i, j, k : b ij a k ij 0 i, j, k :2b ik b ij b jk 1 i : b ii =0
30 Arc-Factored Dependency Parsing as an ILP (from Kubler, McDonald and Nivre 2009) max a i,j,k a k ij log w k ij such Can add that: non-local constraints & preference parameters Riedel & Clarke 06, Martins et al. 09 i,k a k i0 =0 j : i,k a k ij =1 Constrain arc assignments to produce a tree i, j, k : b ij a k ij 0 i, j, k :2b ik b ij b jk 1 i : b ii =0
31 Dynamic Prog/Chart-based methods
32 Dynamic Prog/Chart-based methods Question: are there efficient non-projective chart parsing algorithms for unrestricted trees? Most likely not: we could just augment them to get tractable non-local non-projective models
33 Dynamic Prog/Chart-based methods Question: are there efficient non-projective chart parsing algorithms for unrestricted trees? Most likely not: we could just augment them to get tractable non-local non-projective models Gomez-Rodriguez et al. 09, Kuhlmann 09 For well-nested dependency trees of gap-degree 1 Kuhlmann & Nivre: Accounts for >> 99% of trees O(n 7 ) deductive/chart-parsing algorithms
34 Dynamic Prog/Chart-based methods Question: are there efficient non-projective chart parsing algorithms for unrestricted trees? Most likely not: we could just augment them to get tractable non-local non-projective models Gomez-Rodriguez et al. 09, Kuhlmann 09 For well-nested dependency trees of gap-degree 1 Kuhlmann & Nivre: Accounts for >> 99% of trees O(n 7 ) deductive/chart-parsing algorithms Chart-parsing == easy to extend beyond arc-factored assumptions
35 What is next? Getting back to grammars? Non-projective unsupervised parsing? Efficiency?
36 Getting Back to Grammars Almost all research has been grammar-less All possible structures permissible Just learn to discriminate good from bad Unlike SOTA phrase-based methods All explicitly use (derived) grammar
37 Getting Back to Grammars
38 Getting Back to Grammars Projective == CF Dependency Grammars Gaifman (65), Eisner & Blatz (07), Johnson (07)
39 Getting Back to Grammars Projective == CF Dependency Grammars Gaifman (65), Eisner & Blatz (07), Johnson (07) Mildly context sensitive dependency grammars Restricted chart parsing for well-nested/gap-degree 1 Bodirsky et al. (05): capture LTAG derivations
40 Getting Back to Grammars Projective == CF Dependency Grammars Both just put constraints on output Annealing algs == repair algs in CDGs Gaifman (65), Eisner & Blatz (07), Johnson (07) Mildly context sensitive dependency grammars Restricted chart parsing for well-nested/gap-degree 1 Bodirsky et al. (05): capture LTAG derivations ILP == Constraint Dependency Grammars (Maruyama 1990) CDG constraints can be added to ILP (hard/soft)
41 Getting Back to Grammars Projective == CF Dependency Grammars Questions and parsing speeds? Both just put constraints on output Annealing algs == repair algs in CDGs Gaifman (65), Eisner & Blatz (07), Johnson (07) Mildly context sensitive dependency grammars Restricted chart parsing for well-nested/gap-degree 1 1. Bodirsky Can we et flush al. (05): out capture the connections LTAG derivations further? 2. Can we use grammars to improve accuracy ILP == Constraint Dependency Grammars (Maruyama 1990) CDG constraints can be added to ILP (hard/soft)
42 Non-projective Unsupervised Parsing McDonald and Satta 07 Not true w/ valence All projective Dependency model w/o valence (arity) is tractable Klein & Manning 04, Smith 06, Headden et al. 09 Valence++ required for good performance
43 Non-projective Unsupervised Parsing McDonald and Satta 07 Not true w/ valence All projective Dependency model w/o valence (arity) is tractable Klein & Manning 04, Smith 06, Headden et al. 09 Valence++ required for good performance Non-projective Unsupervised Systems?
44 Swedish O(nL) Efficiency / Resources O(n 3 + nl) O(n 3 L 2 ) O(n 3 L 2 ) O(n 2 kl 2 ) Malt Joint MST pipeline MST joint MST Joint Feat Hash MST Joint Feat Hash Coarse to Fine LAS Parse time ~ ~ Model size - 88 Mb 200 Mb 11 Mb 15 Mb # features - 16 M 30 M 30 M 30 M Pretty good, but still not there! -- A*?, More pruning?
45 Summary
46 Where we ve been Arc-factored: Eisner / MST Beyond arc-factored: NP-hard Approximations ILP Chart-parsing on defined subset Summary
47 Where we ve been Arc-factored: Eisner / MST Beyond arc-factored: NP-hard Approximations ILP Chart-parsing on defined subset Summary What s next The return of grammars? Non-projective unsupervised parsing Making models practical on web-scale
Advanced Graph-Based Parsing Techniques
Advanced Graph-Based Parsing Techniques Joakim Nivre Uppsala University Linguistics and Philology Based on previous tutorials with Ryan McDonald Advanced Graph-Based Parsing Techniques 1(33) Introduction
The Real Story on Synchronous Rewriting Systems. Daniel Gildea Computer Science Department University of Rochester
The Real Story on Synchronous Rewriting Systems Daniel Gildea omputer Science Department University of Rochester Overview Factorization Tree Decomposition Factorization of LFRS YK Parsing w 2 : [, x 1,
Polyhedral Outer Approximations with Application to Natural Language Parsing
Polyhedral Outer Approximations with Application to Natural Language Parsing André F. T. Martins 1,2 Noah A. Smith 1 Eric P. Xing 1 1 Language Technologies Institute School of Computer Science Carnegie
13A. Computational Linguistics. 13A. Log-Likelihood Dependency Parsing. CSC 2501 / 485 Fall 2017
Computational Linguistics CSC 2501 / 485 Fall 2017 13A 13A. Log-Likelihood Dependency Parsing Gerald Penn Department of Computer Science, University of Toronto Based on slides by Yuji Matsumoto, Dragomir
Natural Language Processing CS Lecture 06. Razvan C. Bunescu School of Electrical Engineering and Computer Science
Natural Language Processing CS 6840 Lecture 06 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Statistical Parsing Define a probabilistic model of syntax P(T S):
Introduction to Data-Driven Dependency Parsing
Introduction to Data-Driven Dependency Parsing Introductory Course, ESSLLI 2007 Ryan McDonald 1 Joakim Nivre 2 1 Google Inc., New York, USA E-mail: ryanmcd@google.com 2 Uppsala University and Växjö University,
Marrying Dynamic Programming with Recurrent Neural Networks
Marrying Dynamic Programming with Recurrent Neural Networks I eat sushi with tuna from Japan Liang Huang Oregon State University Structured Prediction Workshop, EMNLP 2017, Copenhagen, Denmark Marrying
Quasi-Second-Order Parsing for 1-Endpoint-Crossing, Pagenumber-2 Graphs
Quasi-Second-Order Parsing for 1-Endpoint-Crossing, Pagenumber-2 Graphs Junjie Cao, Sheng Huang, Weiwei Sun, Xiaojun Wan Institute of Computer Science and Technology Peking University September 5, 2017
Quasi-Synchronous Phrase Dependency Grammars for Machine Translation. lti
Quasi-Synchronous Phrase Dependency Grammars for Machine Translation Kevin Gimpel Noah A. Smith 1 Introduction MT using dependency grammars on phrases Phrases capture local reordering and idiomatic translations
Relaxed Marginal Inference and its Application to Dependency Parsing
Relaxed Marginal Inference and its Application to Dependency Parsing Sebastian Riedel David A. Smith Department of Computer Science University of Massachusetts, Amherst {riedel,dasmith}@cs.umass.edu Abstract
Decoding and Inference with Syntactic Translation Models
Decoding and Inference with Syntactic Translation Models March 5, 2013 CFGs S NP VP VP NP V V NP NP CFGs S NP VP S VP NP V V NP NP CFGs S NP VP S VP NP V NP VP V NP NP CFGs S NP VP S VP NP V NP VP V NP
Lab 12: Structured Prediction
December 4, 2014 Lecture plan structured perceptron application: confused messages application: dependency parsing structured SVM Class review: from modelization to classification What does learning mean?
Treebank Grammar Techniques for Non-Projective Dependency Parsing
Treebank Grammar Techniques for Non-Projective Dependency Parsing Marco Kuhlmann Uppsala University Uppsala, Sweden marco.kuhlmann@lingfil.uu.se Giorgio Satta University of Padua Padova, Italy satta@dei.unipd.it
Spectral Unsupervised Parsing with Additive Tree Metrics
Spectral Unsupervised Parsing with Additive Tree Metrics Ankur Parikh, Shay Cohen, Eric P. Xing Carnegie Mellon, University of Edinburgh Ankur Parikh 2014 1 Overview Model: We present a novel approach
Computational Linguistics
Computational Linguistics Dependency-based Parsing Clayton Greenberg Stefan Thater FR 4.7 Allgemeine Linguistik (Computerlinguistik) Universität des Saarlandes Summer 2016 Acknowledgements These slides
Personal Project: Shift-Reduce Dependency Parsing
Personal Project: Shift-Reduce Dependency Parsing 1 Problem Statement The goal of this project is to implement a shift-reduce dependency parser. This entails two subgoals: Inference: We must have a shift-reduce
Computational Linguistics. Acknowledgements. Phrase-Structure Trees. Dependency-based Parsing
Computational Linguistics Dependency-based Parsing Dietrich Klakow & Stefan Thater FR 4.7 Allgemeine Linguistik (Computerlinguistik) Universität des Saarlandes Summer 2013 Acknowledgements These slides
Dependency Parsing. Statistical NLP Fall (Non-)Projectivity. CoNLL Format. Lecture 9: Dependency Parsing
Dependency Parsing Statistical NLP Fall 2016 Lecture 9: Dependency Parsing Slav Petrov Google prep dobj ROOT nsubj pobj det PRON VERB DET NOUN ADP NOUN They solved the problem with statistics CoNLL Format
Vine Pruning for Efficient Multi-Pass Dependency Parsing. Alexander M. Rush and Slav Petrov
Vine Pruning for Efficient Multi-Pass Dependency Parsing Alexander M. Rush and Slav Petrov Dependency Parsing Styles of Dependency Parsing greedy O(n) transition-based parsers (Nivre 2004) graph-based
Probabilistic Context Free Grammars. Many slides from Michael Collins
Probabilistic Context Free Grammars Many slides from Michael Collins Overview I Probabilistic Context-Free Grammars (PCFGs) I The CKY Algorithm for parsing with PCFGs A Probabilistic Context-Free Grammar
Maximum Flow Problem (Ford and Fulkerson, 1956)
Maximum Flow Problem (Ford and Fulkerson, 196) In this problem we find the maximum flow possible in a directed connected network with arc capacities. There is unlimited quantity available in the given
Soft Inference and Posterior Marginals. September 19, 2013
Soft Inference and Posterior Marginals September 19, 2013 Soft vs. Hard Inference Hard inference Give me a single solution Viterbi algorithm Maximum spanning tree (Chu-Liu-Edmonds alg.) Soft inference
Statistical Methods for NLP
Statistical Methods for NLP Stochastic Grammars Joakim Nivre Uppsala University Department of Linguistics and Philology joakim.nivre@lingfil.uu.se Statistical Methods for NLP 1(22) Structured Classification
Parsing. Unger s Parser. Laura Kallmeyer. Winter 2016/17. Heinrich-Heine-Universität Düsseldorf 1 / 21
Parsing Unger s Parser Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Winter 2016/17 1 / 21 Table of contents 1 Introduction 2 The Parser 3 An Example 4 Optimizations 5 Conclusion 2 / 21 Introduction
CMPSCI611: The Matroid Theorem Lecture 5
CMPSCI611: The Matroid Theorem Lecture 5 We first review our definitions: A subset system is a set E together with a set of subsets of E, called I, such that I is closed under inclusion. This means that
Unit 1A: Computational Complexity
Unit 1A: Computational Complexity Course contents: Computational complexity NP-completeness Algorithmic Paradigms Readings Chapters 3, 4, and 5 Unit 1A 1 O: Upper Bounding Function Def: f(n)= O(g(n)) if
Natural Language Processing. Classification. Features. Some Definitions. Classification. Feature Vectors. Classification I. Dan Klein UC Berkeley
Natural Language Processing Classification Classification I Dan Klein UC Berkeley Classification Automatically make a decision about inputs Example: document category Example: image of digit digit Example:
Lecture 15. Probabilistic Models on Graph
Lecture 15. Probabilistic Models on Graph Prof. Alan Yuille Spring 2014 1 Introduction We discuss how to define probabilistic models that use richly structured probability distributions and describe how
Language and Statistics II
Language and Statistics II Lecture 19: EM for Models of Structure Noah Smith Epectation-Maimization E step: i,, q i # p r $ t = p r i % ' $ t i, p r $ t i,' soft assignment or voting M step: r t +1 # argma
Structured Prediction Models via the Matrix-Tree Theorem
Structured Prediction Models via the Matrix-Tree Theorem Terry Koo Amir Globerson Xavier Carreras Michael Collins maestro@csail.mit.edu gamir@csail.mit.edu carreras@csail.mit.edu mcollins@csail.mit.edu
NP Completeness and Approximation Algorithms
Chapter 10 NP Completeness and Approximation Algorithms Let C() be a class of problems defined by some property. We are interested in characterizing the hardest problems in the class, so that if we can
Parsing. Unger s Parser. Introduction (1) Unger s parser [Grune and Jacobs, 2008] is a CFG parser that is
![Parsing. Unger s Parser. Introduction (1) Unger s parser [Grune and Jacobs, 2008] is a CFG parser that is](/thumbs/88/115486668.jpg "Parsing. Unger s Parser. Introduction (1) Unger s parser [Grune and Jacobs, 2008] is a CFG parser that is")
Introduction (1) Unger s parser [Grune and Jacobs, 2008] is a CFG parser that is Unger s Parser Laura Heinrich-Heine-Universität Düsseldorf Wintersemester 2012/2013 a top-down parser: we start with S and
Proof of Theorem 1. Tao Lei CSAIL,MIT. Here we give the proofs of Theorem 1 and other necessary lemmas or corollaries.
Proof of Theorem 1 Tao Lei CSAIL,MIT Here we give the proofs of Theorem 1 and other necessary lemmas or corollaries. Lemma 1 (Reachability) Any two trees y, y are reachable to each other. Specifically,
Probabilistic Context-Free Grammars. Michael Collins, Columbia University
Probabilistic Context-Free Grammars Michael Collins, Columbia University Overview Probabilistic Context-Free Grammars (PCFGs) The CKY Algorithm for parsing with PCFGs A Probabilistic Context-Free Grammar
Easy Problems vs. Hard Problems. CSE 421 Introduction to Algorithms Winter Is P a good definition of efficient? The class P
Easy Problems vs. Hard Problems CSE 421 Introduction to Algorithms Winter 2000 NP-Completeness (Chapter 11) Easy - problems whose worst case running time is bounded by some polynomial in the size of the
5 Integer Linear Programming (ILP) E. Amaldi Foundations of Operations Research Politecnico di Milano 1
5 Integer Linear Programming (ILP) E. Amaldi Foundations of Operations Research Politecnico di Milano 1 Definition: An Integer Linear Programming problem is an optimization problem of the form (ILP) min
Discrimina)ve Latent Variable Models. SPFLODD November 15, 2011
Discrimina)ve Latent Variable Models SPFLODD November 15, 2011 Lecture Plan 1. Latent variables in genera)ve models (review) 2. Latent variables in condi)onal models 3. Latent variables in structural SVMs
Stacking Dependency Parsers
Stacking Dependency Parsers André F. T. Martins Dipanjan Das Noah A. Smith Eric P. Xing School of Computer Science, Carnegie Mellon University, Pittsburgh, PA 15213, USA Instituto de Telecomunicações,
Algorithms for Syntax-Aware Statistical Machine Translation
Algorithms for Syntax-Aware Statistical Machine Translation I. Dan Melamed, Wei Wang and Ben Wellington ew York University Syntax-Aware Statistical MT Statistical involves machine learning (ML) seems crucial
Lecture 18: P & NP. Revised, May 1, CLRS, pp
Lecture 18: P & NP Revised, May 1, 2003 CLRS, pp.966-982 The course so far: techniques for designing efficient algorithms, e.g., divide-and-conquer, dynamic-programming, greedy-algorithms. What happens
A Supertag-Context Model for Weakly-Supervised CCG Parser Learning
A Supertag-Context Model for Weakly-Supervised CCG Parser Learning Dan Garrette Chris Dyer Jason Baldridge Noah A. Smith U. Washington CMU UT-Austin CMU Contributions 1. A new generative model for learning
Outline. Parsing. Approximations Some tricks Learning agenda-based parsers
Outline Parsing CKY Approximations Some tricks Learning agenda-based parsers 0 Parsing We need to compute compute argmax d D(x) p θ (d x) Inference: Find best tree given model 1 Parsing We need to compute
Products of Weighted Logic Programs
Under consideration for publication in Theory and Practice of Logic Programming 1 Products of Weighted Logic Programs SHAY B. COHEN, ROBERT J. SIMMONS, and NOAH A. SMITH School of Computer Science Carnegie
Probabilistic Graphical Models
Probabilistic Graphical Models David Sontag New York University Lecture 6, March 7, 2013 David Sontag (NYU) Graphical Models Lecture 6, March 7, 2013 1 / 25 Today s lecture 1 Dual decomposition 2 MAP inference
S NP VP 0.9 S VP 0.1 VP V NP 0.5 VP V 0.1 VP V PP 0.1 NP NP NP 0.1 NP NP PP 0.2 NP N 0.7 PP P NP 1.0 VP NP PP 1.0. N people 0.
/6/7 CS 6/CS: Natural Language Processing Instructor: Prof. Lu Wang College of Computer and Information Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang The grammar: Binary, no epsilons,.9..5
CS281A/Stat241A Lecture 19
CS281A/Stat241A Lecture 19 p. 1/4 CS281A/Stat241A Lecture 19 Junction Tree Algorithm Peter Bartlett CS281A/Stat241A Lecture 19 p. 2/4 Announcements My office hours: Tuesday Nov 3 (today), 1-2pm, in 723
NP-Completeness. Andreas Klappenecker. [based on slides by Prof. Welch]
![NP-Completeness. Andreas Klappenecker. [based on slides by Prof. Welch]](/thumbs/79/79291182.jpg "NP-Completeness. Andreas Klappenecker. [based on slides by Prof. Welch]")
NP-Completeness Andreas Klappenecker [based on slides by Prof. Welch] 1 Prelude: Informal Discussion (Incidentally, we will never get very formal in this course) 2 Polynomial Time Algorithms Most of the
This kind of reordering is beyond the power of finite transducers, but a synchronous CFG can do this.
Chapter 12 Synchronous CFGs Synchronous context-free grammars are a generalization of CFGs that generate pairs of related strings instead of single strings. They are useful in many situations where one
Some Writing Examples
Some Writing Examples Assume that A : ( A = n = x, y A : f(x) = f(y)). Let B = n + 1, x, y B, x y, C := B \ {y}, D := B \ {x}. C = n, D = n, x C, y D. Let u B, u x, u y. u C, f(x) = f(u), u D, f(y) = f(u),
the tree till a class assignment is reached
Decision Trees Decision Tree for Playing Tennis Prediction is done by sending the example down Prediction is done by sending the example down the tree till a class assignment is reached Definitions Internal
Lecture 21: Spectral Learning for Graphical Models
10-708: Probabilistic Graphical Models 10-708, Spring 2016 Lecture 21: Spectral Learning for Graphical Models Lecturer: Eric P. Xing Scribes: Maruan Al-Shedivat, Wei-Cheng Chang, Frederick Liu 1 Motivation
Fisher Information in Gaussian Graphical Models
Fisher Information in Gaussian Graphical Models Jason K. Johnson September 21, 2006 Abstract This note summarizes various derivations, formulas and computational algorithms relevant to the Fisher information
Bringing machine learning & compositional semantics together: central concepts
Bringing machine learning & compositional semantics together: central concepts https://githubcom/cgpotts/annualreview-complearning Chris Potts Stanford Linguistics CS 244U: Natural language understanding
/633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Matroids and Greedy Algorithms Date: 10/31/16
60.433/633 Introduction to Algorithms Lecturer: Michael Dinitz Topic: Matroids and Greedy Algorithms Date: 0/3/6 6. Introduction We talked a lot the last lecture about greedy algorithms. While both Prim
The minimum G c cut problem
The minimum G c cut problem Abstract In this paper we define and study the G c -cut problem. Given a complete undirected graph G = (V ; E) with V = n, edge weighted by w(v i, v j ) 0 and an undirected
Probabilistic Graphical Models: MRFs and CRFs. CSE628: Natural Language Processing Guest Lecturer: Veselin Stoyanov
Probabilistic Graphical Models: MRFs and CRFs CSE628: Natural Language Processing Guest Lecturer: Veselin Stoyanov Why PGMs? PGMs can model joint probabilities of many events. many techniques commonly
Better! Faster! Stronger*!
Jason Eisner Jiarong Jiang He He Better! Faster! Stronger*! Learning to balance accuracy and efficiency when predicting linguistic structures (*theorems) Hal Daumé III UMD CS, UMIACS, Linguistics me@hal3.name
Context-Free Parsing: CKY & Earley Algorithms and Probabilistic Parsing
Context-Free Parsing: CKY & Earley Algorithms and Probabilistic Parsing Natural Language Processing CS 4120/6120 Spring 2017 Northeastern University David Smith with some slides from Jason Eisner & Andrew
Recurrent neural network grammars
Widespread phenomenon: Polarity items can only appear in certain contexts Recurrent neural network grammars lide credits: Chris Dyer, Adhiguna Kuncoro Example: anybody is a polarity item that tends to
Aspects of Tree-Based Statistical Machine Translation
Aspects of Tree-Based Statistical Machine Translation Marcello Federico Human Language Technology FBK 2014 Outline Tree-based translation models: Synchronous context free grammars Hierarchical phrase-based
Markov Networks.
Markov Networks www.biostat.wisc.edu/~dpage/cs760/ Goals for the lecture you should understand the following concepts Markov network syntax Markov network semantics Potential functions Partition function
Maxent Models and Discriminative Estimation
Maxent Models and Discriminative Estimation Generative vs. Discriminative models (Reading: J+M Ch6) Introduction So far we ve looked at generative models Language models, Naive Bayes But there is now much
So# Inference and Posterior Marginals. September 19, 2013
So# Inference and Posterior Marginals September 19, 2013 So# vs. Hard Inference Hard inference Give me a single solucon Viterbi algorithm Maximum spanning tree (Chu- Liu- Edmonds alg.) So# inference Task
STA 4273H: Statistical Machine Learning
STA 4273H: Statistical Machine Learning Russ Salakhutdinov Department of Statistics! rsalakhu@utstat.toronto.edu! http://www.utstat.utoronto.ca/~rsalakhu/ Sidney Smith Hall, Room 6002 Lecture 11 Project
Probabilistic Graphical Models
School of Computer Science Probabilistic Graphical Models Max-margin learning of GM Eric Xing Lecture 28, Apr 28, 2014 b r a c e Reading: 1 Classical Predictive Models Input and output space: Predictive
Spectral Clustering. Guokun Lai 2016/10
Spectral Clustering Guokun Lai 2016/10 1 / 37 Organization Graph Cut Fundamental Limitations of Spectral Clustering Ng 2002 paper (if we have time) 2 / 37 Notation We define a undirected weighted graph
Dependency grammar. Recurrent neural networks. Transition-based neural parsing. Word representations. Informs Models
Dependency grammar Morphology Word order Transition-based neural parsing Word representations Recurrent neural networks Informs Models Dependency grammar Morphology Word order Transition-based neural parsing
P, NP, NP-Complete, and NPhard
P, NP, NP-Complete, and NPhard Problems Zhenjiang Li 21/09/2011 Outline Algorithm time complicity P and NP problems NP-Complete and NP-Hard problems Algorithm time complicity Outline What is this course
Efficient Parsing of Well-Nested Linear Context-Free Rewriting Systems
Efficient Parsing of Well-Nested Linear Context-Free Rewriting Systems Carlos Gómez-Rodríguez 1, Marco Kuhlmann 2, and Giorgio Satta 3 1 Departamento de Computación, Universidade da Coruña, Spain, cgomezr@udc.es
3 : Representation of Undirected GM
10-708: Probabilistic Graphical Models 10-708, Spring 2016 3 : Representation of Undirected GM Lecturer: Eric P. Xing Scribes: Longqi Cai, Man-Chia Chang 1 MRF vs BN There are two types of graphical models:
Preliminaries. Introduction to EF-games. Inexpressivity results for first-order logic. Normal forms for first-order logic
Introduction to EF-games Inexpressivity results for first-order logic Normal forms for first-order logic Algorithms and complexity for specific classes of structures General complexity bounds Preliminaries
Markov Networks. l Like Bayes Nets. l Graphical model that describes joint probability distribution using tables (AKA potentials)
Markov Networks l Like Bayes Nets l Graphical model that describes joint probability distribution using tables (AKA potentials) l Nodes are random variables l Labels are outcomes over the variables Markov
Branch-and-Bound. Leo Liberti. LIX, École Polytechnique, France. INF , Lecture p. 1
Branch-and-Bound Leo Liberti LIX, École Polytechnique, France INF431 2011, Lecture p. 1 Reminders INF431 2011, Lecture p. 2 Problems Decision problem: a question admitting a YES/NO answer Example HAMILTONIAN
Hidden Markov Models. x 1 x 2 x 3 x N
Hidden Markov Models 1 1 1 1 K K K K x 1 x x 3 x N Example: The dishonest casino A casino has two dice: Fair die P(1) = P() = P(3) = P(4) = P(5) = P(6) = 1/6 Loaded die P(1) = P() = P(3) = P(4) = P(5)
Some Algebra Problems (Algorithmic) CSE 417 Introduction to Algorithms Winter Some Problems. A Brief History of Ideas
Some Algebra Problems (Algorithmic) CSE 417 Introduction to Algorithms Winter 2006 NP-Completeness (Chapter 8) Given positive integers a, b, c Question 1: does there exist a positive integer x such that
Undirected Graphical Models: Markov Random Fields
Undirected Graphical Models: Markov Random Fields 40-956 Advanced Topics in AI: Probabilistic Graphical Models Sharif University of Technology Soleymani Spring 2015 Markov Random Field Structure: undirected
Word Embeddings in Feedforward Networks; Tagging and Dependency Parsing using Feedforward Networks. Michael Collins, Columbia University
Word Embeddings in Feedforward Networks; Tagging and Dependency Parsing using Feedforward Networks Michael Collins, Columbia University Overview Introduction Multi-layer feedforward networks Representing
Multilevel Coarse-to-Fine PCFG Parsing
Multilevel Coarse-to-Fine PCFG Parsing Eugene Charniak, Mark Johnson, Micha Elsner, Joseph Austerweil, David Ellis, Isaac Haxton, Catherine Hill, Shrivaths Iyengar, Jeremy Moore, Michael Pozar, and Theresa
Markov Networks. l Like Bayes Nets. l Graph model that describes joint probability distribution using tables (AKA potentials)
Markov Networks l Like Bayes Nets l Graph model that describes joint probability distribution using tables (AKA potentials) l Nodes are random variables l Labels are outcomes over the variables Markov
Massive Experiments and Observational Studies: A Linearithmic Algorithm for Blocking/Matching/Clustering
Massive Experiments and Observational Studies: A Linearithmic Algorithm for Blocking/Matching/Clustering Jasjeet S. Sekhon UC Berkeley June 21, 2016 Jasjeet S. Sekhon (UC Berkeley) Methods for Massive
Section Notes 8. Integer Programming II. Applied Math 121. Week of April 5, expand your knowledge of big M s and logical constraints.
Section Notes 8 Integer Programming II Applied Math 121 Week of April 5, 2010 Goals for the week understand IP relaxations be able to determine the relative strength of formulations understand the branch
1 T 1 = where 1 is the all-ones vector. For the upper bound, let v 1 be the eigenvector corresponding. u:(u,v) E v 1(u)
CME 305: Discrete Mathematics and Algorithms Instructor: Reza Zadeh (rezab@stanford.edu) Final Review Session 03/20/17 1. Let G = (V, E) be an unweighted, undirected graph. Let λ 1 be the maximum eigenvalue
Connotation: A Dash of Sentiment Beneath the Surface Meaning
Connotation: A Dash of Sentiment Beneath the Surface Meaning Con notation com- ( together or with ) notare ( to mark ) 1 Con notation com- ( together or with ) notare ( to mark ) Commonly understood cultural
Some Algebra Problems (Algorithmic) CSE 417 Introduction to Algorithms Winter Some Problems. A Brief History of Ideas
CSE 417 Introduction to Algorithms Winter 2007 Some Algebra Problems (Algorithmic) Given positive integers a, b, c Question 1: does there exist a positive integer x such that ax = c? NP-Completeness (Chapter
Probability and Structure in Natural Language Processing
Probability and Structure in Natural Language Processing Noah Smith, Carnegie Mellon University 2012 Interna@onal Summer School in Language and Speech Technologies Quick Recap Yesterday: Bayesian networks
Products of weighted logic programs
TLP: page 1 of 34 C Cambridge University Press 2011 doi:10.1017/s1471068410000529 1 Products of weighted logic programs SHAY B. COHEN, ROBERT J. SIMMONS and NOAH A. SMITH School of Computer Science, Carnegie
Part of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch. COMP-599 Oct 1, 2015
Part of Speech Tagging: Viterbi, Forward, Backward, Forward- Backward, Baum-Welch COMP-599 Oct 1, 2015 Announcements Research skills workshop today 3pm-4:30pm Schulich Library room 313 Start thinking about
9/12/17. Types of learning. Modeling data. Supervised learning: Classification. Supervised learning: Regression. Unsupervised learning: Clustering
Types of learning Modeling data Supervised: we know input and targets Goal is to learn a model that, given input data, accurately predicts target data Unsupervised: we know the input only and want to make
Cluster algebras, snake graphs and continued fractions. Ralf Schiffler
Cluster algebras, snake graphs and continued fractions Ralf Schiffler Intro Cluster algebras Continued fractions Snake graphs Intro Cluster algebras Continued fractions expansion formula via perfect matchings
NP-Complete Problems and Approximation Algorithms
NP-Complete Problems and Approximation Algorithms Efficiency of Algorithms Algorithms that have time efficiency of O(n k ), that is polynomial of the input size, are considered to be tractable or easy
Lecture 22: Counting
CS 710: Complexity Theory 4/8/2010 Lecture 22: Counting Instructor: Dieter van Melkebeek Scribe: Phil Rydzewski & Chi Man Liu Last time we introduced extractors and discussed two methods to construct them.
Computational and Statistical Learning Theory
Computational and Statistical Learning Theory TTIC 31120 Prof. Nati Srebro Lecture 6: Computational Complexity of Learning Proper vs Improper Learning Learning Using FIND-CONS For any family of hypothesis
Statistical Machine Translation of Natural Languages
1/26 Statistical Machine Translation of Natural Languages Heiko Vogler Technische Universität Dresden Germany Graduiertenkolleg Quantitative Logics and Automata Dresden, November, 2012 1/26 Weighted Tree
Algorithm Design and Analysis
Algorithm Design and Analysis LECTURE 7 Network Flow Application: Bipartite Matching Adam Smith 0//008 A. Smith; based on slides by S. Raskhodnikova and K. Wayne Recently: Ford-Fulkerson Find max s-t flow
Parsing Mildly Non-projective Dependency Structures
Parsing Mildly Non-projective Dependency Structures Carlos Gómez-Rodríguez Departamento de Computación Universidade da Coruña, Spain cgomezr@udc.es David Weir and John Carroll Department of Informatics
Artificial Intelligence
CS344: Introduction to Artificial Intelligence Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 20-21 Natural Language Parsing Parsing of Sentences Are sentences flat linear structures? Why tree? Is
Random Walks and Electric Resistance on Distance-Regular Graphs
Random Walks and Electric Resistance on Distance-Regular Graphs Greg Markowsky March 16, 2011 Graphs A graph is a set of vertices V (can be taken to be {1, 2,..., n}) and edges E, where each edge is an
CS626: NLP, Speech and the Web. Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 14: Parsing Algorithms 30 th August, 2012
CS626: NLP, Speech and the Web Pushpak Bhattacharyya CSE Dept., IIT Bombay Lecture 14: Parsing Algorithms 30 th August, 2012 Parsing Problem Semantics Part of Speech Tagging NLP Trinity Morph Analysis
NATIONAL UNIVERSITY OF SINGAPORE CS3230 DESIGN AND ANALYSIS OF ALGORITHMS SEMESTER II: Time Allowed 2 Hours
NATIONAL UNIVERSITY OF SINGAPORE CS3230 DESIGN AND ANALYSIS OF ALGORITHMS SEMESTER II: 2017 2018 Time Allowed 2 Hours INSTRUCTIONS TO STUDENTS 1. This assessment consists of Eight (8) questions and comprises
Chapter Finding parse trees
Chapter 16 NP Some tasks definitely require exponential time. That is, we can not only display an exponential-time algorithm, but we can also prove that the problem cannot be solved in anything less than