Transition-based dependency parsing

Transcription

1 Transition-based dependency parsing Daniël de Kok

2 Overview Dependency graphs and treebanks. Transition-based dependency parsing. Making parse choices using perceptrons.

3 Today Recap Transition systems Parsing algorithm

4 The book The slides on transition-based dependency parsing are based on Chapters 2 and 3 of Dependency Parsing by Sandra Kübler, Ryan McDonald, and Joakim Nivre This book is highly recommended if you want to learn about statistical dependency parsing.

5 Recap

6 Introduction Binary grammatical relations between tokens. Directed: one token is the head the other the dependent. Each word has a relation to another word.

7 Example Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

8 Example ROOT Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

9 Example ROOT SUBJ Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

10 Example ROOT ATTR SUBJ Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

11 Example ROOT OBJA ATTR SUBJ Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

12 Example ROOT OBJA ATTR SUBJ DET Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

13 Example ROOT PP OBJA ATTR SUBJ DET Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

14 Example ROOT PP OBJA ATTR SUBJ DET PN Behinderte Menschen veranstalteten einen Protesttag in Bremen. Disabled people organized a protest-day in Bremen.

15 Dependency graphs Following Kübler, et al., 2008: A sentence is a sequence of tokens: S = w 0 w 1... w n, wherein w 0 is an artificial root token. Let R = r 1... r m be the set of all possible dependency relation types. A dependency graph G =< V, A > is a labeled directed graph such that the following holds: V {S} A V R V if (w i, r, w j ) A then r r [(w i, r, w j ) A]

16 Dependency trees Dependency tree: A well-formed dependency-graph G =< V, A > for an input sentence S and a dependency relation set R is any dependency graph that is a directed tree originating out of w 0 and has the spanning node set V = V S. Properties: Root property: the root is not a dependent Spanning tree property Connectedness property: Every two words are connected via the reflexive transitive closure. Single-head property Acyclicity property Arc size property: A = V 1

17 Projectivity A dependency graph is non-projective if there are crossing edges. Common in languages with richer morphology and permit freer word orders: Czech Dutch German Why does this matter? Asuming projectivity allows for parsing algorithms that are computationally and conceptually simpler.

18 Example PP AUX ROOT ADV PN DET SUBJ DET OBJA Für diese Behauptung hat Beckmeyer bisher keinen Nachweis geliefert For this allegation has Beckmeyer until-now no proof given

19 Pseudo-projective parsing 1. Make trees projective 2. Train a projective parser on projective trees 3. Parse a new sentence 4. Convert the parsing result to a non-projective tree

20 Example OBJA:PP AUX ROOT ADV PN DET SUBJ DET OBJA Für diese Behauptung hat Beckmeyer bisher keinen Nachweis geliefert For this allegation has Beckmeyer until-now no proof given

21 Parsing techniques Grammar-based parsing: Context-Free Dependency Grammar Constraint Dependency Grammar Statistical dependency-parsing: Transition-based parsing Graph-based parsing

22 Transition systems

23 Introduction Transition system: Abstract machine Consisting of: Configurations (states) Transitions Initial state One or more accepting/final states Example: finite state automaton

24 Introduction (2) Transition-based dependency parsing also uses a transition system. However, in contrast to finite state automata: States have internal structure. Transitions correspond dependency tree derivation steps. We create states on-demand.

25 Configurations A configuration for a sentence S = w 0 w 1... w n is a triple c =< σ, β, A >, where: σ is a stack of words, w i σ : w i V S β is a buffer of words, w i β : w i V S A is a set of dependency arcs (w i, r, w j ) V S R V S

26 Configurations (2) A configuration represents a (partially) parsed sentence: σ: partially processed words β: unprocessed words 1 A: partially-built dependency tree 1 In some transition systems, the head of the buffer is partially processed.

27 Example σ = [ROOT, muß] β = [AWO-Konten, prüfen] A = {(muß, SUBJ, Staatsanwalt)}

28 Initial/final configuration For a sentence S: The initial configuration, c 0 (S), is ([w 0 ], [w 1... w n ], ) A final configuration has the form (σ, [], A)

29 Transitions Our states are now complex configurations as defined before. A transition can be seen as: A partial function from one state to another (like in finite state automata) A parsing action that e.g. adds an arc to the dependency tree. In the basic system for dependency parsing, three types of transitions are defined: Left-Arc r, Right-Arc r, and Shift.

30 Notation σ w i : the stack which results from pushing w i on the stack σ. w i β: a buffer with w i as the head and β as the tail.

31 Shift (σ, w i β, A) (σ w i, β, A) Remove the next word from the buffer, push it on the top of the stack. Precondition: buffer is non-empty

32 Shift (Example) Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ]

33 Left-Arc r (σ w i, w j β, A) (σ, w j β, A (w j, r, w i )) Add a dependency arc (w j, r, w i ) to the arc set. w i is the word on top of the stack. w j the next word in the buffer. w i is popped from the stack. Preconditions: 1. Stack and buffer are not empty. 2. i 0

34 Left-Arc r (example) (σ w i, w j β, A) (σ, w j β, A (w j, r, w i )) Operation σ β A [ROOT, Staatsanwalt] [muß, ] A LA SUBJ [ROOT] [muß, ] A = A (muß, SUBJ, Staatsanwalt)

35 Right-Arc r (σ w i, w j β, A) (σ, w i β, A (w i, r, w j )) Add a dependency arc (w i, r, w j ) to the arc set. w i is on top of the stack. w j is the next word in the buffer. w i is popped from the stack and replaces w j as the head of the buffer. Precondition: stack and buffer are not empty.

36 Right-Arc r (Example) (σ w i, w j β, A) (σ, w i β, A (w i, r, w j )) Operation σ β A [ROOT, muß] [prüfen] A RA AUX [ROOT] [muß] A = A (muß, AUX, prüfen)

37 Right-Arc r (Example) (σ w i, w j β, A) (σ, w i β, A (w i, r, w j )) Operation σ β A [ROOT, muß] [prüfen] A RA AUX [ROOT] [muß] A = A (muß, AUX, prüfen) Question: why is the head put on the buffer?

38 Transition sequence A transition sequence for a sentence S is a sequence of configurations, C 0,m = (c 0, c 1,..., c m ) such that 1. c 0 is the initial configuration c 0 (S) for S 2. c m is a terminal configuration, 3. for every i in 1... m, there is a transition t in the transition system such that c i = t(c i 1 ). The final dependency tree is G cm =< V S, A cm >

39 Example (Dependency tree) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check

40 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...]

41 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ]

42 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)}

43 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)} SH [ROOT, muß] [AWO-Konten, ] A 1

44 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)} SH [ROOT, muß] [AWO-Konten, ] A 1 SH [ROOT, muß, AWO-Konten] [prüfen] A 1

45 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)} SH [ROOT, muß] [AWO-Konten, ] A 1 SH [ROOT, muß, AWO-Konten] [prüfen] A 1 LA OBJA [ROOT, muß] [prüfen] A 2 = A 1 {(prüfen, OBJA, AWO-Konten)}

46 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)} SH [ROOT, muß] [AWO-Konten, ] A 1 SH [ROOT, muß, AWO-Konten] [prüfen] A 1 LA OBJA [ROOT, muß] [prüfen] A 2 = A 1 {(prüfen, OBJA, AWO-Konten)} RA AUX [ROOT] [muß] A 3 = A 2 {(muß, AUX, prüfen)}

47 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)} SH [ROOT, muß] [AWO-Konten, ] A 1 SH [ROOT, muß, AWO-Konten] [prüfen] A 1 LA OBJA [ROOT, muß] [prüfen] A 2 = A 1 {(prüfen, OBJA, AWO-Konten)} RA AUX [ROOT] [muß] A 3 = A 2 {(muß, AUX, prüfen)} RA ROOT [] [ROOT] A 4 = A 3 {(ROOT, ROOT, muß)}

48 Example (Transition sequence) ROOT SUBJ AUX OBJA ROOT Staatsanwalt muß AWO-konten prüfen ROOT DA has-to AWO-accounts check Operation σ β A [ROOT] [Staatsanwalt,...] SH [ROOT, Staatsanwalt] [muß, ] LA SUBJ [ROOT] [muß, ] A 1 = {(muß, SUBJ, Staatsanwalt)} SH [ROOT, muß] [AWO-Konten, ] A 1 SH [ROOT, muß, AWO-Konten] [prüfen] A 1 LA OBJA [ROOT, muß] [prüfen] A 2 = A 1 {(prüfen, OBJA, AWO-Konten)} RA AUX [ROOT] [muß] A 3 = A 2 {(muß, AUX, prüfen)} RA ROOT [] [ROOT] A 4 = A 3 {(ROOT, ROOT, muß)} SH [ROOT] [] A 4

49 Properties of resulting graphs A graph produced using in a transition sequence using this transition system has the following properties: Single-headedness: since Left-Arc r and Right-Arc r remove the dependent token from the stack/buffer, this token cannot be attached as a dependant of another token. Root: the special ROOT token is never a dependeny of another token, Left-Arc r is not allowed when ROOT is on top of the stack. It is not possible to have ROOT as the next element on the buffer, while having a non-empty stack.

50 Properties of resulting graphs (2) Spanning: the dependency graph contains each token of the input sentence. The graph does not necessarily have the connectedness property: e.g. consider a transition sequence consisting of Shifts only. The resulting graphs/trees are projective.

51 In-class assignment Proof (informally) that the arc-standard system cannot introduce non-projective relations.

52 Try it yourself! OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Reminder: Shift: (σ, w i β, A) (σ w i, β, A) Left-Arc r : (σ w i, w j β, A) (σ, w j β, A (w j, r, w i )) Right-Arc r : (σ w i, w j β, A) (σ, w i β, A (w i, r, w j ))

53 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...]

54 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...]

55 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...]

56 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)}

57 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)}

58 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2

59 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2 SH [..., Spendengeld] [?] A 2

60 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2 SH [..., Spendengeld] [?] A 2 RA PUNCT [ROOT, Veruntreute] [Spendengeld] A 3 = A 2 {(Spendengeld, PUNCT,?)}

61 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2 SH [..., Spendengeld] [?] A 2 RA PUNCT [ROOT, Veruntreute] [Spendengeld] A 3 = A 2 {(Spendengeld, PUNCT,?)} RA OBJA [ROOT] [Veruntreute] A 4 = A 3 {(Veruntreute, OBJA, Spendengeld)}

62 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2 SH [..., Spendengeld] [?] A 2 RA PUNCT [ROOT, Veruntreute] [Spendengeld] A 3 = A 2 {(Spendengeld, PUNCT,?)} RA OBJA [ROOT] [Veruntreute] A 4 = A 3 {(Veruntreute, OBJA, Spendengeld)} RA ROOT [] [ROOT] A 5 = A 4 {(ROOT, ROOT, Veruntreute)}

63 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2 SH [..., Spendengeld] [?] A 2 RA PUNCT [ROOT, Veruntreute] [Spendengeld] A 3 = A 2 {(Spendengeld, PUNCT,?)} RA OBJA [ROOT] [Veruntreute] A 4 = A 3 {(Veruntreute, OBJA, Spendengeld)} RA ROOT [] [ROOT] A 5 = A 4 {(ROOT, ROOT, Veruntreute)} SH [ROOT] [] A 5

64 Solution OBJA ROOT SUBJ DET PUNCT ROOT Veruntreute die AWO Spendengeld? ROOT Did-embezzle the AWO donation-money? Operation σ β A [ROOT] [Veruntreute,...] SH [ROOT, Veruntreute] [die,...] SH [..., die] [AWO,...] LA DET [ROOT, Veruntreute] [AWO,...] A 1 = {(AWO, DET, die)} RA SUBJ [ROOT] [Veruntreute,...] A 2 = A 1 {(Veruntreute, SUBJ, AWO)} SH [ROOT, Veruntreute] [Spendengeld,?] A 2 SH [..., Spendengeld] [?] A 2 RA PUNCT [ROOT, Veruntreute] [Spendengeld] A 3 = A 2 {(Spendengeld, PUNCT,?)} RA OBJA [ROOT] [Veruntreute] A 4 = A 3 {(Veruntreute, OBJA, Spendengeld)} RA ROOT [] [ROOT] A 5 = A 4 {(ROOT, ROOT, Veruntreute)} SH [ROOT] [] A 5

65 Issues The transition system is clearly non-deterministic: Item(s) in the buffer and top of the stack is not ROOT: Shift, Left-Arc r, and Right-Arc r Item(s) in the buffer and top of the stack is ROOT: Shift and Right-Arc r Empty stack, non-empty buffer: Shift For Left-Arc r and Right-Arc r, is any r R possible?

66 Parsing algorithm

67 Introduction As observed: there is usually more than one transition that is valid for a non-terminal configuration. Each choice leads to a different transition sequence and (typically) dependency graph. We are interested in the best parse: the parse that is the most likely interpretation of a sentence. Choose the transition that will lead to the best parse.

68 Oracle Assume that we were given an oracle function o. The oracle function will, for a given configuration c, give a transition o(c) = t, such that t is the correct transition. Using the o, one can implement a deterministic parser that will give the best parse.

69 Parsing using an oracle c c 0 (S) while isterminal(c) do t o(c) c t(c) end while

70 Properties O(n), where n is S, provided that the oracle and transition functions can be computed in constant time: The buffer size is bound by n: The maximum size of the buffer is S 1 (in the initial configuration). None of the transitions increases the size of the buffer. The stack size is also bound by n: Each Shift increases the stack size by 1. Maximum stack size: Shift until all buffer tokens are on the stack. Maximum possible number of Shifts: β Each transition decreases the stack or buffer size by 1.

71 Classifier Of course, such an oracle function is normally not available. Rather than o, we use a classification function λ. λ is a function learned from training data. λ predicts the best transition based on features a configuration. Intuition: if a similar configuration occurred in the training data, it can say something about which transition should be chosen.

72 Parsing using a classifier c c 0 (S) while isterminal(c) do t λ c c t(c) end while

73 More on classifiers Next week!