Lecture 7: Introduction to syntax-based MT

Transcription

1 Lecture 7: Introduction to syntax-based MT Andreas Maletti Statistical Machine Translation Stuttgart December 16, 2011 SMT VII A. Maletti 1

2 Lecture 7 Goals Overview Tree substitution grammars (tree automata) Synchronous grammars (tree transducers) SMT VII A. Maletti 2

3 Contents 1 Overview 2 Tree reresentations 3 Bar-Hillel Construction 4 Variants 5 Tree Transducers SMT VII A. Maletti 3

4 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: And then the matter was decided, and everything was ut in lace Outut: SMT VII A. Maletti 4

5 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: then the matter was decided, and everything was ut in lace Outut: SMT VII A. Maletti 4

6 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was decided, and everything was ut in lace Outut: SMT VII A. Maletti 4

7 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was decided, and everything was ut in lace Outut: f SMT VII A. Maletti 4

8 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was decided, and everything was ut in lace Outut: f kan SMT VII A. Maletti 4

9 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was decided, and everything was ut in lace Outut: f kan SMT VII A. Maletti 4

10 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was decided, and everything was ut in lace Outut: f kan SMT VII A. Maletti 4

11 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was decided, and everything was ut in lace Outut: f kan SMT VII A. Maletti 4

12 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter, and everything was ut in lace Outut: f kan An tm AlHsm SMT VII A. Maletti 4

13 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter and everything was ut in lace Outut: f kan An tm AlHsm SMT VII A. Maletti 4

14 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter everything was ut in lace Outut: f kan An tm AlHsm w SMT VII A. Maletti 4

15 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was ut in lace Outut: f kan An tm AlHsm w SMT VII A. Maletti 4

16 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter was ut in lace Outut: f kan An tm AlHsm w SMT VII A. Maletti 4

17 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: the matter in lace Outut: f kan An tm AlHsm w wdet SMT VII A. Maletti 4

18 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: in lace Outut: f kan An tm AlHsm w wdet Al>mwr SMT VII A. Maletti 4

19 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: lace Outut: f kan An tm AlHsm w wdet Al>mwr fy SMT VII A. Maletti 4

20 Word-based system (FST) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: Outut: f kan An tm AlHsm w wdet Al>mwr fy nsab ha SMT VII A. Maletti 4

21 Phrase-based machine translation Schema Inut Machine translation system Language model Outut Phrase-based systems Inut Segmenter Machine translation system Language model Outut SMT VII A. Maletti 5

22 Phrase-based machine translation Schema Inut Machine translation system Language model Outut Phrase-based systems Inut Segmenter Machine translation system Language model Outut SMT VII A. Maletti 5

23 Phrase-based system (FST+Perm) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: And then the matter was decided, and everything was ut in lace Outut: SMT VII A. Maletti 6

24 Phrase-based system (FST+Perm) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: And then 1 the matter 5 was decided 2, and everything 3 was ut 4 in lace 6 Outut: SMT VII A. Maletti 6

25 Phrase-based system (FST+Perm) And then the matter was decided, and everything was ut in lace f kan An tm AlHsm w wdet Al>mwr fy nsab ha Derivation Inut: And then 1 the matter 5 was decided 2, and everything 3 was ut 4 in lace 6 Outut: f kan 1 An tm AlHsm 2 w 3 wdet 4 Almwr 5 fy nsab ha 6 SMT VII A. Maletti 6

26 Machine translation (cont d) Phrase-based systems Inut Segmenter Machine translation system Language model Outut Syntax-based systems Inut Parser Machine translation system Language model Outut SMT VII A. Maletti 7

27 Machine translation (cont d) Phrase-based systems Inut Segmenter Machine translation system Language model Outut Syntax-based systems Inut Parser Machine translation system Language model Outut SMT VII A. Maletti 7

28 Syntax-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 8

29 Syntax-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 8

30 Syntax-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 8

31 Syntax-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 8

32 Syntax-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 8

33 Parser S S CC S CC ADVP NP-SBJ-9 VP and NP-SBJ-1 VP And RB DT NN VBD VP NN VBD VP then the matter was VBN NP-9 everything was VBN NP-1 PP decided, ut * IN NP in NN lace And then the matter was decided, and everything was ut in lace (thanks to KEVIN KNIGHT for the data) SMT VII A. Maletti 9

34 Semantics-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 10

35 Semantics-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 10

36 Semantics-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 10

37 Semantics-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 10

38 Semantics-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 10

39 Semantics-based Aroach Overview English German Semantics Syntax Phrase SMT VII A. Maletti 10

40 Contents 1 Overview 2 Tree reresentations 3 Bar-Hillel Construction 4 Variants 5 Tree Transducers SMT VII A. Maletti 11

41 Parsing and CFG Examle (Context-free grammar) S NP VP VP VBP ADVP JJ Colorless NNS ideas RB furiously NP JJ JJ NNS ADVP RB JJ green VBP slee Derivation S Colorless green ideas slee furiously SMT VII A. Maletti 12

42 Parse tree Examle S NP VP JJ JJ NNS VBP ADVP Colorless green ideas slee RB furiously Remark We are interested in the arse tree, not just whether S w! SMT VII A. Maletti 13

43 Parse tree Examle S NP VP JJ JJ NNS VBP ADVP Colorless green ideas slee RB furiously Remark We are interested in the arse tree, not just whether S w! SMT VII A. Maletti 13

44 Parse tree Examle S NP VP JJ JJ NNS VBP ADVP Colorless green ideas slee RB furiously Remark We are interested in the arse tree, not just whether S w! But there can be exonentially many arse trees for a sentence. SMT VII A. Maletti 13

45 Packed tree language Remark A tree language is often called forest in NLP. Examle S S NP VP NP VP JJ JJ NNS VBP ADVP JJ JJ NNS VBP ADVP Colorless green ideas slee RB green green ideas slee RB furiously furiously SMT VII A. Maletti 14

46 Packed tree language Remark A tree language is often called forest in NLP. Examle S S NP VP NP VP JJ JJ NNS VBP ADVP JJ JJ NNS VBP ADVP Colorless green ideas slee RB green green ideas slee RB S furiously S furiously NP NP VP JJ green JJ Colorless JJ green NNS ideas VBP slee ADVP RB furiously SMT VII A. Maletti 14

47 Local tree language Definition A local tree grammar is a grammar with rules of the form S S N 1... N k SMT VII A. Maletti 15

48 Local tree language Examle S NP S VP NP NP JJ JJ NNS VP VBP VP ADVP ADVP ADVP RB JJ JJ Colorless JJ JJ green NNS NNS ideas VBP VBP slee RB RB furiously Derivation S NP VP JJ JJ NNS VBP ADVP Colorless green ideas slee RB furiously SMT VII A. Maletti 16

49 Local tree language Definition The tree languages generated by local tree grammars are the local tree languages. Theorem The set of derivations of a context-free grammar forms a local tree language. SMT VII A. Maletti 17

50 Local tree language Definition The tree languages generated by local tree grammars are the local tree languages. Theorem The set of derivations of a context-free grammar forms a local tree language. SMT VII A. Maletti 17

51 Local tree language Question Is the tree language consisting of only S NP VP JJ JJ NNS VBP ADVP Colorless green ideas slee RB furiously a local tree language? SMT VII A. Maletti 18

52 Local tree language Question Is the tree language consisting of only S NP VP JJ JJ NNS VBP ADVP Colorless green ideas slee RB furiously a local tree language? Answer NO! SMT VII A. Maletti 18

53 Local tree language Notes Local tree languages have undesirable roerties: not closed under union cannot reresent all finite languages... SMT VII A. Maletti 19

54 Regular tree language Definition A regular tree grammar is a grammar with rules of the form q S q 1... q k The such generated languages are the regular tree languages. Remark Regular tree grammars are local tree grammars with hidden states. SMT VII A. Maletti 20

55 Regular tree language Definition A regular tree grammar is a grammar with rules of the form q S q 1... q k The such generated languages are the regular tree languages. Remark Regular tree grammars are local tree grammars with hidden states. SMT VII A. Maletti 20

56 Regular tree language Examle q S S q NP q NP q VP NP q VP JJ q JJ q NNS VP JJ JJ q VBP q ADVP q c q JJ JJ q NNS NNS q VBD VBD q g q i q s q ADVP ADVP q RB q RB RB q f Derivation q S SMT VII A. Maletti 21

57 Regular tree language Examle q S S q NP q NP q VP NP q VP JJ q JJ q NNS VP JJ JJ q VBP q ADVP q c q JJ JJ q NNS NNS q VBD VBD q g q i q s q ADVP ADVP q RB q RB RB q f Derivation q S q NP S q VP SMT VII A. Maletti 21

58 Regular tree language Examle q S S q NP q NP q VP NP q VP JJ q JJ q NNS VP JJ JJ q VBP q ADVP q c q JJ JJ q NNS NNS q VBD VBD q g q i q s q ADVP ADVP q RB q RB RB q f Derivation q S S q NP q VP NP S q VP JJ q JJ q NNS SMT VII A. Maletti 21

59 Regular tree languages Princial roerties Finite languages are regular Closed under all Boolean oerations Closed under relabelings, linear homomorhisms, inverse homomorhisms Can be determinized and minimized (bottom-u) Summary They are basically the tree version of finite-state automata with the same nice roerties. SMT VII A. Maletti 22

60 Trees and their yield Definition The yield of a tree is the string of its leaves (in natural order). Theorem The yield language of a regular tree language is context-free. Each context-free language is the yield of a regular tree language. SMT VII A. Maletti 23

61 Questions Question Is {σ(t, t) t arbitrary tree} regular? SMT VII A. Maletti 24

62 Questions Question Is {σ(t, t) t arbitrary tree} regular? Answer NO! SMT VII A. Maletti 24

63 Questions Question Is {σ(t, t) t arbitrary tree} regular? Answer NO! Question Is {σ(γ n (α), γ n (α)) n N} regular? SMT VII A. Maletti 24

64 Questions Question Is {σ(t, t) t arbitrary tree} regular? Answer NO! Question Is {σ(γ n (α), γ n (α)) n N} regular? Answer NO! Remark Not every tree language with context-free yield language is regular! SMT VII A. Maletti 24

65 Back to arsing Observation Most CFG-arsers are regular tree grammars (+ control) because they are based on a CFG ( local tree grammar) and have hidden states (or features) Alternative The features can be made exlicit in the arse tree structure. SMT VII A. Maletti 25

66 Back to arsing Observation Most CFG-arsers are regular tree grammars (+ control) because they are based on a CFG ( local tree grammar) and have hidden states (or features) Alternative The features can be made exlicit in the arse tree structure. SMT VII A. Maletti 25

67 Contents 1 Overview 2 Tree reresentations 3 Bar-Hillel Construction 4 Variants 5 Tree Transducers SMT VII A. Maletti 26

68 Binarization Inut tree S Binarized tree S NP VP NP VP JJ JJ NNS VBP VBP ADVP Colorless green ideas slee RB slee RB furiously Colorless JJ NNS furiously green ideas Theorem A tree language is regular if and only if its binarization is regular SMT VII A. Maletti 27

69 Binarization Inut tree S Binarized tree S NP VP NP VP JJ JJ NNS VBP VBP ADVP Colorless green ideas slee RB slee RB furiously Colorless JJ NNS furiously green ideas Theorem A tree language is regular if and only if its binarization is regular SMT VII A. Maletti 27

70 Individual runs Run on the yield () Colorless ( 1 ) ideas ( 2 ) slee ( 3 ) furiously ( ) Run on the inut tree S, q NP, q VP, q 1 JJ, q NNS, q VBP, q 1 ADVP, q 2 Colorless, w ideas, w slee, w RB, q 2 furiously, w SMT VII A. Maletti 28

71 Bar-Hillel construction Run on the yield () Colorless ( 1 ) ideas ( 2 ) slee ( 3 ) furiously ( ) Comosite run, S, q,, NP, q, 2 2, VP, q 1,, JJ, q, 1 1, NNS, q, 2 2, VBP, q 1, 3 3, ADVP, q 2,, Colorless, w, 1 1, ideas, w, 2 2, slee, w, 3 3, RB, q 2, 3, furiously, w, SMT VII A. Maletti 29

72 Bar-Hillel construction (cont d) Illustration σ q σ q = = = = = 1 q 1 1 q 1 2 q 2 ν(, α, q) α q = e q SMT VII A. Maletti 30

73 Bar-Hillel construction (cont d) Theorem The regular restriction of a regular tree language is regular Remark Comlexity: O(mn 3 ) m: size of the regular tree grammar n: size of the regular grammar (or inut string) Conclusion We can arse with regular tree grammars in O(mn 3 ). SMT VII A. Maletti 31

74 Contents 1 Overview 2 Tree reresentations 3 Bar-Hillel Construction 4 Variants 5 Tree Transducers SMT VII A. Maletti 32

75 To-down Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, S, q, 5 1, NP, q, 3 3, VP, q 1, 5 SMT VII A. Maletti 33

76 To-down Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, S, q, 5 1, NP, q, 3 3, VP, q 1, 5 1, JJ, q, 2 2, NNS, q, 3 SMT VII A. Maletti 33

77 To-down Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, S, q, 5 1, NP, q, 3 3, VP, q 1, 5 1, JJ, q, 2 2, NNS, q, 3 3, VBP, q 1, 4 4, ADVP, q 2, 5 1, Colorless, w, 2 2, ideas, w, 3 3, slee, w, 4 4, RB, q 2, 5 4, furiously, w, 5 SMT VII A. Maletti 33

78 Bottom-u Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, Colorless, w, 2 SMT VII A. Maletti 34

79 Bottom-u Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, JJ, q, 2 1, Colorless, w, 2 SMT VII A. Maletti 34

80 Bottom-u Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, JJ, q, 2 1, Colorless, w, 2 2, NNS, q, 3 2, ideas, w, 3 SMT VII A. Maletti 34

81 Bottom-u Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, NP, q, 3 1, JJ, q, 2 1, Colorless, w, 2 2, NNS, q, 3 2, ideas, w, 3 SMT VII A. Maletti 34

82 Bottom-u Bar-Hillel construction Run on the yield (1) Colorless (2) ideas (3) slee (4) furiously (5) Comosite run 1, S, q, 5 1, NP, q, 3 3, VP, q 1, 5 1, JJ, q, 2 2, NNS, q, 3 3, VBP, q 1, 4 4, ADVP, q 2, 5 1, Colorless, w, 2 2, ideas, w, 3 3, slee, w, 4 4, RB, q 2, 5 4, furiously, w, 5 SMT VII A. Maletti 34

83 Summary Key oints regular tree grammar as efficient tree data structure context-free behavior basis for most syntax-based translation models Further models weaker models are generally inadequate tree adjoining grammars (more exressive, but worse comutational roerties) Automata on directed acyclic grahs (see Daniel s lecture) SMT VII A. Maletti 35

84 Contents 1 Overview 2 Tree reresentations 3 Bar-Hillel Construction 4 Variants 5 Tree Transducers SMT VII A. Maletti 36

85 Extended To-down Tree Transducer Definition Each rule now has an inut and an outut side, which are both full trees (not just a single symbol followed by states) Examle σ q σ q r q q r α x x α x x SMT VII A. Maletti 37

86 Link Structure σ q q SMT VII A. Maletti 38

87 Link Structure σ q q SMT VII A. Maletti 38

88 Derivation Rules σ q q σ q r q r α x x α x x SMT VII A. Maletti 39

89 Derivation Rules σ q q σ q r q r α x x α x x SMT VII A. Maletti 39

90 Derivation Rules σ q q σ q r q r α x x α x x σ q q SMT VII A. Maletti 39

91 Derivation Rules σ q q σ q r q r α x x α x x σ q q σ q q SMT VII A. Maletti 39

92 Derivation Rules σ q q σ q r q r α x x α x x σ q q σ α q α q SMT VII A. Maletti 39

93 Derivation Rules σ q q σ q r q r α x x α x x σ α q α q σ α q α q SMT VII A. Maletti 39

94 Derivation Rules σ q q σ q r q r α x x α x x σ α q α q σ α q σ q r α q q r SMT VII A. Maletti 39

95 Derivation Rules σ q q σ q r q r α x x α x x σ α q σ q r α q q r σ α σ q r α q r SMT VII A. Maletti 39

96 Derivation Rules σ q q σ q r q r α x x α x x σ α σ q r α q r σ α σ α SMT VII A. Maletti 39

97 Semantics of XTOP σ α σ α SMT VII A. Maletti 40

98 Semantics of XTOP α σ σ α SMT VII A. Maletti 40

99 References Bar-Hillel, Perles, Shamir: On formal roerties of simle hrase structure grammars. Language and Information, 1964 Berstel, Reutenauer: Recognizable formal ower series on trees. Theoret. Comut. Sci. 18, , 1982 Borchardt: The theory of recognizable tree series. Ph.D. thesis TU Dresden, 2005 Gécseg, Steinby: Tree Automata. Akadémiai Kiadó, Budaest 1984 Nederhof, Satta: Probabilistic arsing as intersection. In IWPT 2003 Thank you for your attention! SMT VII A. Maletti 41