CroMo - Morphological Analysis for Croatian

Size: px

Start display at page:

Download "CroMo - Morphological Analysis for Croatian"

Belinda Leonard
5 years ago
Views:

1 - Morphological Analysis for Croatian Damir Ćavar 1, Ivo-Pavao Jazbec 2 and Tomislav Stojanov 2 Linguistics Department, University of Zadar 1 Institute of Croatian Language and Linguistics 2 FSMNLP 2008

3 Scenario Synchronic and diachronic study of language change and acquisition models Language data from a long period of time, and three major dialects in Croatia implying: Variation wrt. e.g. string-based morphology or feature bundles Ongoing discovery wrt. string combinatorics and features Research questions require quantitative and qualitative information: of phonological, morphological, syntactic and semantic tokens and feature bundles, and their correlation and variation at various stages over time

4 Morphological segmentation and annotation and lemmatization, and... Segmenting words: isponapijali su se they got drunk a little bit to satisfaction is po napija li Annotating segments: aspect prefix aspect prefix from stem-lemma napiti plural participle Extending the annotation: to a certain saturation a little bit get drunk from root-lemma piti past event

5 FSA Architecture Mapping of morph-groups to DFSAs (Mealy or Moore machine): 0 č p š v root (-index 1 5 i t e 2 3 a 4 v root )-index 0 n p v pref (-index 1 3 a o 2 4 v pref )-index asp v pref )-index asp o 8 v suf )-index pres 1st pl m 2 v suf )-index pres 1st sg š 3 v suf )-index pres 2st sg 0 ε t j 1 4 v suf )-index pres 3rd sg e 5 v suf )-index pres 2nd pl v suf (-index 6 v suf )-index 2nd sg imper u 7 v suf )-index pres 3rd pl

6 FSA Architecture Mapping ambiguity on emission: emission tuple 1 to n Label DFSAs with variable names Use rules referring to variable names for modeling of morphotactic regularities: verbaspectprefs*. verbatiroots. verbinflsuf

7 FSA Architecture Generating potentially cyclic DFSAs: m š v suf )-index pres 1st sg o v suf )-index pres 2st sg 19 v suf )-index pres 1st p p 0 3 n ε ε o 4 ε v pref )-index asp 5 č p š v root (-index 6 8 i t e 7 9 a 10 v root )-index ε 11 ε t j v suf )-index pres 3rd sg e 16 v suf )-index pres 2nd v pref (-index 1 a 2 ε v pref )-index asp v suf (-index 17 u v suf )-index 2nd sg imper 18 v suf )-index pres 3rd

8 FSA Architecture Ambiguity mapped on emission tuple: e 4 [ v suff )-index; adj suff ) 0 g [ v root (-index; adj root (-index ] 1 o 2 r 3 [ v root )-index; adj root )-index ] [ v suff (-index; adj suff (-index ]

9 FSA Architecture Lemmatization as a rule: Rightmost root is the semantic head Root-lemma: generate canonical word-form from the right-most root neprijatelja ne + prijatelj + a NEG + N-root + ACC not friend = enemy friend not compositional! but useful for semantic field analysis! root-lemma: neprijatelja prijatelj Stem/base-lemma: generate canonical word-form from the stem without inflectional suffixes base-lemma: neprijatelja neprijatelj

10 FSA Architecture Lemmatization (Hack): emission of byte-offset for suffix-elimination pointer to suffix string Clean solution: g:(g, (FV,...)) 0 1 o:(o, ()) 2 r:(r, ()) 3 e:(a, (FV,...)) 4

11 Implementation C++ wrapper for final application Ragel code (automaton definition) generated from morpheme DBs and rules, with associated feature bundles (extended version of Ragel, ( V. 6.1) for handling ambiguity via introduction of multiple emission symbols = emission tuples) Ragel generated C code (jump-code) Morpheme tables Rules Ragel code Code Code DOT Binary

12 Implementation Emission (feature bundles): as one bit-vector Features mapped from the General Ontology for Linguistic Description (upper ontology) possibility: reasoning over linguistic concepts and features Optimization: mapping of concepts and their relations on a compressed bit-vector, maintaining inheritance and implicatures top-node concept sub-class terminal-classes

13 Hardware: dual core 2.4 GHz Lexical base: 120,000 morphemes (and allomorphs) Speed: approx. 50,000 tokens per second with average morpheme count of 2.5 per token Size: binary footprint approx. 5 MB Compilation (tables Ragel + C; Ragel C + DOT; gcc bin): approx. 5 minutes, min. 4 GB RAM for monolithic architecture

14 Interoperability issues addressed: GOLD platform independent code code-page independence Extensible (turnaround time of some minutes) Minimally invasive and minimalistic Open-source

Introduction to Computers & Programming

16.070 Introduction to Computers & Programming Theory of computation: What is a computer? FSM, Automata Prof. Kristina Lundqvist Dept. of Aero/Astro, MIT Models of Computation What is a computer? If you