Overview. Regular Expressions and Finite-State. Motivation. Regular expressions. RE syntax Additional functions. Regular languages Properties

Transcription

1 Overview L445/L545/B659 Dept. of Linguistis, Indiana University Spring 2016 languages Finite-state tehnology is: Fast and effiient Useful for a variety of language tasks Three main topis we ll disuss: Expressions (REs) () of Languages languages REs and are mathematially equivalent, but help us approah problems in different ways 1 / 36 2 / 36 Some useful tasks involving language More useful tasks involving language Find all phone numbers in a text, e.g., ourrenes suh as When you all (614) , you reah the fax mahine. Find multiple adjaent ourrenes of the same word in a text, as in I read the the book. Determine the language of the following utterane: Frenh or Polish? Czy pasazer jaday do Warszawy moze jeha przez Londyn? languages Look up the following words in a ditionary: laughs, beame, unidentifiable, Thatherization Determine the part-of-speeh of words like the following, even if you an t find them in the ditionary: onurbation, adene, disproportionality, lyriism, parlane Suh tasks an be addressed using so-alled finite-state mahines (FSMs) We an start to understand FSMs by starting with the pratial: regular languages 3 / 36 4 / 36 The syntax of regular (1) A regular expression is a desription of a set of strings, i.e., a language. They an be used to searh for ourrenes of these strings A variety of unix tools (grep, sed), editors (emas), and programming languages (perl, python) inorporate regular. Just like any other formalism, regular as suh have no linguisti ontents, but they an be used to refer to linguisti units. languages onsist of strings of haraters:, A100, natural language, 30 years! disjuntion: ordinary disjuntion: devoured ate, famil(y ies) harater lasses: [Tt]he, be[oa]me ranges: [A-Z] (a apital letter) negation: [ˆa] (any symbol but a) [ˆA-Z0-9] (not an upperase letter or number) languages 5 / 36 6 / 36

2 The syntax of regular (2) The syntax of regular (3) ounters optionality:? olou?r any number of ourrenes: * (Kleene star) [0-9]* years at least one ourrene: + [0-9]+ dollars wildard for any harater:. beg.n for any harater in between beg and n Parentheses to group items together ant(farm)? Esaped haraters to speify haraters with speial meanings: \*, \+, \?, \(, \), \, \[, \] languages Operator preedene, from highest to lowest: parentheses () ounters * +? harater sequenes disjuntion fire ing = fire or ing fir(e ing) = fir followed by either e or ing Note: The various unix tools and languages differ w.r.t. the exat syntax of the regular they allow. languages 7 / 36 8 / 36 Additional funtionality for some RE uses (1) Additional funtionality for some RE uses (2) Although not a part of our disussion about regular languages, some tools allow for more funtionality Anhors: anhor to various parts of the string ˆ = start of line do not onfuse with [ˆ...] used to express negation $ = end of line \b non-word harater word haraters are digits, undersores, or letters, i.e., [0-9A-Za-z ] languages Use aliases to designate partiular reurrent sets of haraters \d = [0-9]: digit \D = [ˆ\d]: non-digit \w = [a-za-z0-9 ]: alphanumeri \W = [ˆ\w]: non-alphanumeri \s = [\r\t\n\f]: whitespae harater \r: spae, \t: tab, \n: newline, \f: formfeed \S [ˆ\s]: non-whitespae languages 9 / / 36 Some RE pratie Formal language theory What does \$[0-9]+(\.[0-9][0-9]) signify? Write a RE to apture the times on a digital wath (hours and minutes). Think about: the (im)possible values for the hours the (im)possible values for the minutes languages We will view any formal language as a set of strings The language uses a finite voabulary Σ (alled an alphabet), and a set of string-ombining operations languages are the simplest lass of formal languages = lass of languages definable by REs = lass of languages haraterizable by languages 11 / / 36

3 languages of regular languages (1) How an the lass of regular languages (speified by regular ) be haraterized? Let Σ be the set of all symbols of the language, the alphabet, then: 1. {} is a regular language 2. a Σ: {a} is a regular language 3. If L 1 and L 2 are regular languages, so are: 3.1 the onatenation of L 1 and L 2 : L 1 L 2 = {xy x L 1, y L 2 } 3.2 the union of L 1 and L 2 : L 1 L the Kleene losure of L: L = L 0 L 1 L 2... where L i (roughly) represents L onatenated with itself i times, i.e., languages The regular languages are losed under the following (where L 1 and L 2 are regular languages): onatenation: L 1 L 2 set of strings with beginning in L 1 & ontinuation in L 2 Kleene losure: L 1 set of repeated onatenation of a string in L 1 union: L 1 L 2 set of strings in L 1 or in L 2 omplementation: Σ L 1 set of all possible strings that are not in L 1 languages L 0 = {ǫ}, L i+1 = {ab a L i and b L} 13 / / 36 of regular languages (2) What sorts of aren t regular? The regular languages are losed under (L 1 and L 2 regular languages): differene: L 1 L 2 set of strings whih are in L 1 but not in L 2 intersetion: L 1 L 2 set of strings in both L 1 and L 2 reversal: L R 1 set of the reversal of all strings in L 1 languages In natural language, examples inlude enter-embedding onstrutions. These dependenies are not regular: (1) a. The at loves Mozart. b. The at the dog hased loves Mozart.. The at the dog the rat bit hased loves Mozart. d. The at the dog the rat the elephant admired bit hased loves Mozart. (2) (the noun) n (transitive-verb) n 1 loves Mozart languages Similar ones would be regular: (3) A*B* loves Mozart 15 / / 36 Finite state mahines Finite state mahines (or automata) (FSM, FSA) reognize or generate regular languages, exatly those speified by regular. Example: expression: olou?r Finite state mahine (representation): 0 6 o 5 l 4 o 2 r u r languages 1 3 Aepting/Rejeting strings The behavior of an FSA is ompletely determined by its transition table. The assumption is that there is a tape, with the input symbols read off onseutive ells of the tape. The mahine starts in the start (initial) state, about to read the ontents of the first ell on the input tape. The FSA uses the transition table to deide where to go at eah step A string is rejeted in exatly two ases: 1. a transition on an input symbol takes you nowhere 2. the state you re in after proessing the entire input is not an aept (final) state Otherwise, the string is aepted. languages 17 / / 36

4 Defining finite state automata Example FSA A finite state automaton is a quintuple (Q, Σ, E, S, F ) with Q a finite set of states Σ a finite set of symbols, the alphabet S Q the set of start states F Q the set of final states E a set of edges Q (Σ {ǫ}) Q The transition funtion d an be defined as d(q, a) = {q Q (q, a, q ) E} languages FSA to reognize strings of the form: [ab]+ i.e., L = { a, b, ab, ba, aab, bab, aba, bba,... } FSA is defined as: Q = {0, 1} Σ = {a, b} F = {1} E = {(0, a, 1), (0, b, 1), (1, a, 1), (1, b, 1)} languages 19 / / 36 FSA: set of zero or more a s FSA: set of all lowerase alphabeti strings ending in b L = { ǫ, a, aa, aaa, aaaa,... } Q = {0} Σ = {a} F = {0} E = {(0, a, 0)} languages L aptured by [a-z]*b = {b, ab, tb,..., aab, abb,... } Q = {0, 1} Σ = {a, b,,..., z} F = {1} E = {(0, a, 0), (0,, 0), (0, d, 0),..., (0, z, 0), (0, b, 1), (1, b, 1), (1, a, 0), (1,, 0), (1, d, 0),... (1, z, 0)} languages How would we hange this to make it: \b[a-z]*b\b 21 / / 36 FSA: the set of all strings in [ab]* with exatly 2 a s Language aepted by an FSA The extended set of edges Ê Q Σ Q is the smallest set suh that Do this yourself languages (q, σ, q ) E : (q, σ, q ) Ê languages It might help to first rewrite a more preise regular expression for this (q 0, σ 1, q 1 ), (q 1, σ 2, q 2 ) Ê : (q 0, σ 1 σ 2, q 2 ) Ê First, be lear what the domain is (all strings in [ab]*) And then figure out how to narrow it down The language L(A) of a finite state automaton A is defined as L(A) = {w q s S, q f F, (q s, w, q f ) Ê} 23 / / 36

5 FSA for simple NPs Finite state transition networks (FSTN) Where d is an alias for determiners, a for adjetives, and n for nouns: Q = {0, 1, 2} Σ = {d, a, n} F = {2} E = {(0, d, 1), (0, ǫ, 1)(1, a, 1), (1, n, 2), (2, n, 2)} languages Finite state transition networks are graphial desriptions of finite state mahines: nodes represent the states start states are marked with a short arrow final states are indiated by a double irle ars represent the transitions languages 25 / / 36 Example for a finite state transition network Finite state transition tables S0 a S1 S2 b b S3 languages Finite state transition tables are an alternative, textual way of desribing finite state mahines: the rows represent the states start states are marked with a dot after their name final states with a olon the olumns represent the alphabet the fields in the table enode the transitions languages b expression speifying the language generated or aepted by the orresponding FSM: ab b+ 27 / / 36 The example speified as finite state transition table Some properties of finite state mahines a b d S0. S1 S2 S1 S3: S2 S2,S3: S3: languages Reognition problem an be solved in linear time (independent of the size of the automaton). There is an algorithm to transform eah automaton into a unique equivalent automaton with the least number of states. languages 29 / / 36

6 Deterministi Finite State Example: Determinization of FSA A finite state automaton is deterministi iff it has no ǫ transitions and for eah state and eah symbol there is at most one appliable transition. Every non-deterministi automaton an be transformed into a deterministi one: Define new states representing a disjuntion of old states for eah non-determinay whih arises. Define ars for these states orresponding to eah transition whih is defined in the non-deterministi automaton for one of the disjunts in the new state names. languages 1 1 a b a languages b d d {3,5} e a a a e {5,6} 5 e a a a {4,5} 7 7 e, a / / 36 Why finite-state? From to Transduers Finite number of states Number of states bounded in advane determined by its transition table Therefore, the mahine has a limit to the amount of memory it uses. Its behavior at eah stage is based on the transition table, and depends just on the state itõs in, and the input. So, the urrent state reflets the history of the proessing so far. Classes of formal languages whih are not regular require additional memory to keep trak of previous information, e.g., enter-embedding onstrutions languages Needed: mehanism to keep trak of path taken A finite state transduer is a 6-tuple (Q, Σ 1, Σ 2, E, S, F ) with Q a finite set of states Σ 1 a finite set of symbols, the input alphabet Σ 2 a finite set of symbols, the output alphabet S Q the set of start states F Q the set of final states E a set of edges Q (Σ 1 {ǫ}) Q (Σ 2 {ǫ}) languages 33 / / 36 Transduers and determinization A finite state transduer understood as onsuming an input and produing an output annot generally be determinized. Example: a:b a :b b:b 3 a : : languages Summary Notations for haraterizing regular languages: Finite state transition networks Finite state transition tables Finite state mahines and regular languages: Definitions and some properties Finite state transduers languages a: 35 / / 36