Finite-state Machines: Theory and Applications Unweighted Finite-state Automata Thomas Hanneforth Institut für Linguistik Universität Potsdam December 10, 2008 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 1 / 99
Overview 1 Applications of finite-state machines 2 Finite state acceptors and transducers: formal characterization 3 Closure properties and algebra of finite state acceptors 4 Closure properties and algebra of finite state transducers 5 Equivalence transformations on finite state acceptors 6 Equivalence transformations on finite state transducers 7 Decidability properties of unweighted finite-state acceptors and transducers Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 2 / 99
Outline 1 Applications of Finite-State Machines Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 3 / 99
Some Applications of Finite-State Machines in Computational Linguistics Morphological analysis: lemmatization, word segmentation, segmentation disambiguation Spelling correction Lexicon representation Part-of-Speech-Tagging Shallow parsing, Chunking Speech recognition, speech synthesis Optimality theory Language modeling & Statistical language processing Statistical machine translation Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 5 / 99
Outline 2 Finite state acceptors and transducers: formal characterization Finite-state acceptors Finite-state transducers Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 6 / 99
Finite state acceptors and transducers: formal characterization
Finite-state acceptors Non-deterministic finite-state acceptor Definition (Non-deterministic finite-state acceptor (NFA)) A non-deterministic finite-state acceptor A is a 5-tuple Q, Σ, q 0, F, δ where Q is a non-empty set of states Σ is a non-empty set and called the alphabet of A q 0 Q, the start state F Q, the set of final states δ : Q Σ {ε} 2 Q, the transition function. Notes δ may be a partial function (and usually is) Nondeterminism: the transition function δ maps a state q and an alphabet symbol a to a set of successor states. A transition may be labeled with ε, the neutral element of concatenation. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 8 / 99
Finite-state acceptors Example (Nondeterministic FSA A lex accepting some animal names) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 9 / 99
Finite-state acceptors Deterministic finite-state acceptor Definition (Deterministic finite-state acceptor (DFA)) A deterministic finite-state acceptor A is a 5-tuple Q, Σ, q 0, F, δ, where Q is a non-empty set of states Σ is a non-empty set and called the alphabet of A q 0 Q, the start state F Q, the set of final states δ : Q Σ Q, the transition function. Notes Again, δ may be a partial function, DFSA are by definition ε-free, that is, contain no ε-transitions. DFSA and NDFA have the same generative power that is both concepts are equivalent (cf. subset construction). Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 10 / 99
Finite-state acceptors Example (Deterministic version of A lex ) Deterministic acyclic FSA are also called tries. Tries are useful for lexicon representation. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 11 / 99
Finite-state acceptors Examples Example (DFSA A NP accepting English noun phrase patterns) Example (DFSA A English accepting some English sentences) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 12 / 99
Finite-state acceptors Generalized transition function, language Definition (Generalized transition function δ ) δ is the reflexive and transitive closure of δ δ (q, ε) = q, q Q δ (q, aw) = δ (δ(q, a), w) Definition (Language of a DFSA A) L(A) = {w Σ δ (q 0, w) F } We also say that L(A) is recognized by A. Definition (Regular language) The language is called regular if there exists some DFA which recognizes it. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 13 / 99
Finite-state transducers Definition Definition ((Non-deterministic) finite-state transducer (NFST)) A (non-deterministic) finite-state transducer T is a 7-tuple Q, Σ,, q 0, F, δ, σ, where Q is a non-empty set of states Σ is a non-empty set and called the input alphabet of T is a non-empty set and called the output alphabet of T q 0 Q, the start state F Q, the set of final states δ : Q Σ {ε} 2 Q, the transition function. σ : Q Σ {ε} Q, the output function. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 14 / 99
Finite-state transducers Alternative definition To simplify some definitions, we combine transition and output function to a set of transitions. In addition, we restrict the output function to single symbols or ε. Definition (Normalized finite-state transducer) A normalized finite-state transducer T is a 6-tuple Q, Σ,, q 0, F, E, where Q is a non-empty set of states Σ is a non-empty set, the input alphabet of T is a non-empty set, the output alphabet of T q 0 Q, the start state F Q, the set of final states E Q (Σ {ε}) ( {ε}) Q, the set of transitions. Note that every transducer can be transformed into a normalized transducer. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 15 / 99
Finite-state transducers Example (T lex mapping surface forms to morph. features) Note that fish is nondeterministically mapped to { NOUN sg, NOUN pl} Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 16 / 99
Finite-state transducers Example (Laughter machine T laugh ) The input string laugh is mapped to the infinite set {ha n n 1} Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 17 / 99
Finite-state transducers Example (Bracketing machine T bracket ) Every occurrence of ab is enclosed within brackets. For example, the input string cabbabc is mapped to c{ab}b{ab}c by traversing the state sequence 0 0 2 3 4 0 0 2 3 4 0 0 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 18 / 99
Finite-state transducers Language Definition (Language of a FST) The language L(T ) of a FST T = Q, Σ,, q 0, F, δ, σ is defined in the following way: L(T ) = { u, v δ (q 0, u) F v σ (q 0, u)} δ is recursively defined: δ (q, ε) = {q} and δ (q, wa) = q δ (q,w) δ(q, a) σ is the generalized output function and defined like this: σ (q, ε) = {ε} and σ (q, wa) = σ (q, w) q δ (q,w) σ(q, a, p) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 19 / 99
Finite-state transducers Transduction mapping Definition (Transduction mapping) The transduction mapping T : Σ of a FST T is defined as: T (x) = {y x, y L(T )} Definition (Functional transducer) A transducer T is called functional if T (x) 1 for all x Σ Example T bracket is functional. T lex is not functional. Definition (Ambiguous transducer) A transducer T is called ambiguous, if T (x) > 1 for some x Σ. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 20 / 99
Finite-state transducers Ambiguous transducers Definition (Finitely ambiguous transducer) A transducer T is called finitely ambiguous, if T (x) is finite for all x Σ. Example T lex is finitely ambiguous. Definition (Infinitely ambiguous transducer) A transducer T is called infinitely ambiguous, if T (x) is infinite for some x Σ. Example T laugh is infinitely ambiguous. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 21 / 99
Finite-state transducers Deterministic finite-state transducer Definition (Deterministic finite-state transducer (DFST)) A deterministic finite-state transducer T is a 7-tuple Q, Σ,, q 0, F, δ, σ, where Q is a non-empty set of states Σ is a non-empty set and called the input alphabet of T is a non-empty set and called the output alphabet of T q 0 Q, the start state F Q, the set of final states δ : Q Σ Q, the (deterministic) transition function. σ : Q Σ, the (deterministic) output function. Theorem Every deterministic transducer is functional. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 22 / 99
Outline 3 Closure properties and algebra of finite state acceptors Union Concatenation Star closure Plus closure Reversal Complementation Intersection Difference Substitution Homomorphism Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 23 / 99
Closure properties and algebra of finite state acceptors
Closure properties and algebra of finite state acceptors Definition (Closure of a set) Let S be a set and let f k be a k-ary function taking k-tuples over S as arguments. We say that S is closed under f k if for all a i S f k (a 1, a 2,..., a k ) S. Note Closure properties are important for the modularity based on a specific formalism. They allow to build complex things out of simpler ones by combining them with a number of operations. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 25 / 99
Closure properties and algebra of finite state acceptors Closure properties of regular languages The set of languages which is recognized by finite-state acceptors (the regular languages) is closed under Union Concatenation Plus and star closure Reversal Complementation Intersection Difference Homomorphism and substitution Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 26 / 99
Closure properties and algebra of finite state acceptors Union Example (Union of two acceptors) A 2 A 1 A 1 A 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 27 / 99
Closure properties and algebra of finite state acceptors Concatenation Example (Concatenation of two acceptors) A 1 A 2 A 1 A 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 28 / 99
Closure properties and algebra of finite state acceptors Star closure Example (Star (Kleene) closure of an acceptor) A A Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 29 / 99
Closure properties and algebra of finite state acceptors Plus closure Example (Plus closure of an acceptor) A A + Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 30 / 99
Closure properties and algebra of finite state acceptors Reversal Definition (Reversal of a string) The reversal of a string w Σ denoted by w R is defined as: ɛ R = ɛ Example (a w) R = w R a, a Σ w Σ obama R = bama R o = ama R bo = ma R abo = a R mabo = ɛ amabo Definition (Reversal of a string set) Let S Σ be a set of strings. The reversal of S - denoted by S R is defined as: S R = {w R w S}. Theorem The set of regular languages is closed under reversal. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 31 / 99
Closure properties and algebra of finite state acceptors Reversal Example (Reversal of an acceptor) A A R Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 32 / 99
Closure properties and algebra of finite state acceptors Complementation Given an alphabet Σ and a FSA A you sometimes need a FSA A representing all strings x over Σ which are not in A. Formally: L(A) = {x Σ x L(A)} or L(A) = Σ L(A) L(A) L(A) Theorem The set of regular languages is closed under complementation. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 33 / 99
Closure properties and algebra of finite state acceptors Complementation: algorithm Algorithm: 1 Determinize A and obtain A. 2 Make A complete by adding a sink state s and adding for each state q and each symbol a Σ not already used at q a transition δ(q, a) = s. 3 Exchange final and non-final states. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 34 / 99
Closure properties and algebra of finite state acceptors Complementation Example (Complementation of a finite-state acceptor) A (Σ = {a, c, r, t}) A Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 35 / 99
Closure properties and algebra of finite state acceptors Complementation Example (Why complementation works) Consider a trie for W = {cat, camel, dog, f rog}. Definition (Definition of a trie) Let W be a finite set of words over Σ. Let P ref(w ) the set of all prefixes of W. Define a DFA A = P ref(w ), Σ, ε, W, δ with a Σ, x, xa P ref(w ): δ(x, a) = xa. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 36 / 99
Closure properties and algebra of finite state acceptors Complementation Example (A trie for W = {cat, camel, dog, f rog}) States are labeled with prefixes of W. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 37 / 99
Closure properties and algebra of finite state acceptors Complementation In a trie a special acyclic DFA each state corresponds to a single prefix of a word in W. In a general DFA A, each state q corresponds to a set of prefixes of the words in L(A), the left language of q. Definition (Left language) The left language of a state q symbolically L (q) is defined as: L (q) = {w Σ δ (q 0, w) = q} Example (Left language) L (1) = {a(ba) n n 0} Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 38 / 99
Closure properties and algebra of finite state acceptors Complementation: Why can we only complementize DFAs? Example ( Complementation of an NFA) The complementized NFA still accepts ab. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 39 / 99
Closure properties and algebra of finite state acceptors Complementation: Importance Complementation (negation) is important for the inherent robustness of methods based on finite-state automata. A NLP system is called robust if there does not exist an input string for which it fails. That means: A robust NLP system accepts Σ. This is immediately related to negation: if there is some input string w which is not accepted by some DFA A (say, for example, a DFA representing some NP grammar about stock indices), one could use A to accept w: L(A) L(A) = Σ. This property does not carry over to context-free languages: they are not closed under complementation. Context-sensitive languages are perhaps surprisingly again closed under complementation. But their recognition problem is notoriously difficult. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 40 / 99
Closure properties and algebra of finite state acceptors Complementation: the sting in the tail: complexity Theorem Consider an NFA A with state set Q. The state complexity of an equivalent DFA A can be in the worst case in O( Σ 2 Q ). Example (Worst case of determinization) Consider the regular language L = Σ a(a b) k for some k (with Σ = {a, b}). While an NFA for L has k + 2 states, the equivalent DFA has 2 k+1 states. k = 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 41 / 99
Closure properties and algebra of finite state acceptors Intersection If we know that FSAs are closed under complementation and union then we also know that they are closed under intersection. Why? By DeMorgan! A B A B But this approach is very complex, since it requires three complementation operations which in turn require determinization. There is a more direct method: we let pair of states of the original FSAs be states of the intersection FSA. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 42 / 99
Closure properties and algebra of finite state acceptors Intersection: Example Example (Intersection with the product state construction) A 1 A 2 A 1 A 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 43 / 99
Closure properties and algebra of finite state acceptors Intersection: formal definition Definition (Intersection of two finite-state acceptors) Let A 1 = Q 1, Σ 1, q 01, F 1, δ 1 and A 2 = Q 2, Σ 2, q 02, F 2, δ 2 be two FSAs. A 1 A 2, the intersection of A 1 and A 2 is an acceptor: A = Q 1 Q 2, Σ 1 Σ 2, q 01, q 02, F 1 F 2, δ where p, q δ( p, q, a) if p δ 1 (p, a) and q δ 2 (q, a) for all a Σ 1 Σ 2. This mathematical approach generates in the worst as in the best case a FSA with Q 1 Q 2 states. But a lot of these states may not contribute to the language Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 44 / 99
Closure properties and algebra of finite state acceptors Intersection: useless states Definition (Inaccessible and non-coaccessible states) Let A be a finite-state automaton (acceptor or transducer) with start state q 0. A state q in A is called inaccessible if there is no path in A from q 0 to q. A state q in A is called non-coaccessible if there is no path in A from q to a final state of A. A state is called useless if it is inaccessible or non-coaccessible. A finite-state automaton A is called trim or connected if it has no useless states. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 45 / 99
Closure properties and algebra of finite state acceptors Intersection: removal of useless states Algorithm connect(a) Require: FSM A with start state q 0, state set Q and final state set F Ensure: A without useless states 1: Perform a depth-first search starting at q 0 and mark each visited state 2: Delete each unmarked state q and all its ingoing and outgoing transitions 3: Reverse A 4: Unmark all states in Q 5: Perform a depth-first search starting at all states q F and mark each visited state 6: Delete each unmarked state q and all its ingoing and outgoing transitions 7: Reverse A Complexity of connect(a) If A has Q number of states and E number of transitions, the complexity of connect(a) is in O( Q + E ). Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 46 / 99
Closure properties and algebra of finite state acceptors Intersection: algorithm Require: FSAs A 1 = Q 1, Σ 1, q 01, F 1, δ 1 and A 2 = Q 2, Σ 2, q 02, F 2, δ 2 Ensure: A = A 1 A 2 1: F := Q := 2: ENQUEUE(S, q 01, q 02 ) 3: while S do 4: q 1, q 2 := DEQUEUE(S) 5: for all a Σ 1 Σ 2 do 6: if q 1 δ 1(q 1, a) q 2 δ 2(q 2, a) then 7: δ( q 1, q 2, a) := δ( q 1, q 2, a) { q 1, q 2 } 8: if q 1, q 2 / Q then 9: Q := Q { q 1, q 2 } 10: if q 1 F 1 q 2 F 2 then 11: F := F { q 1, q 2 } 12: end if 13: ENQUEUE(S, q 1, q 2 ) 14: end if 15: end if 16: end for 17: end while 18: CONNECT (A) 19: return A Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 47 / 99
Closure properties and algebra of finite state acceptors Intersection: practical importance Closure under intersection means that we can develop constraints independently of each other and then enforce their validity simultaneously by intersecting them A lot of finite-state based NLP is based on intersection: (Two-level-) Morphology, Constraint based grammar, pattern matching etc. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 48 / 99
Closure properties and algebra of finite state acceptors Difference Definition (Difference) Let A 1 and A 2 two FSAs. The difference A 1 A 2 is defined as: A 1 A 2 A 1 A 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 49 / 99
Closure properties and algebra of finite state acceptors Difference Example (Difference) A 1 A 2 A 1 A 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 50 / 99
Closure properties and algebra of finite state acceptors Substitution Definition (Substitution) A substitution is a mapping s : Σ 2 for two alphabets Σ and. s is generalized to s : Σ 2 by: s (ε) = ε s (xa) = s (x)s(a) Theorem (Closure under substitution) The set of regular languages is closed under substitution with regular languages. Note A lot of finite-state based NLP is based on closure under substitution. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 51 / 99
Closure properties and algebra of finite state acceptors Substitution Example (Substitution in computational morphology) A morphology rule as a FSA A A 1 A 2 Result of the substitution A{NST EM = A 1, NINF L = A 2 } Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 52 / 99
Closure properties and algebra of finite state acceptors Homomorphism Definition (Homomorphism) A homomorphism is a mapping h : Σ for two alphabets Σ and. Definition (Inverse homomorphism) Given a homomorphism h, the inverse homomorphic image h 1 of a language L is defined as: h 1 (L) = {w h(w) L} Theorem (Closure under homomorphism) The set of regular languages is closed under homomorphism and inverse homomorphism Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 53 / 99
Outline 4 Closure properties and algebra of finite state transducers Projection Composition Cross product Inversion Intersection Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 54 / 99
Closure properties and algebra of finite state transducers
Closure properties and algebra of finite state transducers The set of finite state transducers is closed under Union Concatenation Closure Reversal Projection (note that this leads to FSAs) Composition Inversion Finite state transducers are not closed under Complementation Intersection (but acyclic and ε-free transducers are) Difference Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 56 / 99
Closure properties and algebra of finite state transducers Projection Definition (First and second projection) Let T = Q, Σ,, q 0, F, S be a transducer. The first projection of T symbolically π 1 (T ) is the FSA A = Q, Σ, q 0, F, δ where a Σ {ε}, δ(p, a) = {q b : p, a, b, q S} The second projection of T symbolically π 2 (T ) is the FSA A = Q, Σ, q 0, F, δ where b {ε}, δ(p, b) = {q a Σ : p, a, b, q S} Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 57 / 99
Closure properties and algebra of finite state transducers Projection Example (Projection) Transducer T π 1 (T ) π 2 (T ) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 58 / 99
Closure properties and algebra of finite state transducers Composition Composing a transducer T 1 with a transducer T 2 (formally T 1 T 2 ) means: take some input u for T 1, collect the output v of T 1, feed it as input into T 2 and collect the output w of T 2. Definition (Composition relation) Let T 1 = Q 1, Σ 1, 1, q 01, F 1, S 1 and T 2 = Q 2, Σ 2, 2, q 02, F 2, S 2 be transducers. L(T 1 T 2 ) = { u, w Σ 1 2 v 1 Σ 2 : u, v L(T 1) v, w L(T 2 )} Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 59 / 99
Closure properties and algebra of finite state transducers Composition Definition (ε-free composition) Let T 1 = Q 1, Σ 1, 1, q 01, F 1, E 1 and T 2 = Q 2, Σ 2, 2, q 02, F 2, E 2 be two normalized, ε-free FSTs. T 1 T 2, the composition of T 1 and T 2, is the transducer T = Q 1 Q 2, Σ 1, 2, q 01, q 02, F 1 F 2, E where E = { p, q, a, b, p, q c 1 Σ 2 : p, a, c, p E 1 q, c, b, q E 2 } Properties of composition The composition operation is not commutative, that is, in general: T 1 T 2 T 2 T 1 The composition operation is associative, that is: T 1 T 2 T 3 = (T 1 T 2 ) T 3 = T 1 (T 2 T 3 ) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 60 / 99
Closure properties and algebra of finite state transducers Composition How does composition work? Whenever T 1 contains a transition: and T 2 contains a transition: T will contain a transition: Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 61 / 99
Closure properties and algebra of finite state transducers Composition Example (Composition) FST mapping NP-patterns to = FST repeatedly mapping NP category words to their categories Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 62 / 99
Closure properties and algebra of finite state transducers Application Definition (Identity transducer) Let A = Q, Σ, q 0, F, δ be a FSA. The identity transducer ID(A) is defined by Q, Σ, Σ, q 0, F, E where E = { p, a, a, q p, q Q, a Σ {ε} : q δ(p, a)} Example (Identity transducer) Definition (Application) The application of a FST T to a FSA A symbolically T [A] is defined as T [A] π 2 (ID(A) T ) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 63 / 99
Closure properties and algebra of finite state transducers Application Example FSA A FSA ID(A) FST T ID(A) T T [A] = π 2 (ID(A) T ) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 64 / 99
Closure properties and algebra of finite state transducers Composition: relationship to intersection Composition can be considered as a generalization of intersection. The intersection of two FSAs A 1 and A 2 can be defined as follows: A 1 A 2 = π 1 (ID(A 1 ) ID(A 2 )) So, intersecting two FSAs is done by composing their identity transducers and afterwards projecting one of the tapes. Composing two transducers X and Y means synchronizing (intersecting) their inner tapes and then combining the outer tapes: Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 65 / 99
Closure properties and algebra of finite state transducers Composition: handling ε-transitions It is possible to generalize the composition definition to transducers with ɛ-transitions: Definition (Transducer composition) Let T 1 = Q 1, Σ 1, 1, q 01, F 1, E 1 and T 2 = Q 2, Σ 2, 2, q 02, F 2, E 2 be two normalized FSTs. T 1 T 2, the composition of T 1 and T 2, is the transducer T = Q 1 Q 2, Σ 1, 2, q 01, q 02, F 1 F 2, E E ε E i,ε E o,ε where 1 E = { p, q, a, b, p, q c 1 Σ 2 : p, a, c, p E 1 q, c, b, q E 2 } 2 E ε = { p, q, a, b, p, q p, a, ε, p E 1 q, ε, b, q E 2 } 3 E i,ε = { p, q, ε, a, p, q q, ε, a, q E 2 p Q 1 } 4 E o,ε = { p, q, a, ε, p, q p, a, ε, p E 1 q Q 2 } Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 66 / 99
Closure properties and algebra of finite state transducers Composition: handling ε-transitions There are four different ways, how ε and alphabet symbols on the second tape of T 1 and the first tape of T 2 can interact: 1 T 1 contains a a : c-transition and T 2 contains a c : b-transition: this is handled in the same way as in the ε-free case 2 T 1 contains a a : ε-transition and T 2 contains a ε : b-transition T contains a a : b-transition. That is: ε is treated as a regular symbol. 3 T 1 stays in the same state, T 2 moves on: 4 T 1 moves on, T 2 stays in the same state: Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 67 / 99
Closure properties and algebra of WFSA Composition Example (Composition of two unweighted FSTs) T 1 T 2 T 1 T 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 68 / 99
Closure properties and algebra of finite state transducers Composition: Application Composition is a very important operation for building processing or filtering cascades, for example in robust parsing and morphological analysis. Since composition is not commutative, the order of a transducer cascade C = T 1 T 2... T k matters. This may lead to problems related to feeding, counter-feeding, bleeding and counter-bleeding. Since the composition operation is associative, the order in which the compositions in C are computed does not matter. This entails some freedom degrees for implementing such cascades. Note, that the state complexity of T 1 T 2... T k is Q 1 Q 2... Q k in the worst case. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 69 / 99
Closure properties and algebra of finite state transducers Cross product Definition ((Cartesian) Product) Given two sets S 1 and S 2, the Cartesian product S 1 S 2 is defined as: Theorem (Product of regular sets) S 1 S 2 = { x, y x S 1 y S 2 } Let A 1 = Q 1, Σ 1, q 01, F 1, δ 1 and A 2 = Q 2, Σ 2, q 02, F 2, δ 2 be two finite-state acceptors. Then L(A 1 ) L(A 2 ) is representable by a finite-state transducer A 1 A 2. Proof. A 1 A 2 ID(A 1 ) T Σ 1 ε T ε Σ 2 ID(A 2 ) Note Cross product is the core of all replacement operations. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 70 / 99
Closure properties and algebra of finite state transducers Cross product Example (Cross product) A 1 A 2 T Σ 1 ε T ε Σ 2 A 1 A 2 Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 71 / 99
Closure properties and algebra of finite state transducers Inversion Definition (Inversion) Let T = Q, Σ,, q 0, F, E be a transducer. The inversion of T symbolically T 1 is the FST T 1 = Q,, Σ, q 0, F, E 1 where E 1 = { p, b, a, q p, a, b, q E} Note Thus, inversion simply exchanges input- and output tapes of a transducer. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 72 / 99
Closure properties and algebra of finite state transducers Inversion Example (Morphological analysis vs. generation) FST T Morph mapping words to morphological categories FST T 1 Morph mapping morphological categories to words Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 73 / 99
Closure properties and algebra of finite state transducers Why are FSTs not closed under intersection? Example T an b n c T an b c n The intersection of T a n b n c and T a n b cn would result in the relation R = { a n, b n c n n 0} which is not regular and thus not representable by a finite-state transducer. Note This has consequences for creating applications based on finite-state transducers. They cannot be based on the intersection of constraints represented as transducers. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 74 / 99
Closure properties and algebra of finite state transducers Why are FSTs not closed under intersection? Intuitively, the existence of ε within loops leading to infinite ambiguity is the reason why FSTs are not closed under intersection Thus, ε-free FSTs also called equal-length transducers are closed under intersection The same is true for acyclic FSTs, where we have some freedom where to realize the ε-transitions By DeMorgan, non-closure under intersection leads to non-closure under complementation Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 75 / 99
Outline 5 Equivalence transformations on finite-state acceptors ε-removal Determinization Minimization Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 76 / 99
Equivalence transformations on finite-state acceptors
Equivalence transformations on finite-state acceptors Equivalence transformations Equivalence transformations are operations on automata which change the topology of an automaton without changing its language. They usually serve optimization purposes, that is, they create smaller and/or faster automata. Finite-state acceptors admit the following equivalence transformations: ε-removal Determinization Minimization Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 78 / 99
Equivalence transformations on finite-state acceptors ε-removal Definition ( ) Let p and q be states in Q and let w be a string in Σ. Let w be a relation Q Q, such that p, q w if there is a path labeled with w from p to q. Definition (ε-closure) Given a NFA A = Q, Σ, q 0, F, δ, ε-closure(q) = {q} {p Q q ε p}. Definition (ε-free FSA) Let A = Q, Σ, q 0, F, δ be a FSA. Define A, the equivalent ε-free FSA with L(A ) = L(A), as A = Q, Σ, q 0, F, δ where: δ (q, a) = ε-closure( q ε-closure(q) δ(q, a)), q Q, a Σ F = F {q 0 }, if ε-closure(q 0 ) F, else F = F. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 79 / 99
Equivalence transformations on finite-state acceptors ε-removal Example Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 80 / 99
Equivalence transformations on finite-state acceptors Determinization Definition (Subset construction) Let A = Q, Σ, q 0, F, δ be a FSA. Define A, the equivalent DFA with L(A ) = L(A) as A = 2 Q, Σ, {q 0 }, F, δ with : F = {S Q S F } δ (S, a) = q S δ(q, a), a Σ, S Q Note The complexity of an algorithm which implements this in a naive way is exponential. In the normal case, most of the subset states in the DFA are not accessible / coaccessible. A better algorithm based on a state queue avoids inaccessible states. But this doesn t change the complexity in the worst case. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 81 / 99
Equivalence transformations on finite-state acceptors Determinization Example Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 82 / 99
Equivalence transformations on finite-state acceptors Minimization Definition (Minimal DFA) Let A = Q, Σ, q 0, F, δ be a DFA. L is minimal if A = Q, Σ, q 0, F, δ : L(A ) = L(A) Q Q Definition (Right language) Let A = Q, Σ, q 0, F, δ be a DFA. The right language of a state q Q symbolically L (q) is defined as: L (q) = {w Σ δ (q, w) F } Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 83 / 99
Equivalence transformations on finite-state acceptors Minimization Definition (Equivalent states) Two states p and q are called equivalent if L (p) = L (q). This holistic definition based on right languages can be turned into a recursive definition of equivalence of states: Definition (State equivalence ) Let A = Q, Σ, q 0, F, δ be a DFA. Two states p and q are called equivalent symbolically p q, if: p q if p F q F a Σ : δ(p, a) δ(q, a). Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 84 / 99
Equivalence transformations on finite-state acceptors Minimization Based on state equivalence, we come up with a definition of a minimal DFA: Theorem (Minimal DFA I) Let A = Q, Σ, q 0, F, δ be a DFA. A is minimal iff p, q Q : p q L (p) L (q). By substituting the recursive definition of state equivalence into the last theorem, we arrive at: Theorem (Minimal DFA II) Let A = Q, Σ, q 0, F, δ be a DFA. A is minimal iff p, q Q : p q p F q F a Σ : δ(p, a) δ(q, a). Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 85 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Theorem (Myhill-Nerode) The following propositions are equivalent: 1 L is recognized by a DFA A L = Q, Σ, q 0, F, δ. 2 L is the union of some equivalence classes of a right invariant equivalence relation R with finite index. 3 R L (x R L y iff z Σ : xz L yz L) is of finite index. Notes The Myhill-Nerode theorem links states with subsets of Σ. It is central to the theorem that the number of states in a DFA is finite. The Myhill-Nerode theorem assumes complete DFAs, that is, the corresponding transition function δ is total. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 86 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Definition (Equivalence relation, equivalence class) Let S be a set and E S S a binary relation. E is called a equivalence relation if E is reflexive, symmetric and transitive. If E is a equivalence relation, we call [x] E = {y x E y} the equivalence class of x wrt E. Properties of equivalence relations 1 x [x] E, x S 2 [x] E = [y] E [x] E [y] E =, x, y S 3 [x] E = S x S Definition (Index of a equivalence relation) Let E be a equivalence relation. The index I E of E is the number of E s equivalence classes. E is of finite index if I E is finite. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 87 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Definition (Right-invariant equivalence relation) Let R be a equivalence relation over Σ. R is called right-invariant (with respect to concatenation) if x, y, z Σ : x R y xz R yz Definition (Left language) Let A = Q, Σ, q 0, F, δ be a DFA. Define the left language L (p) of a state p Q as L (p) = {w Σ δ(q 0, w) = p}. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 88 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Myhill-Nerode theorem. We prove the theorem by chaining 1 2, 2 3 and 3 1. 1 2. Let A be a DFA recognizing L. Define R A as x R a y if δ(q 0, x) = δ(q 0, y), x, y Σ. Subproof: R A is right-invariant equivalence relation of finite index (the index of R A is Q ). The equivalence classes of R A are the left languages L (p), p Q. L (q) = L. Example q F L (1) = {ε, a(ba) b}, L (2) = {a(ba) }, L (3) = {a(ba) c(d) } L = L (2) L (3) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 89 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Myhill-Nerode theorem (continued). 2 3. R = R A is an refinement of R L, that is, every equivalence class of R A is contained in some equivalence class of R L. 1 Assume that x R A y. 2 Since R A is right-invariant, xz R A yz, for all z Σ. 3 Thus xz L if and only if yz L. 4 Thus xz R L yz and the equivalence class of x wrt R A is contained in the equivalence class of x wrt R L. 5 Since the index of R A is finite (at most equal to Q ) and the index of R L is less or equal to the index of R A, we conclude that R L is of finite index. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 90 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Example (R A is a refinement of R L ) State Corresponding equiv. class of R A 3 e 9 frie 15 dog 17 doll 19 dollar 22 coll 24 collar State Corresponding equiv. class of R L 7 dog, collar, dollar, end, friend, frog 8 e, frie 12 doll 16 coll Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 91 / 99
Equivalence transformations on finite-state acceptors Minimization: Myhill-Nerode theorem Myhill-Nerode theorem (continued). 3 1. Given R L, construct a new FSA A = Q, Σ, q 0, F, δ as follows: 1 Q = {[x] [x] is a equivalence class of Σ under R L } 2 q 0 = [ε] 3 δ : Q Σ Q : δ ([x], a) = [xa], [x] Q a Σ 4 F = {[x] x L} Example δ ([e], n) = [en] δ ({frie, e}, n) = {frien, en} δ ([en], d) = [end] δ ({frien, en}, d) = {friend, end} a a Inspired by CAKE: Friend is a four-letter word Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 92 / 99
Equivalence transformations on finite-state acceptors Minimization: algorithms Approaches 1 Union-Find-based: Find all states p and q with L (p) = L (q) and merge them. 2 Partition-based: Starting at sets of non-equivalent states, partition these sets further until each set contains only equivalent states. Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 93 / 99
Outline 6 Equivalence transformations on finite-state transducers Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 94 / 99
Equivalence transformations on finite-state transducers
Outline 7 Decidability properties of unweighted finite-state acceptors and transducers Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 96 / 99
Decidability properties of unweighted finite-state acceptors and transducers
Decidability properties of unweighted finite-state acceptors and transducers Given two finite-state acceptors A and A, the following properties are decidable: L(A) = L(A) = Σ L(A) = L(A ) L(A) L(A ) Given two finite-state transducers T and T, the following properties are decidable: T is functional Given two finite-state transducers T and T, the following properties are undecidable: L(T ) = L(T ) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 98 / 99
Version history 16.10.08: version 0.1 (initial version) 20.10.08: version 0.2 (some error corrections, added definition of composition of FSTs with ε-transitions) 09.11.08: version 0.3 (added example for ε-composition, enhanced example for cross product, added subtitles) 07.12.08: version 0.4 (completed minimization section) Thomas Hanneforth (Universität Potsdam) Finite-state Machines: Theory and Applications December 10, 2008 99 / 99