Computability and Complexity

Computability and Complexity Rewriting Systems and Chomsky Grammars CAS 705 Ryszard Janicki Department of Computing and Software McMaster University Hamilton, Ontario, Canada janicki@mcmaster.ca Ryszard Janicki Computability and Complexity 1 / 48

Rewriting Systems: Idea Another view of computation. Each program instruction also rewrite one string into another! Ryszard Janicki Computability and Complexity 2 / 48

Rewriting Systems Denition A rewriting system is a tuple: S = (Σ, P), where: Σ is an alphabet (nite), P Σ Σ is a set of productions (nite). A production (u, v) P is usually written as u P v or u v, if P is clear from the context. Ryszard Janicki Computability and Complexity 3 / 48

Most Known Rewriting Systems Thue and semi-thue systems ( 1900) Post systems ( 1920-1940) Markov algorithms (early 1950) Chomsky grammars ( 1958) context-free context-sensitive monotone general Lindenmayer systems (early 1970) Ryszard Janicki Computability and Complexity 4 / 48

Thue and semi-thue systems Denition A rewriting system S = (Σ, P) is a semi-thue system, if the following rewriting relation (rule) is used: w 1 P w 2 u v P. w 1 = xuy w 2 = xvy A sequence w 1, w 2,..., w n such that w i P w i+1 is called a derivation. The relation = P i=0 i (reexive and transitive closure of P ) is called a derivation relation. P A semi-thue system (Σ, P) is called a Thue system if P is symmetric. Sometimes, a semi-thue system is denes as S = (Σ, P, σ), where σ Σ + plays the role of initial string. Ryszard Janicki Computability and Complexity 5 / 48

Basic Problems and Example Example Basic problems for rewriting systems: the word problem: let x, y Σ, is it true that x y? the characterization problem: for which x, y Σ, x y holds? Σ = {A, C, E, I, L, M, N, O, P, R, T, W } P = {EAT AT, AT EAT, ATE A, A ATE, LATER LOW, LOW LATER, PAN PILLOW, PILLOW PAN, CARP ME, ME CARP} Ryszard Janicki Computability and Complexity 6 / 48

Context-Free Grammars: Intuitions The original motivation for context-free grammars was a description of a natural language, in particular, English. < sentence > < noun phrase >< verb phrase > < noun phrase > < adjective >< noun phrase > < noun phrase > < noun > < noun > boy < noun > girl < adjective > little < adjective > big < sentence > < noun phrase >< verb phrase > < adjective >< noun phrase >< verb phrase > big < noun phrase >< verb phrase > big < noun >< verb phrase > big girl < verb phrase >... Ryszard Janicki Computability and Complexity 7 / 48

< sentence > < noun-phrase >< verb-phrase > < noun-phrase > < cmplx-noun > < cmplx-noun >< prep-phrase > < verb-phrase > < cmplx-verb > < cmplx-verb >< prep-phrase > < prep-phrase > < prep >< cmplx-noun > < cmplx-noun > < article >< noun > < cmplx-verb > < verb >< verb >< noun-phrase > < article > a the < noun > boy girl ower < verb > touches likes sees < prep > with One can easily derive: a boy sees the boy sees a ower a girl with a ower likes a boy a ower with a boy sees a girl < s > < n-p >< v-p > < cmplx-n >< v-p > < article >< n >< v-p > a < n >< v-p > a boy < v-p > a boy < cmplx-v > a boy < verb > a boy sees Ryszard Janicki Computability and Complexity 8 / 48

Context-Free Grammars: Example < expr > < expr > + < expr > < expr > < expr > < expr > < expr > (< expr >) < expr > a < expr > b < expr > c < expr > < expr > < expr > (< expr >) < expr > (< expr >) a (< expr > + < expr >) a (< expr > +b) a (c + b) a Ryszard Janicki Computability and Complexity 9 / 48

Chomsky Grammars Denition A (Chomsky) grammar is a 4-tuple: G = (V, T, P, σ), where: V is a nite set of variables (non-terminals), T is a nite set of terminals, V T =, σ V is the start symbol (axiom), P (V T ) + (V T ) is a nite set of productions. For every (x, y) P we shall rather write x y. Ryszard Janicki Computability and Complexity 10 / 48

Direct Derivation, Derivation and Language Denition Let G = (V, T, P, σ) be a grammar. A direct derivation is a relation (V T ) (V T ) dened as follows, for all u, w (V T ) u w def. x y P. z, v (V T ). u = zxv w = zyv. A relation = i=0 i, or u w u = w ( u 1,..., u m, m > 0. u u 1... u m w), is called a derivation. Denition Let G = (V, T, P, σ) be a grammar. The set of sequences; L(G) = {x σ x x T } is said to be the language generated by G. Ryszard Janicki Computability and Complexity 11 / 48

Denition (Right-Linear Grammar) A grammar G = (V, T, P, σ) is called right-linear (or regular) if every production satises: x y P = ( A, B V. z T. x = A y = zb) ( A, B V. z T. x = A y = z) Let L RL denotes the set of all languages generated by right linear grammars. Denition (Strict Right-Linear Grammar) A grammar G = (V, T, P, σ) is called right-linear (or regular) if every production satises: x y P = ( A, B V. z T {ε}. x = A y = zb) ( A, B V. z T. x = A y = z) Let L RL denotes the set of all languages generated by right linear grammars. Ryszard Janicki Computability and Complexity 12 / 48

Right-Linear Grammars Proposition For every right linear grammar G, there exists a strictly right linear grammar G such that L(G) = L(G ). Proof. Note that: A a 1 a 2... a k B A a 1 A 1 A 1 a 2 A 2... A k 1 a k B which ends the proof. Ryszard Janicki Computability and Complexity 13 / 48

Context-Free and Context-Sensitive Grammars Denition A grammar G = (V, T, P, σ) is called context-free (or CF-grammar) if every production satises: x y P = A V. x = A Let L CF denotes the set of all languages generated by context-free grammars. Denition A grammar G = (V, T, P, σ) is called context-sensitive (or CS-grammar) if and only if production satises: x ε P = x = σ x y P. # σ (y) = 0 x y y ε P = u, v, t (V T ). A V. x = uav y = utv t 1 Let L CS denotes the set of all languages generated by context-sensitive grammars. Ryszard Janicki Computability and Complexity 14 / 48

Monotone Grammars Denition A grammar G = (V, T, P, σ) is called monotone if and only if production satises: x ε P = x = σ x y P. # σ (y) = 0 x y y ε P = x y Let L M denotes the set of all languages generated by monotone grammars. Corollary 1 Every context-sensitive grammar is monotone. 2 Not every context-free grammar is monotone. Ryszard Janicki Computability and Complexity 15 / 48

Right-Linear Grammars and Automata Theorem Proof. (sketch). L RG = L R. Main idea: σ of a grammar correspond to the initial state of an automaton, and Ryszard Janicki Computability and Complexity 16 / 48

Regular Languages vs Context-Free Languages Theorem Every regular language is context-free but not vice versa, i.e. L R L CF. Proof. Clearly every right linear grammar is context-free, i.e. L R L R. Consider the following context-free grammar: σ ε aσb It generates the language {a n b n n 0} which is not regular. Ryszard Janicki Computability and Complexity 17 / 48

Special Form of Monotone Grammars Lemma For every monotone grammar G there is a monotone grammar G such that every production of G is either of the form A 1... A n B 1... B m where A i, B i are non-terminals, or A a where A is a non-terminal and a is terminal. Proof. (sketch). aabab BAaaBB in G C a ABC a C c BAC a C a BB C a a in G C b b Ryszard Janicki Computability and Complexity 18 / 48

Context-Free Languages vs Monotone Languages Theorem For every monotone grammar G there exists a context-free grammar G such that: L(G) = L(G ). Proof. (sketch). Recall that for the context-free grammars we have the pattern: uaw utw where t 1. The idea of the construction is the following: 1 2 3 4 5 1 2 3 4 5 6 7 8 ABABC BCBACBAC in G ABABC C 1 BABC C 1 BABC C 1 C 2 ABC C 1 C 2 ABC C 1 C 2 C 3 BC in G C 1 C 2 C 3 BC C 1 C 2 C 3 C 4 C C 1 C 2 C 3 C 4 C C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 Ryszard Janicki Computability and Complexity 19 / 48

Proof. (sketch, continuation). C 1 C 2 C 3 C 4 C C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 C 1 C 2 C 3 C 4 C 5 C 6 C 7 C 8 BC 2 C 3 C 4 C 5 C 6 C 7 C 8 BC 2 C 3 C 4 C 5 C 6 C 7 C 8 BC C 3 C 4 C 5 C 6 C 7 C 8 in G BC C 3 C 4 C 5 C 6 C 7 C 8 BC BC 4 C 5 C 6 C 7 C 8 BCBC 4 C 5 C 6 C 7 C 8 BCBAC 5 C 6 C 7 C 8 BCBAC 5 C 6 C 7 C 8 BCBAC C 6 C 7 C 8 BCBAC C 6 C 7 C 8 BCBAC BC 7 C 8 BCBACBC 7 C 8 BCBACBAC 8 BCBACBAC 8 BCBACBAC The above construction can easily be extended to the general case. Corollary L CF = L M. Ryszard Janicki Computability and Complexity 20 / 48

Chomsky Hierarchy Theorem L RG = L R L CF L CS = L M L G = L TM, where L TM is the set of all languages recognized by Turing Machines. The `red' relations have not been proven yet. Ryszard Janicki Computability and Complexity 21 / 48

CF-Grammars: Left-Most and Right-Most Derivations Consider the following CF-grammar: E E E E + E E E (E) a b c It is a shorthand for a more formal denition: V = {E}, σ = E, T = {a, b, c, +,,, (, ), } and P = {E E E, E E + E, E E E, E (E), E a, E b, E c}. Derivation: E E E (E) E (E) c (E + E) c (E + b) c (a + b) c Left-Most Derivation: E E E (E) E (E + E) E (a + E) E (a + b) E (a + b) c Right-Most Derivation: E E E E c (E) c (E + E) c (E + b) c (a + b) c Ryszard Janicki Computability and Complexity 22 / 48

Derivation Tree The following left-most derivation E E E (E) E (E + E) E (a + E) E (a + b) E (a + b) c is equivalent to the below derivation tree: Ryszard Janicki Computability and Complexity 23 / 48

Derivations and Derivation Trees Theorem Let G = (V, T, σ, P) be a CF-grammar. Then we have: 1 σ x there is a derivation tree in G that yields x, 2 σ x σ L x, 3 σ x σ R x, where L is a left-most derivation and R is a right-most derivation. Proof. By induction on the length of x. Ryszard Janicki Computability and Complexity 24 / 48

Ambiguity: Intuition (1) Consider the following grammar: E E + E E E a b c It has the following two dierent left-most derivations: 1 E E E E + E E a + E E a + b E a + b c 2 E E + E a + E a + E E a + b E a + b c The derivation (1) yields to the left tree below while the derivation (2) yields to the right tree below: (a + b) c a + (b c) PROBLEM! Ryszard Janicki Computability and Complexity 25 / 48

Ambiguity: Example of a Fix A possible x for the grammar from previous page: σ A + σ A σ A A a b c There is only one left-most derivation of the string a + b c, namely: σ A + σ a + σ a + A σ a + b σ a + b A a + b c a + (b c) Ryszard Janicki Computability and Complexity 26 / 48

Ambiguity and Inherent Ambiguity Denition A grammar G is ambiguous if the exists x L(G) such that are two dierent left-most derivations yielding x. We can replace left-most derivations with right-most derivations or derivation trees. A grammar G is unambiguous if it is not ambiguous. A context-free language L is inherently ambiguous if every grammar G such that L(G) = L is ambiguous. Proposition The language L = {a n b n c m d m n 1, m 1} {a n b m c m d n n 1, m 1} is inherently ambiguous. Ryszard Janicki Computability and Complexity 27 / 48

Building Better Grammars An arbitrary context-free grammar may have: useless symbols ε-productions of the form A ε unit productions of the form A B Ryszard Janicki Computability and Complexity 28 / 48

Useless Symbols σ aab aσ BaB a A aσa Ba b B BB BBa C aσa BcB a Type A The symbol B is useless since (B x) for all x T, so we cannot get rid of B. Type B The symbol C is useless since σ x implies C is not in x, so C cannot be generated starting from σ. The order of removing useless symbols is important! Type A symbols must be erased rst. Ryszard Janicki Computability and Complexity 29 / 48

Erasing Useless Symbols: Example σ AB σ a A a 1 Erase type A, i.e. the variable B σ a A a 2 Erase type B, i.e. the variable A σ a 1 Erase type B, i.e. none! σ AB σ a A a 2 Erase type A, i.e. the variable B σ a A a Ryszard Janicki Computability and Complexity 30 / 48

ε-productions σ σaa Aa aba A Ab aa ε ab B aa b ε σ σaa Aa aba a aa σa A Ab aa ab a b B aa b a Ryszard Janicki Computability and Complexity 31 / 48

ε-productions: Removal Intuition Rule: ε's are replaced with consequences of their applications. σ abab acc... A ε... B ε... C AB... σ abab = σ abbb aab abb ab σ acc = σ acc ac Ryszard Janicki Computability and Complexity 32 / 48

Unit Productions: Removal Intuition σ aaσb aba A aab a B B ab Ba b σ aaσb aba A aab a B ab Ba b B ab Ba b Ryszard Janicki Computability and Complexity 33 / 48

Better Grammars We can always get rid of: useless symbols ε-productions if ε / L(G) unit productions Ryszard Janicki Computability and Complexity 34 / 48

Useless Symbols: Formal Denition Denition Let G = (V, T, P, σ) be a CF-grammar. A symbol α V T is useful is there is a derivation: σ uαv w for some u, v (V T ) and w T. Otherwise α is useless. There are two aspects of usefulness: some terminal must be derivable from α α must occur in some string derivable from σ Ryszard Janicki Computability and Complexity 35 / 48

Removing Useless Symbols of Type A Lemma (A) Let G = (V, T, P, σ) be a CF-grammar. We can construct G = (V, T, P, σ) such that L(G) = L(G ) and A V. w T. A w. Proof. 1 If A x P and w T then A V. 2 If A x 1... x n P and x i T V then A V. We can repeat (2) until all possibilities are exhausted. Then P = {A x A V A x P}. Ryszard Janicki Computability and Complexity 36 / 48

Removing Useless Symbols of Type B Lemma (B) Let G = (V, T, P, σ) be a CF-grammar. We can construct G = (V, T, P, σ) such that L(G) = L(G ) and α V T. x, y (V T ). σ xαy. Proof. Clearly σ V. If A V and A x 1... x n P, then x i V = x i V and x i T = x i T. We repeat it until all cases are considered. Then P = {A x A V A x P x (V T ) }. Ryszard Janicki Computability and Complexity 37 / 48

Removing Useless Symbols of Type A+B Theorem After Lemma A followed by Lemma B, the new gramar does not have useless symbols. Note the the order Lemma A rst and Lemma B second is important. See page 30. Ryszard Janicki Computability and Complexity 38 / 48

Removing ε-productions Theorem Let G = (V, T, P, σ) be a CF-grammar. There is a CF-grammar G with no useless symbols and no ε-productions such that L(G ) = L(G) \ {ε}. Proof. (sketch). We remove all useless symbols rst. Then: (a) We add a new start variable σ 0 and σ 0 σ. (b) We remove A ε and 1 each B u 1 Au 2 A... Au n is replaced by B u 1 Ãu 2 Ã... Ãu n, where à is either A or ε. All cases must be included. For example: B uavaw is replaced by B uavaw uvaw uavw uvw. 2 B A is replaces by B A ε unless B ε was previously removed We repeat (1) and (2) until all ε-productions are removed. Ryszard Janicki Computability and Complexity 39 / 48

Removing Unit Productions Theorem Every context-free language without ε can be dened by a CF-grammar with no useless symbols, ε-productions, or unit productions. Proof. (sketch). Let G = (V, T, P, σ) be a CF-grammar such that ε / L(G) and G has no useless symbols or ε-productions. Dene G = (V, T, P, σ) where P = {A x P x / V } {A x B x P. x / V A B}, where means G derivation in the grammar G. Note that G does not have any unit production and L(G ) = L(G). Less formally the above construction can be described as follows: We remove A B and whenever B u appears, we add A u unless it was a unit production previously removed. We repeat it until all unit productions are eliminated. Ryszard Janicki Computability and Complexity 40 / 48

Chomsky Normal Form Denition A CF-grammar is in Chomsky Normal Form if each production is of the form A BC or A a, where A, B, C V and a T. Theorem Every CF-language without ε can be generated by a CF-grammar in Chomsky Normal Form. Proof. (sketch). Let G be a grammar that generates a given CF-language. We may assume that G does not have any useless symbols, ε and unit productions. We replace A u 1 u 2... u k, where u i V T, by A u 1 A 1 A 1 u 2 A 2 A i are new variables......... A k 2 u k1 u k We replace any u T by a new variable U and add U u. Ryszard Janicki Computability and Complexity 41 / 48

G : σ ba ab A baa aσ a = B abb bσ b σ ba implies σ C b A, C b b σ ab implies σ C a B, C a a A baa implies A C b C AA, C AA AA A aσ implies A C a σ, A a implies A a, = B abb implies B C a C BB, C BB BB B bσ implies B C b σ, B b implies B b, G : σ C b A C a B A C b C AA C a σ a B C a C BB C b σ b C AA AA C BB BB Ryszard Janicki Computability and Complexity 42 / 48

Greibach Normal Form Denition A CF-grammar is in Greibach Normal Form if each production is of the form A aa 1... A n or A a, where A, A i V and a T. Theorem Every CF-language without ε can be generated by a CF-grammar in Greibach Normal Form. Ryszard Janicki Computability and Complexity 43 / 48

Proof. (sketch). Let L be a CF-language and G = (V, T, P, σ) be a CF-grammar such that L = L(G). Assume G is in Chomsky Normal Form and V = {A 1,..., A m }. 1 We can eectively dene a new grammar G such that L(G ) = L(G) = L, and every production of G is of the form: A i A j x and i j or A i a. Note that if L = then there exists at least one k such that there is a production: A k A k x. 2 Every A k A k x can be replaced by the productions: B k x B k xb k A k αb k, for every A k α, α A k y 3 Now we have only productions of the form (a) A i A j x, i < j (b) A i aγ, a T (c) B i γ, γ (V {B 1,..., B i 1 }) We may now replace (using a nite number of substitutions) the above productions by Greibach productions (see the example next page). Ryszard Janicki Computability and Complexity 44 / 48

From Chomsky to Greibach: an Example A 1 A 2 A 3 A 2 A 3 A 1 b A 3 A 1 A 2 a Step 1. We replace A 3 A 1 A 2 by A 3 A 2 A 3 A 2, and next by A 3 A 3 A 1 A 3 A 2 ba 3 A 2. Clearly A 3 2 A G 3 A 1 A 3 A 2 and A 3 2 ba G 3 A 2. Thus we now have G with P of the form: A 1 A 2 A 3 A 2 A 3 A 1 b A 3 A 3 A 1 A 3 A }{{} 2 α=a 3 γ ba 3 A 2 }{{} α a }{{} α Ryszard Janicki Computability and Complexity 45 / 48

Step 2: The only `troublemaker' is A 3 A 3 A 1 A 3 A 2. It has to be replaced by: B 3 A 1 A 3 A 2 B 3 A 1 A 3 A 2 B 3 A 3 ba 3 A }{{} 2 B 3 α }{{} a α Thus we now have G with P of the form: B 3 A 1 A 2 A 3 A 2 A 3 A 1 b A 3 ba 3 A 2 B 3 ab 3 ba 3 A 2 a B 3 A 1 A 3 A 2 A 1 A 3 A 2 B 3 Note that L(G ) = L(G ) = L(G) = L. For example left A 3 G right A 3 G left left A 3 A 1 A 3 A 2 G A 3 A 1 A 3 A 2 A 1 A 3 A 2 G ba 3 A 1 A 3 A 2 A 1 A 3 A 2, right right ba 3 A 2 B 3 G ba 3 A 2 A 1 A 3 A 2 B 3 ba G 3A 1 A 3 A 2 A 1 A 3 A 2. Ryszard Janicki Computability and Complexity 46 / 48

Step 3: A 3 is now in Greibach Form. A 2 A 3 A 1 is not, so it has to be replaced by: A 2 ba 3 A 2 B 3 A 1 ab 3 A 1 ba 3 A 2 A 2 aa 1 b. A 1 A 2 A 3 is not in Greibach Form, so A 1 ba 3 A 2 B 3 A 1 A 3 ab 3 A 1 A 3 ba 3 A 2 A 1 A 3 aa 1 A 3 ba 3. At this point only B 3 A 1 A 3 A 2 A 1 A 3 A 2 B 3 is left, so: B 3 ba 3 A 2 B 3 A 1 A 3 A 3 A 2 ba 3 A 2 B 3 A 1 A 3 A 3 A 3 A 2 B 3 ab 3 A 1 A 3 A 3 A 2 ab 3 A 1 A 3 A 3 A 2 B 3 ba 3 A 2 A 1 A 3 A 3 A 2 ba 3 A 2 A 1 A 3 A 3 A 2 B 3 aa 1 A 3 A 3 A 2 aa 1 A 3 A 3 A 2 B 3 ba 3 A 3 A 2 ba 3 A 3 A 2 B 3 Ryszard Janicki Computability and Complexity 47 / 48

Popular Trick Greibach Form is only problematic when the rst symbol on the right hand side of a production is a variable like A BaBACc. When the rst symbol on the right hand side of a production is a terminal, for example A bcababa, the construction of a Graibach Form is almost trivial, we just replace each terminal a which is not the rst one by a variable C a and we add a production C a a. For example A bcababa is replaced by A bc c ABC a BA and C a a, C c c. Ryszard Janicki Computability and Complexity 48 / 48