Computational Models: Class 3

Computational Models: Class 3 Benny Chor School of Computer Science Tel Aviv University November 2, 2015 Based on slides by Maurice Herlihy, Brown University, and modifications by Iftach Haitner and Yishay Mansour. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 1 / 55

Computational Models, Lecture 3: Plan Two approaches for proving languages are non regular: 1 Pumping Lemma 2 Myhill-Nerode Theorem (not in Sipser s book) Closure properties Algorithmic questions for NFAs Context free grammers Sipser, 1.4, 2.1, 2.2 Hopcroft and Ullman, 3.4 Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 2 / 55

What We Proved Last Time (among other things) Theorem 1 A language is described by a regular expression, iff it is regular. We will now review the construction of the more involved part (if L is accepted by a DFA/NFA, then L can be described by a regular expression), and go over a proof of the correctness of the construction. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 3 / 55

Generalized Non-Deterministic Finite Automata (reminder) We now define generalized non-deterministic finite automata (GNFA). An NFA: A GNFA: Each transition labeled with a symbol or ε, reads zero or one symbols, takes matching transition, if any. Each transition labeled with a regular expression, reads zero or more symbols, takes transition whose regular expression matches string, if any. GNFAs are natural generalization of NFAs. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 4 / 55

A Special Form of GNFA (reminder) We will focus on a special type of GNFA Start state has outgoing arrows to every other state, but no incoming arrows. Unique accept state has incoming arrows from every other state, but no outgoing arrows. Except for start and accept states, an arrow goes from every state to every other state, including itself. Easy to transform any GNFA into special form. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 5 / 55

GNFA Model of Computation (reminder) Definition 2 A GNFA G = (Q, Σ, δ, q s, q a ) accepts a string w Σ, if there exists parsing w = w 1 w 2 w k, where each w i Σ, and r 0,..., r k Q, such that r 0 = q s, r k = q a, and w i L(δ(r i 1, r i )), for every 0 < i k. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 6 / 55

The Construction: DFA/NFA = Regular Expression Strategy sequence of transformations, changing one automata to an equivalent one. Last automata in sequence will be a 2 state special form GNFA (yielding a regular expression). Given an n-state DFA. Transform into an (n + 2)-state GNFA. While GNFA has more than 2 states, transform it into equivalent GNFA with one fewer state. Eventually reach 2-state GNFA (states are just start and accept). Label on single transition is the desired regular expression. Q.: why do we not care if we start from DFA or NFA here? Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 7 / 55

Converting strategy ( =) 3-state DFA 5-state GNFA 4-state GNFA 3-state GNFA 2-state GNFA regular expression Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 8 / 55

Removing One State While there are more than two states, we choose any state q r that differs from the start and accept state. We remove the state q r, and then repair the GNFA by altering regular expression of other transitions. R 4 q i q j R 1 R 3 q r R 2 q i R 1 R* 2 R 3 UR 4 q j This process is done for any ordered pair of states (q i, q j ). Each such alteration corresponds to strings "going from q i to q j directly or via q r. In particular, if i j, we do it separately for (q i, q j ) and (q j, q i ). And if i = j, we do it for (q i, q i ). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 9 / 55

Proof of Correctness (reminder) We showed that if G k is a k-state GNFA which is generated from the GNFA G k+1 by the removal of one state, q r q start, q accept, then L(G k ) = L(G k+1 ). To do that, we showed that for every string w Σ, w L(G k+1 ) = w L(G k ) w L(G k ) = w L(G k+1 ) R 4 q i q j R 1 R 3 q r R 2 q i R 1 R* 2 R 3 UR 4 q j We will not repeat the (rather tedious) proof during today s lecture. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 10 / 55

Summing it up We proved L(G k+1 ) = L(G k ). Likewise, L(G k ) = L(G k 1 ).. Likewise, L(G 3 ) = L(G 2 ). Hence, L(G 2 ) and (the regular expression) and the language of the original DFA/NFA accept the same language. We proved (in lecture 2): A language is described by a regular expression iff it is regular. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 11 / 55

Summing It Up (again) Theorem 3 A language is described by a regular expression, iff it is regular. We have made a lot of progress understanding what finite automata can do. But what can they not do? Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 12 / 55

And Now for Something Somewhat Different: Let Us Be Negative, For a Change Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 13 / 55

Negative Results Is there a DFA that accepting B = {0 n 1 n : n 0} C = {w : # 1 (w) = # 0 (w)} D = {w : # 01 (w) = # 10 (w)} # s (w) the number of times s appears in w. All languages are over {0, 1}. Consider B: DFA must remember how many 0 s it has seen Impossible with finite state. The other languages seem to be exactly the same. Question: Is this a proof? Answer: Well, not quite. In fact, D is regular... Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 14 / 55

Part I The Pumping Lemma for Regular Languages Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 15 / 55

Regular Languages can be Pumped For any regular language L there exists l > 0 (the pumping length) s.t.: Any s L longer than l, can be pumped into a longer string in L. This is a powerful technique for showing that a language is not regular. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 16 / 55

The Pumping Lemma Lemma 4 For any regular language L, exists l > 0 (the pumping length) s.t.: every s L with s l can be written as s = xyz such that: 1 xy i z L for every i 0, 2 y > 0, and 3 xy l. Remarks: Without the second condition, the theorem would be trivial. The third condition is technical and sometimes useful. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 17 / 55

Proving the Pumping Lemma Let M = (Q, Σ, δ, q 1, F) be a DFA accepting L, and let l = Q. Let s L be with s l, and consider the sequence of states M traverse as it reads s = s 1... s n : s 1 s 2 s 3 s 4 s 5 s 6... s n q 1 q 20 q 9 q 17 q 12 q 13 q 9 q 2 q 5 F Since the sequence of states is of length s + 1 > l, and there are only l different states in Q, at least one state is repeated (by the pigeonhole principle). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 18 / 55

Proving the Pumping Lemma, cont. Let q 9 be the repeating state. Write s = xyz By inspection, M accepts xy k z for every k 0. y > 0, because the state q 9 is repeated. To ensure that xy l, pick first state repetition, which must occur no later than l + 1 states in sequence. and formally, by induction on k 0 Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 19 / 55

Application # 1 Corollary 5 B = {0 n 1 n : n > 0} is not regular. Proof: By contradiction. Suppose B is regular and let l be its pumping length. Consider the string s = 0 l 1 l B. Let x, y, z be strings guaranteed by the pumping lemma (i.e. s = xyz) 1 xy k z B for every k 0, 2 y > 0, and 3 xy l. If y is all 0, then xy 2 z has too many 0 s. If y is all 1, then xy 2 z has too many 1 s. If y is mixed, then xy 2 z is not of right form. In this proof, we did not use the third property. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 20 / 55

Application # 2 Corollary 6 C = {w : # 1 (w) = # 0 (w)} is not regular. Proof: By contradiction. Suppose C is regular. Let l be the pumping length. Consider the string s = 0 l 1 l C. Let x, y, z be strings guaranteed by the pumping lemma (i.e. s = xyz) 1 xy i z B for every k 0, 2 y > 0, and 3 xy l. Since xy l, the string y is all 0 s. Thus, xy 2 z / C (more 0 s than 1 s). Could we have used s = (01) l? Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 21 / 55

An Alternative Proof Claim 7 C = {w : # 1 (w) = # 0 (w)} is not regular. Proof: By contradiction. Suppose C is regular. Let D = 0 1, then D is regular. Thus, D C should be regular as well (why?). But D C = B, and by corollary 5, B is not regular. Contradiction. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 22 / 55

Application # 3 Corollary 8 E = {0 i 1 j : i > j} is not regular. Proof: By contradiction. Suppose E is regular. Let l be its pumping length. Consider the string s = 0 l 1 l 1 E. By the pumping lemma, s = xyz, where xy k z E for every k 0, y > 0 and xy l. Since xy l, y consists of 0s only. But xy 0 z = xz / E (it does not have more 0s than 1s). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 23 / 55

Application # 4 Corollary 9 The language Primes {0, 1} all strings whose length is a prime number is not regular. Proof: Suppose Primes is regular, accepted by DFA M, and let l be its pumping length. Let s = 1 p Primes, where p l is a prime (why does such p exist?). By the pumping lemma, s = xyz, where xy k z Primes for every k 0. Let y = m. Hence, xy p+1 z = 1 p+mp Primes Ahaa: p(m + 1) is not prime... Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 24 / 55

Another Example Consider the language L = {a i b n c n : n 0, i 1} {b n c m : n, m 0}. Any s L can be pumped: If s = a i b n c n, then set x = ε and y = a. If s = b n c m, then set x = ε and y = b. If s = c m, then set x = ε and y = c. (in all cases z is set arbitrarily). Is L regular? Well, no! So, how would you prove it? Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 25 / 55

Part II The Myhill-Nerode Characterization of Regular Languages Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 26 / 55

The Equivalence Relation L Let L Σ be a language. Define an equivalence relation L on pairs of strings: Let x, y Σ. We say that x L y if for every string z Σ, xz L if and only if yz L. It is easy to see that L is indeed an equivalence relation (reflexive, symmetric, transitive) on Σ. In addition, if x L y then for every string z Σ, xz L yz as well (this is called right invariance). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 27 / 55

The Equivalence Relation L, cont. Like every equivalence relation, L partitions Σ to (disjoint) equivalence classes. For every string x, let [x] Σ denote its equivalence class w.r.t. L (if x L y then [x] = [y] equality of sets). Question is, how many equivalence classes does L induce? In particular, is the number of equivalence classes of L finite or infinite? Well, it could be either finite or infinite. This depends on the language L. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 28 / 55

Four Examples L 1 = {w : # 1 (w) mod 4 = 0} L1 has finitely many equivalence classes. The equivalent classes are: [1], [11], [111], [1111] L 2 = {0 n 1 n : n N} L2 has infinitely many equivalence classes. [0] [0 2 ] [0 3 ]... L 3 = {a i b n c n : n 0, i 1} {b n c m : n, m 0} L3 has infinitely many equivalence classes. [ab] [ab 2 ] [ab 3 ]... Let L 4 {1} contain all strings whose length is a prime number. Then L4 has infinitely many equivalence classes. [2] [3] [5] [7] [11]... Proof sketches for above statements on the board. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 29 / 55

Myhill-Nerode Theorem Theorem 10 (Myhill-Nerode Theorem) Let L Σ be a language. L is regular L has finitely many equivalence classes. Four specific consequences: L 1 {0, 1} contains all strings where the number of 1s is divisible by 4. Then L 1 is regular. L 2 {0, 1} contains all strings of the form 0 n 1 n. Then L 2 is not regular. L 3 = {a i b n c n : n 0, i 1} {b n c m : n, m 0}. Then L 3 is not regular. Let L 4 {1} contain all strings whose length is a prime number. Then L 4 is not regular. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 30 / 55

Myhill-Nerode Theorem: Proof = Suppose L is regular. Let M = (Q, Σ, δ, q 0, F) be a DFA accepting it. For every x Σ, let α(q 0, x) Q be the state where the computation of M on input x ends. The relation M on pairs of strings is defined as follows: x M y if α(q 0, x) = α(q 0, y). Clearly, M is an equivalence relation. Furthermore, if x M y, then for every z Σ, also xz M yz. Therefore, xz L if and only if yz L. This means that x M y = x L y. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 31 / 55

Myhill-Nerode Theorem: Proof (cont.) = The equivalence relation M has finitely many equivalence classes (at most the number of states in M). We saw that x M y = x L y, so the number of equivalence classes of L is less or equal than the number of equivalence classes of M. Therefore, L has finitely many equivalence classes. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 32 / 55

Myhill-Nerode Theorem: Proof (cont.) = Suppose L has finitely many equivalence classes. We ll construct a DFA M that accepts L. Let x 1,..., x n Σ be representatives for the finitely many equivalence classes of L. The states of M are the equivalence classes [x 1 ],..., [x n ]. The transition function δ is defined as follows: For all a Σ, δ([x i ], a) = [x i a] (the equivalence class of x i a). Hey, why is this a proper definition of δ? (Hint: right invariance). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 33 / 55

Myhill-Nerode Theorem: Proof (cont.) = The initial state is [ε]. The accept states are F = {[x i ] x i L}. Easy: On input x Σ, M ends at state [x] (why?). Therefore M accepts x iff x L (why?). So L is accepted by DFA, hence L is regular. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 34 / 55

Further Consequences of the Myhill-Nerode Theorem We can use the ideas in the proof of the Myhill-Nerode theorem to prove that a given DFA, which accepts a language L, has the minimal number of states among all DFAs that accept L. Examples: Two DFAs for the language {w : # 1 (w) mod 3 = 0} (on board). a s b a q 1 r 1 a b a b a b q 2 r 2 b. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 35 / 55

Minimal DFAs Claim 11 Let L Σ be a regular language, and suppose that L has n equivalence classes. Let M be a DFA accepting L, and let m be the number of its states. Then n m. The claim was proved in the course of the proof of the Myhill-Nerode Theorem, when we saw that x M y implies x L y (in fancier words, the relation M is a refinement of the relation L ). Therefore, an M with n states is minimal in terms of the number of states (among all DFAs accepting L). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 36 / 55

Minimal DFAs, cont. Claim 12 Let L Σ be a regular language, and suppose that L has n equivalence classes. There is a DFA, M that accepts L, and has exactly n states. Proof (sketch): We show how to construct such M. We label the states of M by the n equivalence classes of L, namely [x 1 ], [x 2 ],..., [x n ]. Note that it does not matter which representative x i is used the i-th class won t change. The start state is, well, [ε]. The accepting states are those [x i ] for which x i L. The transition function is δ([x i ], a) = [x i a] (use right invariance to see that δ is well defined). That s it. End of construction Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 37 / 55

Minimal DFAs, cont. Claim 13 Let L Σ be a regular language, and suppose that L has n equivalence classes. There is a DFA, M that accepts L, and has exactly n states. That s it. End of construction. Really? We better prove that the DFA constructed above indeed accepts the language L. Proof (sketch): We show the stronger claim that for every x Σ, the DFA on input x reaches state [x]. This will prove the claim, since by the definition of M, x L iff [x] is an accepting state. And how shall we prove the stronger claim? Indeed, by induction on the length of x.. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 38 / 55

Minimal DFAs, cont. Claim 12 enables us to show that a given DFA, M, is minimal with respect to its language, L. To construct such M using it, we should know the number and nature of equivalence classes of L. The claim does not tell us how to take a given DFA and convert it to a minimal one. There is actually an algorithm that performs such minimization (its input is a DFA, and its output is a minimal DFA for the same language). We will not describe this algorithm. In addition, Claim 12 says nothing about minimal NFAs. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 39 / 55

Part III Closure Properties of Regular Languages Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 40 / 55

Simple Closure Properties (also seen in recitation) Regular languages are closed under complement. 1 Let M = (Q, Σ, δ, q 0, F ) be a DFA that accepts L. 2 Then M = (Q, Σ, δ, q 0, Q \ F ) is a DFA that accepts L = Σ \ L. This is a proof by construction (done at the recitation on week 1). 3 Complement for NFA?! Regular languages are closed under intersection. 1 L 1 L 2 = L 1 L 2. This is a proof by reduction to closure under union and complement. 2 You saw a proof, using a direct construction of deterministic finite automata, in the recitation. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 41 / 55

The Quotient Operation For languages L 1, L 2 Σ (not necessarily regular), define L 1 /L 2 = {x Σ : y L 2 such that xy L 1 } Examples: L 1 = {a 2n b 2n c 2n : n 0} and L 2 = {b n c 2n : n 0}. For xy to be an element of L 1 when y = b n c 2n is an element of L 2, x must be of the form a 2n b n. Then L 1 /L 2 = {a 2n b n : n 0}. Note that neither L 1 nor L 2 are regular. L 3 = a b c and L 4 = {b n c 2n : n 1}. Then L 3 /L 4 = a b. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 42 / 55

Closure under Quotient Recall, L 1 /L 2 = {x Σ : y L 2 such that xy L 1 } Theorem 14 Regular languages are closed under quotient (to the right) by any language. Proof: Let L 1 be a regular language and let L 2 be an arbitrary language. Let M = (Q, Σ, δ, q 0, F) be a DFA for L 1. We describe a DFA for L 1 /L 2 by keeping the structure of M but modifying the set of accepting states. The new set of accepting states is F = {q Q : y L 2, such that M from state q on the string y reaches an accepting state}. Then the DFA M = (Q, Σ, δ, q 0, F ) accepts L 1 /L 2. F is well defined, but might be hard to determine explicitly. This is an existential, or non constructive, proof. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 43 / 55

Part IV Algorithmic Questions for NFAs and DFAs Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 44 / 55

Algorithmic Questions for NFAs and DFAs Q.: Given an NFA, N, and a string w, is w L(N)? Answer: Construct the DFA equivalent to N and run it on w. Q.: Is L(N) =? Answer: This is a reachability question in graphs: Is there a path in the states graph of N from the start state to some accepting state? There are simple, efficient algorithms for this task (studied extensively in the algorithms course). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 45 / 55

More Questions Q.: Is L(N) = Σ? Answer: Check if L(N) =. Q.: Given N 1 and N 2, is L(N 1 ) L(N 2 )? Answer: Check if L(N 2 ) L(N 1 ) =. Q.: Given N 1 and N 2, is L(N 1 ) = L(N 2 )? Answer: Check if L(N 1 ) L(N 2 ) and L(N 2 ) L(N 1 ). In the future, we will see that for stronger models of computations, where many of these problems cannot be solved by any algorithm. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 46 / 55

And Now for Something Somewhat Different: Context Free Languages Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 47 / 55

Part V Context Free Languages Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 48 / 55

Summary - Regular Languages So far we saw Finite automata, Regular languages, Regular expressions, Myhill-Nerode theorem and pumping lemma for regular languages. Next, we will introduce stronger machines and languages with more expressive power: pushdown automata, context-free languages, context-free grammars Context free grammars are essential in compilers construction and programming languages. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 49 / 55

Context-Free Grammars An example of a context free grammar, G 1 : A 0A1 A B B # Terminology: Each line is a substitution rule or production. Each rule has the form: symbol string. The left-hand symbol is a variable (usually upper-case). A string consists of variables and terminals. One variable is the start variable. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 50 / 55

Rules for Generating Strings Write down the start variable (lhs of top rule). Pick a variable written down in current string and a derivation that starts with that variable. Replace that variable with right-hand side of that derivation. Repeat until no variables remain. Return final string (concatenation of terminals). Process is inherently non deterministic. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 51 / 55

Example Grammar G 1 : A 0A1 A B B # Derivation with G 1 : A 0A1 00A11 000A111 000B111 000#111 Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 52 / 55

A Parse Tree A A A A B 0 0 0 # 1 1 1 Question: What strings can be generated in this way from the grammar G 1? Answer: Exactly those of the form 0 n #1 n (n 0). Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 53 / 55

Context-Free Languages The language generated in this way is called the language of the grammar. For example, L(G 1 ) is {0 n #1 n n 0}. Any language generated by a context-free grammar is called a context-free language. Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 54 / 55

A Useful Abbreviation Rules with same variable on left hand side A 0A1 A B are written as: A 0A1 B Benny Chor (Tel Aviv University) Computational Models Class 3: Plan November 2, 2015 55 / 55