Topics COSC 2001 Introduction to the Theory of Computation Dr. David Forster Basic concepts of theoretical CS, with practical applications: Regular Languages Context Free Languages Recursively Enumerable Languages Finite Automata & Regular Expressions Context Free Grammars & Pushdown Automata Turing Machines, Computability, Halting Problem NP completeness (time permitting!) Text editor Parser, Compiler General computer model COSC 2001 2 Topics Today Administrivia Motivation & Overview Review: Set theory, relations, proofs Alphabets, Strings, and Languages Administrivia Lecture hours: R 19:00 to 22:00, CSB 121 Office hours: TR 16:00 to 17:00, CSB 2024 email: david@cs.yorku.ca subject: 2001: web page: www.cs.yorku.ca/course/2001/ Text: Kinber & Smith, Theory of Computing: A Gentle Introduction HMU = Hopcroft, Motwani & Ullman, Introduction to Automata Theory, Languages, and Computation COSC 2001 3 COSC 2001 4 Administrivia (II) Evaluation: 5 assignments 10% midterm 30% final 60% Acknowledgements These slides are courtesy Dr. Nancy Ide, Vassar, with some modifications and additions by David Forster. COSC 2001 5 COSC 2001 6 Slides presented May 9th and 16th, 2002 1
Motivation Overview (I) Why do we care about automata? modelling circuits, protocols pattern matching, search lexical analysers (RE) (RL) Automata states (including start & final) input arcs Push compilers (CFL) Start off on Push COSC 2001 7 COSC 2001 8 Overview (II) "Structural Representation" grammars Overview (III) Complexity decidable: what's possible? tractable: what's practical? (Sentence Subject Verb Object) regular expressions [a-za-z_][a-za-z_0-9]* COSC 2001 9 COSC 2001 10 Concept Review sets, membership (x R), subset (R S), union (R S), intersection (R S), cartesian product (R S), complement (R) pigeonhole principle relations: reflexive, symmetric, transitive equivalence relations, equivalence classes Formal Proof need it for this course need it to reason about design? programs? systems? 1. seat of your pants 2. verify everything 3. somewhere in between functions, bijections, inverses COSC 2001 11 COSC 2001 12 Slides presented May 9th and 16th, 2002 2
Deductive Proofs sequence of statements: hypothesis deductions conclusion deduction based on: accepted logical principle given fact previous step combination 3. 4. 5. Example of a Formal Proof Theorem: If x is the sum of the squares of four positive integers, then 2 x x 2 Step 1. 2. Statement x = a 2 + b 2 + c 2 + d 2 a 1; b 1; c 1; d 1 a 2 1; b 2 1; c 2 1; d 2 1 x 4 2 x x 2 Justification Given Given (2) & properties of arithmetic (1), (3), & properties of arithmetic (4) & Theorem 1.3* From HMU: theorem 1.4 & fig. 1.3 *: Previously proved in HMU: If x 4, then2 x x 2 COSC 2001 13 COSC 2001 14 Starting a Proof don't know how to start? reduce terms in hypothesis to their definitions e.g., some definitions: def n 1: a set S is finite if S = n, for some n def n 2: if sets S and T are subsets of a set U, then T is the complement of S w.r.t. U if S T = U and S T = φ now prove: Let S be a finite subset of some infinite set U. Let T be the complement of S with respect to U. Then T is infinite. COSC 2001 15 Example: Restating the givens Theorem: Let S be a finite subset of some infinite set U. Let T be the complement of S with respect to U. Then T is infinite. Original Statement S is finite U is infinite T is the complement of S From HMU: fig. 1.4: Restating the givens of Theorem 1.5 New Statement There is an integer n such that S = n For no integer p is U = p S T = U and S T = φ COSC 2001 16 A Proof 1) S and T are disjoint, so S + T = U 2) S is finite, so S = n for some integer n 3) U is infinite, so there is no p s.t. U = p 4) assume T is finite, so T = m 5) now, U = S + T = n + m 6) but this contradicts (3), so assumption (4) must be false, i.e., T must be infinite. Q.E.D. An "If-then" by any other name H C H implies C H only if C C if H Whenever H holds, C follows C hidden hypothesis? ex., "sin 2 θ + cos 2 θ = 1" is really: "if θ is an angle, sin 2 θ + cos 2 θ = 1" COSC 2001 17 COSC 2001 18 Slides presented May 9th and 16th, 2002 3
A if and only if B: If-and-only-If A B A iffb A is equivalent to B A exactly when B Iff (II) two parts, need to prove both: 1. if A then B 2. if B then A if and only-if are converses of each other "if A then B" says nothing about whether B is true when A is false (and vice versa) COSC 2001 19 COSC 2001 20 Quantifiers for all: there exists: order is important: d represents a driver, c a car d c: driver(d) (car(c) & drives(d,c)) "all drivers drive some car" (not necessarily the same one) c d: driver(d) (car(c)& drives(d,c)) "all drivers drive the same car" Additional forms of proof proofs about sets contrapositive proof by contradiction counterexamples COSC 2001 21 COSC 2001 22 Set Equivalences often need to prove two sets are same, e.g., sets of strings, languages, etc. we write: E = F we mean: "x is in E iff x is in F" e.g.: R S = S R so prove: x is in R S iff x is in S R The Contrapositive "if H then C" "if not C then not H" H C if con. F F T T F T T T T F F F T T T T don't confuse with "converse" COSC 2001 23 COSC 2001 24 Slides presented May 9th and 16th, 2002 4
Proof by Contradiction already demonstrated (slide "A Proof") assume opposite of something to prove show that the assumption must be false therefore, thing to prove must be true Counterexamples theorems are statements about infinite number of cases otherwise: merely an "observation" one false case disproves theorem "All computer scientists wear plaid." COSC 2001 25 COSC 2001 26 Inductive Proofs Prove a statement S(X) about a family of objects X (e.g., integers, trees) in two parts: 1. Basis: Prove for one or several small values of X directly. 2. Inductive step: Assume S(Y) for Y smaller than X; prove S(X) using that assumption. COSC 2001 27 Structural Induction (I) structures with recursive definitions: basis case(s): elementary structures inductive step(s): define complex structure in terms of previously defined structures e.g., complex noun phrases: Noun : "rocket", "manifold", "engine" Complex noun phrase : Noun or noun followed by a complex noun phrase thus: "rocket", "rocket engine", "rocket engine manifold", COSC 2001 28 Structural Induction (II) Use following form to prove theorems about recursively defined structures: Basis: Prove S(X) for the basis structure(s) X. Inductive Step: Take structure X formed from Y 1,,Y k, assume S(Y 1 ),,S(Y k ), use to prove S(X) Example (I) A binary tree with n leaves has 2n - 1 nodes Formally, S(T): if T is a binary tree with n leaves, then T has 2n-1 nodes Induction is on the size = number of nodes of T COSC 2001 29 COSC 2001 30 Slides presented May 9th and 16th, 2002 5
Example (II) Basis: If T has 1 leaf, it is a one-node tree. 2(1) 1 = 1. Induction: Assume S(U) for trees with fewer nodes than T. In particular, assume for the subtrees of T. (so now prove S(T) with this assumption) Example (III) 1. T must be a root plus two subtrees U and V 2. If U and V have u and v leaves, respectively, and T has t leaves, then u + v = t 3. By the inductive hypothesis, U and V have 2u-1 and 2v-1 nodes, respectively 4. Then T has 1+(2u-1)+(2v-1) nodes = 2(u+v) 1 = 2t-1, proving the inductive step. COSC 2001 31 COSC 2001 32 Example: Balanced Parentheses Here are two ways to define "balanced parentheses": 1. Grammatically: a) The empty string ε is balanced b) If w is balanced, then (w) is balanced c) If w and x are balanced, then so is wx 2. By Scanning: w is balanced if and only if: a) w has an equal number of left and right parentheses b) Every prefix of w has at least as many left as right parentheses Call these GB and SB properties, respectively. Theorem: A string of parentheses w is GB if and only if it is SB. If An induction on w (length of w). Assume w is SB; prove it is GB. Basis: If w = ε (length = 0), then w is GB by rule (a). Notice that we do not even have to address the question of whether ε is SB (it is, however). Induction: Suppose the statement "SB implies GB" is true for strings shorter than w. COSC 2001 33 COSC 2001 34 Case 1: w is not ε, but has no non-empty prefix that has an equal number of ( and ). Then w must begin with ( and end with ); i.e., w = (x). x must be SB (why?) By the IH, x is GB GB/SB (II) By rule 1(b), (x) is GB; but (x) = w, so w is GB Case 2: w = xy, where x is the shortest non-empty prefix of w with an equal number of ( and ), and y is not ε. x and y are both SB (why?) By the IH, x and y are GB w is GB by rule 1(c) GB/SB (III) COSC 2001 35 COSC 2001 36 Slides presented May 9th and 16th, 2002 6
Only-If GB/SB (IV) An induction on w. Assume w is GB; prove it is SB. Basis:w = ε. Clearly w obeys the conditions for being SB. Induction: Assume GB implies SB" for strings shorter than w, and assume w is not ε. Case 1: w is GB because of rule 1(b); i.e., w = (x) and x is GB. by the IH, x is SB. GB/SB (V) Since x has equal numbers of ('s and )'s, so does (x). Since x has no prefix with more ('s than )'s, so does (x). COSC 2001 37 COSC 2001 38 Case 2: GB/SB (VI) w is not ε and is GB because of rule (c); i.e., w = xy, and x and y are GB. By the IH, x and y are SB. (Aside) Trickier than it looks: we have to argue that neither x nor y could be ε, because if one were, the other would be w, and this rule application could not be the one that first shows w to be GB. xy has equal numbers of ('s and )'s because x and y both do. If w had a prefix with more )'s than ('s, that prefix would either be a prefix of x (contradicting the fact that x has no such prefix) or it would be x followed by a prefix of y (contradicting the fact that y also has no such prefix). Above is an example of proof by contradiction. We assumed our conclusion about w was false and showed it would imply something that we know is false. COSC 2001 39 Languages Alphabet (Σ): finite set of symbols, e.g., - Σ = {0, 1} (binary alphabet) - ASCII Word or String = finite sequence of symbols chosen from some alphabet, e.g., 01101 or abracadabra Language = set of strings chosen from some alphabet COSC 2001 40 Language Examples The set of all binary strings consisting of at least one 0, followed by an equal number of 1's, i.e., {01, 0011, 000111, } C (the set of compilable C programs) English COSC 2001 41 Strings Empty string:ε Length: abcde= 5 uv = u + v Reverse: w R If w= abc, w R = cba Substring: any string of consecutive characters in some w Prefix and suffix: if w = vu, v and u are a prefix and a suffix of w, respectively Convention: Use lower case letters from the beginning of the alphabet for symbols, lower case letters from end of alphabet for strings COSC 2001 42 Slides presented May 9th and 16th, 2002 7
Strings (cont'd) Concatenation : Given strings x and y, concatenation is xy e.g., x = abc, y = def, xy = abcdef Powers Σ k = set of all strings from alphabet Σ with length k Σ 0 = {ε} Σ * = set of all strings from alphabet Σ Σ + = Σ * -{ε} COSC 2001 43 More About Languages Can concatenate languages L 1 L 2 = {xy x L 1, y L 2 } L n = L concatenated with itself n times L 0 = {ε}; L 1 = L Star-closure L * = L 0 L 1 L 2 (note: L + = L * -L 0 ) Languages can be finite or infinite L = {a, aba, bba} L = {a n n > 0} Subtle point: a language may be infinite, but there is some finite set of symbols of which all its strings are composed COSC 2001 44 Finite Automata describe regular languages: simple, highly useful L s graphwith finite number of nodes (states) arcs labelled with one or more symbols from Σ one state is start / initial state some states are final / favoured / accepting states The language of the FA is the set of strings that label paths that go from the start state to some accepting state Example Below FA scans HTML documents, looking for a list of what could be title-author pairs Accepts whenever it finds the end of a list item Could be basis for application COSC 2001 45 COSC 2001 46 Scan HTML Documents HTML Document Example start <ol>, <ul> <li> <li> Any non-tag space b y space <ul> <li> start <ol>, <ul> <li> IALC by HM&U </li> <li> <li> LotR by T </li> </ul> </ol>, </ul> </li> Any non-tag space b y space Any non-tag Any non-tag </ol>, </ul> </li> COSC 2001 47 COSC 2001 48 Slides presented May 9th and 16th, 2002 8
Recognize Simple Identifiers Newspaper Vending Machine quarter start letter letter or digit start dime dime dime quarter dime, quarter other character (delimiter) nickel nickel nickel nickel nickel, dime, quarter quarter COSC 2001 49 COSC 2001 50 Formal Definition of Finite Automaton A FA is represented as the five-tuple: A = (Q, Σ, δ, q 0, F) Q Finite set of states Σ Alphabet of input symbols δ A transition function q 0 start/initial state F Zero or more final/favoured/accepting states FA Transition Function We write: δ(q,a) = p where: q, p are states a is an input symbol Intuitively: if the FA is in state q, and input a is received, then the FA goes to state p (note: q = p is OK) COSC 2001 51 COSC 2001 52 Example: Clamping Logic We may think of an accepting state as representing a "1" output and non-accepting states as representing "0" output. A "clamping" circuit waits for a 1 input, and forever after makes a 1 output. However, to avoid clamping on spurious noise, we'll design an FA that waits for two 1's in a row, and "clam" only then. In general, we may think of a state as representing a summary of the history of what has been seen on the input so far. COSC 2001 53 The states we need are: State q 0, the start state, says that the most recent input (if there was one) was not a 1, and we have never seen two 1's in a row. State q 1 says we have never seen 11, but the previous input was 1. State q 2 is the only accepting state, it says that we have at some time seen 11. Thus, A = ({q 0, q 1, q 2 }, {0, 1}, δ, q 0, {q 2 }), where δ is given by: 0 1 >q 0 q 0 q 1 q 1 q 0 q 2 *q 2 q 2 q 2 By marking the start state with > and accepting states with *, the transition table that defines δ also specifies the entire FA COSC 2001 54 Slides presented May 9th and 16th, 2002 9
Conventions It hel if we can avoid mentioning the type of every name by following some rules: Input symbols are a, b, etc., or digits. Strings of input symbols are u, v,..., z. States are q, p, etc. Transition Diagram An FA can be represented by a graph: nodes = states arc from q to p is labeled by the set of input symbols a such that δ(q, a) = p. No arc if no such a. Start state indicated by word "start" and an arrow. Accepting states get double circles. COSC 2001 55 COSC 2001 56 Example Extension of δ to Paths For the Clamping FA: start 0 1 1 0 0,1 Intuitively, a FA accepts a string w = a 1 a 2 a n if there is a path in the transition diagram that: 1. Begins at the start state, 2. Ends at an accepting state, and 3. Has sequence of labels a 1,a 2,,a n COSC 2001 57 COSC 2001 58 Extension of δ to Paths (II) Formally, extend δ to Basis: i.e., stay put on no input Induction: i.e., see where the FA goes on w, then look for the transition on the last symbol from that state δ (q,w) δ (q,ε) = q δ (q,wa) = δ (δ (q,w),a) Aside Important fact with a straightforward, inductive proof: δ really represents paths. That is, if w = a 1 a 2 a n, and δ ( p i-1,a i ) = p i for all i = 1,..., n, then δ ( p 0,w) = p n COSC 2001 59 COSC 2001 60 Slides presented May 9th and 16th, 2002 10
Acceptance of Strings FA A = (Q, Σ, δ, q 0, F) accepts string w if δ (q 0, w) is in F. Language of a FA FA A accepts the language L(A) = {w δ (q 0, w) is in F}. COSC 2001 61 Type Errors A major source of confusion when dealing with automata (or mathematics in general) is making "type errors." Example: Don't confuse A, a FA, i.e., a program, with L(A), which is of type "set of strings." Example: the start state q 0 is of type "state," but the accepting states F is of type "set of states." Trickier example: Is a a symbol or a string of length 1? Answer: it depends on the context, e.g., is it used in δ(q, a), where it is a symbol, or δ(q, a), where it is a string? COSC 2001 62 Non-deterministic Finite Automata Allow (deterministic) FA to have a choice of 0 or more next states for each state-input pair. Important tool for designing string processors, e.g., grep, lexical analyzers. But "imaginary," in the sense that it has to be implemented deterministically. Example Design an NFA to accept strings over alphabet {1, 2, 3} such that the last symbol appears previously, without any intervening higher symbol, e.g., 11 21112 312123 Trick: use start state to mean "I guess I haven't seen the symbol that matches the ending symbol yet." Three other states represent a guess that the matching symbol has been seen, and remembers what that symbol is. COSC 2001 63 COSC 2001 64 Formal NFA (I) 1,2,3 p q 1 1 1 r 2 2 t N = (Q, Σ, δ, q 0, F) where all is as DFA, but δ(q, a) is a set of states, rather than a single state. 3 3 s 1,2 COSC 2001 65 COSC 2001 66 Slides presented May 9th and 16th, 2002 11
Formal NFA (II) Extension to δ Basis: δ (q, ε) = {q}. Induction: Let: δ (q, w) = {p 1, p 2,, p k } δ (p i, a) = S i for i = 1, 2,, k Then: δ (q, wa) = S 1 S 2 S k. COSC 2001 67 Language of a NFA An NFA accepts w if any path from the start state to an accepting state is labeled w. Formally: L(N) = {w δ (q 0, w) F φ}. COSC 2001 68 Equivalence of DFA and NFA For every NFA there is an equivalent (accepts the same language) DFA. But the DFA can have exponentially many states. From NFA N = (Q N, Σ, δ N, q 0, F N ) create equivalent DFA D = (Q D, Σ, δ D, {q 0 }, F D ): 1. Q D = 2 QN, i.e., the set of all subsets of Q N 2. F D is the set of sets S in Q D such that S F N φ 3. δ D ({q 1,q 2, q k }, a) = δ N (p 1,a) δ N (p 2,a) δ N (p k,a) Key theorem (induction on w ): δ D ({q 0 }, w) = δ N (q 0,w) Consequence: L(D) = L(N) DFA from NFA COSC 2001 69 COSC 2001 70 Subset Construction Starting with q 0 in N, for each a Σ, compute the set S of states accessible from q 0 on input a: this set becomes a new state in D, so δ D ({q 0 }, a) = S Repeat for each new state so discovered, so that for each such set S, and each p i S δ D (S, a) =δ N (p 1, a) δ N (p 2, a) δ N (p k, a) COSC 2001 71 Example: DFA from Clamping NFA 1,2,3 p q 1 1 1 r 2 2 3 3 s 1,2 COSC 2001 72 t Subset construction: δ({p},1)={p,q} δ({p},2)={p,r} δ({p},3)={p,s} δ({p,q},1)={p,q,t} δ({p,q},2)={p,r} δ({p,q},3)={p,s} δ({p,r},1)={p,q,r} ¼ Slides presented May 9th and 16th, 2002 12
Example: δ for Clamping DFA For the NFA example above, there are 32 possible subsets computing transitions "on demand" gives a DFA with 15 states and this δ D : >p pq *pqt pr *prt *t prs *prst pqs *pqst pqr *pqrt pqrs *pqrst pq pqt pqt pqr pqr pqs pqs pqrs pqrs pqst pqst pqrt pqrt pqrst pqrst pr pr pr prt prt prs prs prst prst prs prs prt prt prst prst t t t t t t t t COSC 2001 73 1 2 3 These slides were presented in the first two lectures of COSC 2001.03B, Summer 2002. COSC 2001 74 Slides presented May 9th and 16th, 2002 13