Computability and Complexity

Computability and Complexity Non-determinism, Regular Expressions CAS 705 Ryszard Janicki Department of Computing and Software McMaster University Hamilton, Ontario, Canada janicki@mcmaster.ca Ryszard Janicki Computability and Complexity 1 / 49

Non-determinism vs Determinism: Example Does non-determinism increases the descriptive power? Ryszard Janicki Computability and Complexity 2 / 49

Non-determinism vs Determinism: Example Does non-determinism increases the descriptive power? NO. See an example below and Theorem on next page. Ryszard Janicki Computability and Complexity 2 / 49

Non-determinism vs Determinism Theorem Let L be a language accepted by a non-deterministic nite automaton M, i.e. L = L(M). There exists a deterministic nite automaton M such that L(M ) = L. Proof. Let M = (Q, Σ, δ, q 0, F ) be a non-deterministic automaton, i.e. δ : Q Σ 2 Q, such that L(M) = M. Dene M = (Q, Σ, δ, q 0, F ) as follows: Q = 2 Q δ : Q Σ Q, i.e. δ : 2 Q Σ 2 Q, and δ ({q 1,..., q n }, a) = n δ(q i, a) i=1 q 0 = {q 0} F = {q q Q = 2 Q q F } Ryszard Janicki Computability and Complexity 3 / 49

Proof - continued First we will prove that: δ ({q 0 }, x) = {q 1,..., q n } = δ(q 0, x), where {q 1,..., q n } is a set of states in M but one state in M. The proof is by induction on x (very popular technique for automata and strings theory!). We will also identify δ with ˆδ, and δ with ˆδ here. Base case: x = 0 x = ε = δ ({q 0 }, ε) = {q 0 } = δ(q 0, ε). Induction step: Suppose that the hypothesis is true for x = m. Consider xa = m + 1. We have: δ ({q 0 }, xa) = δ (δ ({q 0 }, x) }{{}, a) = δ(q 0, x) by induction hypothesis Suppose that δ(q 0, x) = {q 1,..., q k }. Ryszard Janicki Computability and Complexity 4 / 49

Proof - continued Hence: δ ({q 0 }, xa) = δ ({q 1,..., q k }, a) On the other hand: δ(q 0, xa) = s δ(q 0,x) δ(s, a) = By the denition of δ = s {q 1,...,q n δ(s, a) = Which means δ ({q 0 }, x) = δ(q 0, x) for all x. k δ(q i, a) i=1 k δ(q i, a) i=1 Ryszard Janicki Computability and Complexity 5 / 49

Proof - end Since F = {q q Q = 2 Q q F }, then δ ({q 0 }, x) F δ ({q 0 }, x) F δ(q 0, x) F. Which means that: L(M ) = {x δ ({q 0 }, x) F } = {x δ(q 0, x) F } = L(M), which ends the proof. Ryszard Janicki Computability and Complexity 6 / 49

Non-determinism and `Black Box' Paradigm The proof can be extended for some innities, but not for all. Important Conclusion: For most of the cases non-determinism is not observable, which means that the `Black Box' approach may not always work, we need some insider information to decide if the process is deterministic or not. Ryszard Janicki Computability and Complexity 7 / 49

Invisible Actions or ε-moves In many cases we need to model invisible actions! Denition A non-deterministic automaton with ε-moves is: M = (Q, Σ, δ, q 0, F ), where: Q, Σ, q 0, F are as usual, and: δ : Q (Σ {ε}) 2 Q. Problem: ε has two interpretations! Ryszard Janicki Computability and Complexity 8 / 49

ε-closure Dene the following family of functions ˆε i : 2 Q 2 Q, i = 0, 1, 2,... as follows: for each set of states X Q ˆε 0 (X ) = X ˆε 1 (X ) = δ(x, ε) = q X δ(q, ε) ˆε i+1 (X ) = δ(ˆε i (X ), ε) = q δ(q, ε) Denition For every X Q, we dene the ε-closure of X as: εcl(x ) = i=0 ˆεi (X ). εcl(x ) is the set of states that contains X and all the states that can be reached from X by arrows labelled with ε. Ryszard Janicki Computability and Complexity 9 / 49

ˆδ and Language for Automata with ε-moves Denition We dene ˆδ : Q Σ 2 Q as follows: ˆδ(q, ε) = εcl({q}) ˆδ(q, xa) = εcl(δ(ˆδ(q, x), a) or ˆδ(q, xa) = εcl(y ), where Y = r ˆδ(q,x) δ(r, a). Clearly it may happen that δ(q, ε) ˆδ(q, ε) Denition The language generated by an automaton with ε-moves M, is dened now standardly as L(M) = {x ˆδ(q 0, x) F }. Ryszard Janicki Computability and Complexity 10 / 49

Removal of ε-moves Theorem If L is accepted by a nondeterministic automaton with ε-moves, then L is also accepted by a nondeterministic automaton without ε-moves. sketch. Let M = (Q, Σ, δ, q 0, F ) be a nondeterministic automaton with ε-moves. We dene M = (Q, Σ, δ, q 0, F ), where s Q. a Σ. δ (q, a) = ˆδ(q, a) { F F {q0 } εcl({q = 0 }) F F otherwise We are removing ε-moves in the construction above. One can show that L(M) = L(M ). Ryszard Janicki Computability and Complexity 11 / 49

Removal of ε-moves: An example Ryszard Janicki Computability and Complexity 12 / 49

Non-determinism, ε-moves: Final Comment For nite automata, neither non-determinism nor ε-moves increase the descriptive power. We can always translate one into another, however in the worst case, a deterministic automaton may have 2 Q states. For deterministic automata, angelic and demonic semantics are identical, as we do not have any choice. Ryszard Janicki Computability and Complexity 13 / 49

Regular Languages Denition (The most popular one) Let Σ be an alphabet. A language L Σ is called regular if and only if there exists a deterministic nite automaton M such that L = L(M). The class of all regular languages will be denoted by L R. Since every non-deterministic automaton with ε-moves can be simulated by an appropriate non-deterministic automaton without ε-moves, and every non-deterministic automaton can be replaced by an appropriate deterministic automaton, in the denition above we can replace `deterministic' with `non-deterministic' or `non-deterministic with ε-moves'. Ryszard Janicki Computability and Complexity 14 / 49

Basic Operators on Languages Let Σ be an alphabet and let A, B Σ. Set operators: Union: A B = {x x A x B}. Intersection: A B = {x x A x B}. Subtraction: A \ B = {x x A x / B}. Complement: A = Σ \ A. Concatenation: AB = A B = {xy x A x B}. Star (Kleene star): A = {x 1 x 2... x k k 0 x i A}. On page 7 of Lecture Notes 1, we dene Σ as Σ = {a 1... a k a i Σ k 0}. Note that Σ = Σ, where on the left hand side we use the denition of Kleene star and on the right hand side we use the denition from page 7 of Lecture Notes 1. The operators ` ', ` ' and `Kleene star' are called regular. Ryszard Janicki Computability and Complexity 15 / 49

Regular Expressions: Intuition `zero or one followed by any number of zeros (0 1)0 {0, 00, 000,..., 1, 10, 100, 1000,...} including none' ab {a, ab, abb, abbb,...} (a b) {a, b} `all strings (including ε) that can be built from a and b' (a ε)(b ε) = ab εb aε εε = ab b a ε {ε, a, b, ab} Ryszard Janicki Computability and Complexity 16 / 49

Denition (Formal Denition of Regular Expressions) Let Σ be an alphabet. A string R built from the elements of Σ {ε,, (, ),, } is a regular expression, if it is dened by the following rules: 1, ε and each a Σ are regular expressions. 2 (R 1 R 2 ) is a regular expression if R 1 and R 2 are regular expressions. 3 (R 1 R 2 ) is a regular expression if R 1 and R 2 are regular expressions. 4 (R) is a regular expression if R is a regular expression. 5 There are no other regular expressions. The set of all regular expressions over the alphabet Σ will be denoted by Rex(Σ). We usually skip some parenthesis. Rules: rst, followed by concatenation, and nally, unless parentheses say dierently. Ryszard Janicki Computability and Complexity 17 / 49

Languages Dened by Regular Expressions Denition (Interpretation) Let L : Rex(Σ) 2 Σ be the following function called interpretation: 1 L( ) =, L(ε) = ε, L(a) = {a} 2 L((R 1 R 2 )) = L(R 1 ) L(R 2 ) 3 L((R 1 R 2 )) = L(R 1 )L(R 2 ) 4 L((R) ) = L(R) Denition (Language) For every regular expression R, L(R) is a language dened by R. A class of all languages dened by regular expressions will be denoted by L REX. Ryszard Janicki Computability and Complexity 18 / 49

Languages Dened by Regular Expressions: Examples L((0 1)0 ) = {0, 00, 000,..., 1, 10, 100, 1000,...} L(ab ) = {a, ab, abb, abbb, abbbb,...} L((a ε)(b ε)) = {ε, a, b, ab} We customarily often identify a regular expression R with L(R) but technically R is not L(R). A question: What is the relationship between L R and L REX? Ryszard Janicki Computability and Complexity 19 / 49

Regular Expressions and Automata Lemma For every regular expression R, L(R) L R, i.e. L(R) is a regular language. In other words, L REX L R. Proof. We have to show that for every regular expression R, there is an automaton (could be non-deterministic with ε-moves) M R such that L(M R ) = L(M). The proof is by the induction on the denition of REX (Σ). 1. R = a, then L(R) = {a}, so the automaton M R is: a 2 2. R = ε, then L(R) = {ε}, so the automaton M R is: 2 3. R =, then L(R) =, so the automaton M R is: 4. Let R = R 1 R 2 and let M R1, M R2 be automata such that L(M R1 ) = L(R 1 ) and L(M R2 ) = L(R 2 ). Dene M R as follows: Ryszard Janicki Computability and Complexity 20 / 49

Proof for R = R 1 R 2 4. Let R = R 1 R 2 and let M R1, M R2 be automata such that L(M R1 ) = L(R 1 ) and L(M R2 ) = L(R 2 ). Dene M R as follows: Clearly L(M R ) = L(M R1 ) L(M R2 ) = L(R 1 ) L(R 2 ) = L(R). Ryszard Janicki Computability and Complexity 21 / 49

Proof for R = R 1 R 2 5. Let R = R 1 R 2 and let M R1, M R2 be automata such that L(M R1 ) = L(R 1 ) and L(M R2 ) = L(R 2 ). Dene M R as follows: Clearly L(M R ) = L(M R1 )L(M R2 ) = L(R 1 )L(R 2 ) = L(R). Ryszard Janicki Computability and Complexity 22 / 49

Proof for R = (R 1 ) 6. Let R = (R 1 ) and let M R1 be a automaton such that L(M R1 ) = L(R 1 ). Dene M R as follows: Clearly L(M R ) = L(M R1 ) = L(R 1 ) = L(R). Hence for every regular expression R, there is an automaton (usually non-deterministic with ε-moves) M R such that L(M R ) = L(M). Ryszard Janicki Computability and Complexity 23 / 49

From Automata to Regular Expressions So far we have shown that every regular expression can be transformed into appropriate automaton. We provide rules for transforming regular expressions into non-deterministic automata with ε-moves, but then we can use the rules for removal of ε-moves, and removal of non-determinism. Hence every regular expression correspond to an equivalent deterministic automaton, i.e. L REX L R. What about an opposite direction? Can we nd an appropriate regular expression for every nite automaton? Is it true that L R L REX? One possible method involves the concept of generalized non-deterministic automata. Ryszard Janicki Computability and Complexity 24 / 49

From Automata to Regular Expressions A generalized non-deterministic nite automaton has: ONE nal state Arrows are labeled by regular expressions No arrow to the initial state No arrow from the nal state Two states are connected by at most one arrow Ryszard Janicki Computability and Complexity 25 / 49

Generalized Non-deterministic Automata Denition Let Σ be an alphabet. A generalized non-deterministic (nite) automaton is a 5-tuple: GM = (Q,, δ, s 0, {s final }), where: Q is the set of states (nite), REX (Σ) is the generalized alphabet (nite), δ : Q 2 Q is the transition function, s 0 Q is the initial state, s final Q is its only nal state, and s Q. R. s 0 / δ(s, R) R. δ(s final, R) = (no arrow to the initial state) (no arrow from the nal state) s, q Q. (q δ(s, R 1 ) q δ(s, R 2 ) = R 1 = R 2 ) (two states are connected by at most one arrow). Ryszard Janicki Computability and Complexity 26 / 49

Languages of Generalized Non-deterministic Automata Denition The set L gen (GM) = {R s final ˆδ(s 0, R)} is the generalized language of GM. Note that L gen (GM) REX (Σ), i.e. every element of L gen (GM) is a regular expression. The set L(GM) = {x R L gen (GM). x L(R)} = is the language accepted/generated by GM. R L gen(gm) In the formula above, L(R) is a language generated by a regular expression R. L(R) Ryszard Janicki Computability and Complexity 27 / 49

Languages of Generalized Automata: Example Consider the following generalized automaton GM: Σ = {a, b, c}, Q = {s 0, s 1, s 2, s final } ={ε, ab, (ac), cb a, a c}={(ε), (ab), (ac), (cb a ), (a c)}, L gen (GM) = ε (ab) ( (ac) (cb a ) (a c) ), L(GM) = (ab) ((ac) (cb a ) (a c)), ε (ab) (ac) (cb a ) = (ab)(ac) (cb a ) L gen (GM), abacacaccb (ab)(ac) (cb a ), so abacacaccb L(GM). Ryszard Janicki Computability and Complexity 28 / 49

Automata vs Generalized Automata Every automaton can easily be transformed into an equivalent generalized automaton in two easy steps: step 1: step 2: Ryszard Janicki Computability and Complexity 29 / 49

Automata vs Generalized Automata: Example Example Ryszard Janicki Computability and Complexity 30 / 49

Algorithm CONVERT (GM): Basis Idea The algorithm starts with an arbitrary generalized automaton and then remove a state, that is neither nal, nor initial, while preserving the language generated by the initial automaton. This process is repeated until only the initial and the nal state remain. A regular expression assigned to the arrow that connects the initial and nal states represents the same language as the one generated by the initial generalized automaton. Ryszard Janicki Computability and Complexity 31 / 49

Algorithm CONVERT (GM): State Removal The state removal is based on the following idea: Ryszard Janicki Computability and Complexity 32 / 49

Algorithm CONVERT (GM) Let Q = n. We may assume that all states are represented by numbers 1,..., n, with s 0 = 1 and s final = n. We also assume that if there is an arrow between states i and j, it is labeled by an regular expression R ij. If n=2 then GM = R 2 and CONVERT (GM) returns R. If n > 2, select q k Q \ {1, n} and dene GM = (Q \ {q k },, δ, s 0, {s final }), where GM is derived from GM by removing the state k, removing all arrows that start with or end at k and, for all i, j dierent from k, including the case i = j, if j δ(i, R ij ), replacing R ij by R ij = R ik(r kk ) R kj R ij. Next we compute CONVERT (GM ) This is a recursive procedure that stops when the only states are 1 and n. Ryszard Janicki Computability and Complexity 33 / 49

Algorithm CONVERT (GM): Example 1 Ryszard Janicki Computability and Complexity 34 / 49

Algorithm CONVERT (GM): Example 2 Ryszard Janicki Computability and Complexity 35 / 49

Antomata vs Regular Expressions: Conclusion Theorem L R = L REX Ryszard Janicki Computability and Complexity 36 / 49

Programs as Automata and Regular Expressions of Relations Consider the well-known procedure factorial, written in a small subset of Maple: factorial := proc(n::posint) local i, fac i:=1; fac:=1; while i < n do begin i:=i+1 fac:=fac i; end; end proc; Since n does not change its value in the above program we may consider it as a constant, so we may assume the above program has two integer variables i and fac. Ryszard Janicki Computability and Complexity 37 / 49

About Relations: Composition and Identity Let X, Y are sets. Any R X Y is a (binary) relation. Composition of Relations: Let R X Y, S Y Z. Then the relation R S X Z, a composition of R and Z is a relation dened as follows: x(r S)z y Y. xry ysz. Identity: For every non-empty set X, the relation I X = {(x, x) x X } X X is called the identity on X. Power: For every set X and every relation R X X, we dene: R 0 = I X R k+1 = R k R, or, equivalently, R k = R}.{{.. R}. k Ryszard Janicki Computability and Complexity 38 / 49

About Relations: Transitive and Reexive Closures Let X be as set and let R X X be a relation. A relation R + = is a transitive closure of R. A relation R = i=1 i=0 is a reexive and transitive closure of R (Kleene star). R i R i We have: R = R + I X and R + = R R. An alternative denitions of R + and R : xr + y i 1. xr i y, xr y i 0. xr i y Ryszard Janicki Computability and Complexity 39 / 49

Programs as Relations Dene D = ZZ ZZ, where ZZ is the set of integers, and denote the elements of D as (i, fac). Each assignment statement can be modeled by a function F i : D D, i = 1, 2, 4, 5, in the following manner: "i:=1" corresponds to F 1 (i, fac) = (1, fac), "fac:=1" corresponds to F 2 (i, fac) = (i, 1), "i:=i+1" corresponds to F 4 (i, fac) = (i + 1, fac), and "fac:=fac*i" maps to F 5 (i, fac) = (i, fac i). The test "i<n" can be modeled by two partial identity functions, I 3, Ī 3 : D D, where I 3 models "i<n", and Ī 3 models its complement, i.e. "i n". More precisely, "i<n" corresponds to I 3 (i, fac), and "i n" corresponds to Ī3(i, fac), where ( denotes undened) I 3 (i, fac) = { (i, fac) if i < n otherwise Ī 3 (i, fac) = { (i, fac) if i n otherwise Ryszard Janicki Computability and Complexity 40 / 49

Programs as Relational Expressions Let R, R 1, R 2 be relations (each function is a relation!) that model the program statements S, S1, S2, respectively. Let T be a test modeled by partial identities I T and Ī T, and let the symbols and denote the composition of relations, and transitive and reexive closure of relations (Kleene star), respectively. Formally, if R, R 1, R 2 X X, then x(r 1 R 2 )y z X. xr 1 z zr 2 y R = i=0 Ri, where R 0 = I X Alternative denition of R : xr y i 0. xr i y Ryszard Janicki Computability and Complexity 41 / 49

Programs as Relational Expressions We can now model the basic programming constructs as follows "S1;S2" is modeled by R 1 R 2, "if T then S1 else S2" is modeled by (I T R 1 ) (Ī T R 2 ), and "while T do S" is modeled by (I T R) Ī T. Using this scheme one can easily model the above program by writing the following (symbolic) relational expression: F = F 1 F 2 (I 3 F 4 F 5 ) Ī 3, or F = F 1 }{{} i:=1 F 2 }{{} fac:=1 i<n i:=i+1 {}}{ F 4 fac:=fac i {}}{ i n {}}{{}}{ ( I 3 F 5 ) Ī }{{ 3 } while i<n do i:=i+1;fac:=fac i od Ryszard Janicki Computability and Complexity 42 / 49

Programs as Automata We can now model the basic programming constructs as follows "S1;S2" is modeled by "if T then S1 else S2" is modeled by, and "while T do S" is modeled by Ryszard Janicki Computability and Complexity 43 / 49

Programs as Automata Using this scheme one can easily model the above program by the following nite deterministic automaton: which clearly corresponds to a regular relational expression: F = F 1 F 2 (I 3 F 4 F 5 ) Ī 3, Ryszard Janicki Computability and Complexity 44 / 49

Programs as Relational Expressions If R 1 and R 2 are (possibly partial) functions, calculating R = R 1 R 2 is easy: R(x 1,..., x n ) = R 2 (R 1 (x 1,..., x n )). If at least one of R 1, R 2 is not a function, in general, we have to use the rule: (x 1,..., x n )R 1 R 2 (z 1,..., z n ) (y 1,..., y n ). (x 1,..., x n )R 1 (y 1,..., y n ) (y 1,..., y n )R 2 (z 1,..., z n ). Nevertheless, it might happen that R 1 R 2 is a function even if both R 1 and R 2 are not. In general R 1 R 2 is not a function, even if both R 1 and R 2 are functions. Similarly, R = i=0 Ri is almost never a function, even if R is a function, since if R is a function, then (x 1,..., x n )R (y 1,..., y n ) i 0. (y 1,..., y n ) = R i (x 1,..., x n )), and this may happen for many, even innite number of i's. Ryszard Janicki Computability and Complexity 45 / 49

Programs as Relational Expressions Lemma (1) 1 For any test T, if R 1 and R 2 are functions then (I T R 1 ) (Ī T R 2 ) is always a function. 2 For any test T, if R is a function, then (I T R) Ī T is either a function or the empty relation. 3 For any test T, if R is a function and (I T R) Ī T, then ((I T R) Ī T )(x) = R k(x) (x) where k(x) is the smallest j such that Ī T (R j (x 1,..., x n ))(x). Ryszard Janicki Computability and Complexity 46 / 49

Dene G = I 3 F 4 F 5 and H = G Ī3, so F = F 1 F 2 H. First note that (F 1 F 2 )(i, fac) = F 2 (F 1 (i, fac)) = (1, 1), so F (i, fac) = H(F 2 (F 1 (i, fac))) = H(1, 1). For the function G we have: G(i, fac) = (I 3 F 4 F 5 )(i, fac) = F 5 (F 4 (I 3 (i, fac))) = Similarly : { (i + 1, fac (i + 1)) if i < n if i n { (i + 2, fac (i + 1) (i + 2)) G 2 if i + 1 < n (i, fac) = G(G(i, fac)) = if i + 1 n Hence : { G j (i + j, fac (i + 1) (i + 2)... (i + j)) if i + j 1 < n (i, fac) = if i + j 1 n Notice that this last step requires a small amount of human ingenuity to see the pattern (although it can be automated in some cases). Ryszard Janicki Computability and Complexity 47 / 49

From Lemma 1(3) it follows H(i, fac) = G k (i, fac) where k = k(i, fac) is the smallest j such that Ī 3 (G j (i, fac)). In this case we can easily show that there is only one such j and that k(i, fac) = n i. Denote fac = fac (i + 1) (i + 2)... (i + j). First note that Ī 3 (G j (i, fac)) implies G j (i, fac), i.e. G j (i, fac) = (i + j, fac ) and i + j 1 < n. Furthermore Ī 3 (i + j, fac ) implies i + j n. From i + j 1 < n and i + j n we immediately get i + j = n, or j = n i. Hence k(i, fac) = n i, i.e. H(i, fac) = G n i (i, fac) = (n, fac (i + 1) (i + 2)... n). This means so F (i, fac) = H(1, 1) = (n, n!), n IN. factorial(n) = n! Ryszard Janicki Computability and Complexity 48 / 49

Programs as Automata and Relational Expression To make this technique feasible for bigger, more realistic programs, we need a tool that would be able to do all those symbolic calculations. The reasoning presented above rely heavily on Lemma 1(3) and is rather typical for human beings. Many steps and observations are not easy to mechanize. Nevertheless, this technique has most likely better prospects to eventually lead to almost automatic theorem provers (at least for some special kind of programs), than for example well known Hoare Logic. On the other hand, for human beings skillful in nding loop invariant, Hoare Logic is probably more convenient. Ryszard Janicki Computability and Complexity 49 / 49