Π( ) Π( ) Π( ) A ε. b b. (d) (c) (a) (b)

Transcription

1 Top-Down Parsing of Conjunctive Languages Alexander Okhotin Faculty of Computational Mathematics and Cyernetics, Moscow State University Astract. This paper generalizes the notion of a strong LL(k) context-free grammar for the case of conjunctive grammars and develops a top-down parsing algorithm for the resulting language family. A top-down parser of a conjunctive language attempts to construct a derivation of the input string, at each step using a nite lookahead to determine which grammar rule to apply. Fragments of formulae that form the derivation are stored in a treestructured pushdown. Two ways to implement top-down parsers are suggested: a tale-driven parser with a tree-structured pushdown and a recursive descent parser. Both techniques naturally extend their context-free counterparts. Keywords: Conjunctive grammar, parsing, top-down, tree-structured pushdown, recursive descent 1. Introduction Conjunctive grammars, introduced in [6, 7], generalize the contextfree grammars y augmenting the formalism of grammar rules with an explicit set-theoretic intersection operation. The importance of intersection for applications (in particular, linguistic applications) has long een noted, since languages are often dened using a set of independent constraints, which essentially means conjunction. Additionally, several non-context-free language constructs common to natural languages, such as multiple agreements and crossed agreements, are known to e representale as nite intersections of context-free languages. This provided motivation for the study of the intersection closure of context-free languages [3], with the intention to use tuples of context-free grammars to denote intersection of languages generated y these grammars, i.e. allow the use of intersection `on the top' of the classical context-free grammars. Conjunctive grammars use an entirely grammatical approach of handling intersection, where an extra intersection operation is added to the traditional union and concatenation, and no restriction is imposed on the use of this operation. The class of conjunctive grammars gives a certain increase in descriptive power over the context-free grammars, Present address: Department of Computing and Information Science, Queen's University, Kingston, Ontario, Canada K7L 3N6. c 2001 Kluwer Academic Pulishers. Printed in the Netherlands. top-down.tex; 14/10/2001; 21:09; p.1

2 2 Alexander Okhotin eing capale oth of denoting non-context-free language constructs (including those not in the intersection closure of context-free languages, e.g. the classical non-context-free language fwcwjw 2 fa; g g) and providing very succinct descriptions of some context-free languages. In addition, conjunctive grammars inherit several attractive properties of context-free grammars, such as the representation of a derivation in the form of a tree and the existence of ecient parsing algorithms. A way to construct a cuic-time and quadratic-space recognizer for an aritrary conjunctive language was given in [7]; this algorithm is a natural generalization of the classical context-free Cocke{Hasami{ Younger algorithm, as descried in [2]. Despite its polynomiality, the algorithm from [7] is still not very useful for many potential practical applications, where a less general ut faster algorithm would e more appealing. In addition, the requirement for a grammar to e in the inary normal form is also a very inconvenient constrain. There exist numerous linear-time parsing algorithms for various suclasses of the context-free grammars; the LL(k) [1, 4] and LR(k) [1, 2] algorithms and some of their variations are the most widely used among them. It turns out that the SLL(k) (strong LL(k)) context-free parsing method can e eectively extended for the case of conjunctive grammars. The construction and explanation of the resulting algorithm is the main concern of this paper. Section 2 gives a asic overview of the conjunctive grammars; in addition to the material taken from [6, 7], a network representation of the conjunctive formulae is introduced, which is used extensively in the rest of the paper. In Section 3, we consider a kind of nondeterministic context-free top-down parser and generalize it for the conjunctive case, giving a denition of the conjunctive top-down parser with a treestructured pushdown. The latter turns out to e deterministic for a certain suclass of conjunctive grammars we name conjunctive SLL(k) (&SLL(k) for short). It is proved that the exact parsing tale for such a parser cannot e algorithmically computed, and Section 4 develops a way to compute a certain `approximation' of this parsing tale, which could e used for a reasonaly large suclass of the conjunctive SLL(k) grammars. Section 5 suggests two dierent methods to implement conjunctive top-down parsing and investigates their complexity. 2. Preliminaries A conjunctive grammar [7] is dened similarly to a context-free grammar, with the only dierence that its rules may consist of multiple conjuncts. top-down.tex; 14/10/2001; 21:09; p.2

3 Top-Down Parsing of Conjunctive Languages 3 DEFINITION 1. A conjunctive grammar is a quadruple G = (; N; P; S), where and N are disjoint nite nonempty sets of terminal and nonterminal symols respectively; P is a nite set of grammar rules, each of the form A! 1 & : : : & n (A 2 N; n > 1; 8i i 2 ( [ N) ); (1) where the strings i are called conjuncts and, as we shall shortly see, there is no loss of generality in the assumption that they are distinct and their order is insignicant; S 2 N is a nonterminal designated as the start symol. Let V e the union of and N. We shall use three special symols: \(", \&" and \)"; it is assumed that none of them is in V. Dene V = [ N [ f\(", '&", ')"g. Let us also dene the set conjuncts(g) = fa! j there is a rule A! 1 & : : : & n 2 P; such that i = for some i (1 6 i 6 n)g (2) According to the semantics of the conjunctive grammars, which is dened elow, a rule of the form (1) indicates that the nonterminal on the left can derive any terminal string that can e derived from each of the conjuncts on the right. Similarly to a context-free grammar, a conjunctive grammar generates strings y the means of nondeterministic transformation of sentential forms, starting from the start symol and ending with a string comprised of terminal symols. However, the ojects eing transformed in course of such a derivation are more complex than in the context-free case: DEFINITION 2. Let G = (; N; P; S) e a conjunctive grammar. The set of conjunctive formulae F is the least suset of V that satises the following conditions: 1. [ N F F. 3. If A; B 2 F (where A 6=, B 6= ), then AB 2 F. 4. If A 1 ; : : : ; A n 2 F (where n > 1; A i may e ), then (A 1 & : : : &A n ) 2 F. In order to make the notation more clear, the empty sustrings appearing in accordance with (ii) will e written out explicitly, i.e. we shall write (A()&) rather than (A()&). top-down.tex; 14/10/2001; 21:09; p.3

4 4 Alexander Okhotin Conjunctive formulae can e naturally represented in the form of ipolar serial-parallel networks (-networks), dened inductively over the structure of a formula, where symols from [ N and the mentioned empty strings form individual arcs laeled with s 2 [ N [ fg, concatenation is represented y serial composition and conjunction is represented y parallel composition, as illustrated in Figure 1. s (a) ε () Π( ) Π( ) (c) (d) Γ( Π( ) ) Π( ) Figure 1. -networks corresponding to (a) s 2 [ N () (c) AB (d) (A 1& : : : &A n). Note that for every formula A there is exactly one corresponding -network (we shall denote it (A)), ut for every -network there are innitely many corresponding formulae (consider A, (A), ((A)), : : :, which all correspond to the same -network). With each path in a -network we shall associate a string over V formed y the laels on the arcs that constitute the path (where -arcs function as identity); paths connecting the poles shall e sometimes called just paths. See Figure 2 for an example of a -network corresponding to a formula. a c c c A c A ε d d d ε Figure 2. -network corresponding to a((c&(c&c)a)(a&)d&()c(d&d))(()). DEFINITION 3. Let G = (; N; P; S) e a conjunctive grammar. De- ne =), G the relation of one-step derivaility on the set of conjunctive formulae. 1. For any s 0 ; s 00 2 V and A 2 N, such that s 0 As 00 2 F, and for all A! 1 & : : : & n 2 P, s 0 As 00 G =) s 0 ( 1 & : : : & n )s 00 (3) top-down.tex; 14/10/2001; 21:09; p.4

5 Top-Down Parsing of Conjunctive Languages 5 2. (The Gluing Rule) For any s 0 ; s 00 2 V and w 2, such that s 0 (w& : : : &w)s 00 2 F (where the numer of ws is greater than zero), G s 0 (w& : : : &w)s 00 =) G s 0 ws 00 (4) {z } n>1 Let =) denote the reexive and transitive closure of e the relation of derivaility in exactly n steps. G =).Let G =) (n) DEFINITION 4. Let G = (; N; P; S) e a conjunctive grammar. The language of a formula is the set of all terminal strings derivale from the formula: L G (A) = fw 2 j A =) G wg. The language generated y the grammar is the language generated y formula S: L(G) = L G (S). We quote the following key theorem from [7]: THEOREM 1. Let G = (; N; P; S) e a conjunctive grammar. Let A 1 ; : : : ; A n ; B e formulae, let A 2 N, let u 2. Then, L G ((A 1 & : : : &A n )) = n\ i=1 L G (AB) = L G (A) L G (B) L G (A i ) L G (A) = [ fl G (( 1 & : : : & m )) j A! 1 & : : : & m 2 P g L G (u) = fug (5a) (5) (5c) (5d) Since the emptiness of the intersection of two given context-free languages is undecidale, the undecidaility of the emptiness prolem for conjunctive languages is immediately estalished. Let us also consider a restricted form of derivation, the derivation with postponed gluing, which is organized in the following way: rst, rules from P are applied until there are no nonterminals left in the formula; after that, the gluing rule is applied until all parentheses and conjunction signs are removed. P Formally, let =) e the relation of derivaility in one step of applying a rule from P, and =) & e the relation of derivaility in one G step of applying the gluing rule (oviously, =) equals =) P [ =)). & The derivations with postponed gluing are derivations of the form A 0 P P =) A 1 =) : : : P =) A m = B 0 & =) B 1 & =) : : : & =) B n = w 2 (6) It is quite clear that derivation with postponed gluing is equivalent in power to the unrestricted derivation, i.e. A G =) w holds i 9B 2 F top-down.tex; 14/10/2001; 21:09; p.5

6 6 Alexander Okhotin such that A =) P B =) & w; this can e formally proved y a direct construction of the derivation with postponed gluing (6) from an aritrary given derivation from A 0 to w. EXAMPLE 1. Consider the following grammar for the classical noncontext-free language fa n n c n j n > 0g: S! AP & QC A! aa j P! P c j Q! aq j C! cc j Following is the derivation with postponed gluing of the string w = ac: S P =) : : : P =) ((a())(()c)&(a())(c())) & =) : : : & =) ac (7) Each conjunctive derivation can e represented in the form of a tree with shared leaves. This tree is constructed inductively y the length of the derivation. 1. The tree corresponding to S is a single node laeled with S. 2. For every application of a rule A! 1 & : : : & n 2 P to a sentential form, the corresponding leaf of the tree (which is laeled with A) is relaeled with A! 1 & : : : & n and supplied with descendants corresponding to all symols from all 1 ; : : : ; n in the same way as a context-free derivation tree is modied upon application of a rule A! 1 : : : n. 3. For every application of the gluing rule, the leaves corresponding to the sustrings eing glued are similarly glued. Thus, terminal leaves of the resulting tree can possily have more than one incoming arc. We shall use notation 6n to denote fu j u 2 ; juj 6 ng. DEFINITION 5. L. Dene Let k > 0. Let e a nite nonempty alphaet, let F irst k (L) = (L \ 6k ) [ fu j u 2 k ; 9v 2 + : uv 2 Lg (8) Since for every singleton language L = fwg, F irst k (L) is also a singleton, we shall adopt notation F irst k (w) for the only string in F irst k (fwg). top-down.tex; 14/10/2001; 21:09; p.6

7 Top-Down Parsing of Conjunctive Languages 7 3. Parser with a Tree-Structured Pushdown In this section we introduce the notion of a nondeterministic parser with a tree-structured pushdown, and show that it is equivalent to the conjunctive derivation (as in Denition 3) in the sense that such a parser has at least one accepting computation on some input string if and only if the string elongs to the language generated y the source grammar The Context-Free Case Let us review the nondeterministic context-free top-down parsing and its deterministic case applicale to strong LL(k) grammars. We shall dene the algorithm and the SLL(k) grammars in a somewhat complicated way, ecause that will sustantially simplify the transition to the conjunctive case. A context-free top-down parser constructs the leftmost derivation of the input string, using a pushdown (in this case it is a conventional linear pushdown) to store the right parts of the sentential forms forming the derivation. Left parts of sentential forms are prexes of the input string; they are not eing held in memory. Formally, the instantaneous description (id) of the parser is a pair (u; ) (u 2, 2 V ), where u is the unread portion of the input string and is the contents of the pushdown. The parser starts from id (w; S), where w is the input string and S is the start symol of the grammar. At each step of the computation one of the following is eing done: 1. Apply a rule. If there is a nonterminal A 2 N at the top of the pushdown, i.e. id = (u; A), then choose (nondeterministically) some rule A! 2 P and move to id (u; ). 2. Advance. If there is a terminal symol a 2 at the top of the pushdown and the unread portion of the string also egins with a, i.e. id = (av; a), then move to id (v; ). 3. Accept. If oth the input string and the pushdown have een exhausted, i.e. id = (; ), then accept the string. 4. Reject. If none of (i)-(iii) is the case, then reject the string. It is easily seen that every computation of the top-down parser follows some leftmost derivation S =) u 1 1 =) : : : =) u n?1 n?1 =) w (8i ju i j 6 ju i+1 j); (9) top-down.tex; 14/10/2001; 21:09; p.7

8 8 Alexander Okhotin where i are eing stored in the pushdown, while u i are eing discarded. Each application of rule y the parser corresponds to one step of (9); advancing along the input string represents internal data management, where symols are rearranged etween i and u i, and ensures that u i are prexes of w. The only source of nondeterminism is the choice of the rule to apply to a nonterminal. SLL(k) parsers use k rst symols of the unread portion of the input string (the lookahead string) in order to `predict' the correct rule; their ehaviour is dened y the following function. DEFINITION 6 (SLL(k) parsing tale). Let k > 0. Let G = (; N; P; S) e a context-free grammar. Dene T SLL(k), a mapping from N 6k to 2 P. A rule A! 2 P is in T SLL(k) (B; u) i A = B and there are strings x; y; z 2, such that 1. S G =) xyz. 2. There is a derivation tree of xyz that contains a node laeled with A; this node's direct descendants form the string. 3. This node is an ancestor of those and only those terminal leaves that correspond to the symols forming y; that implies A =) =) y. 4. F irst k (yz) = u. This derivation tree is shown in Figure 3(a). (a) () A A α & &α 1 n α α1 α n x y z x y Figure 3. Derivation trees: (a) Context-free case () Conjunctive case. z A context-free grammar is said to e SLL(k) if jt SLL(k) (A; u)j 6 1 for all A 2 N and u 2 6k, i.e. the choice of rule is always deterministic; top-down.tex; 14/10/2001; 21:09; p.8

9 Top-Down Parsing of Conjunctive Languages 9 there exists an algorithm to compute T SLL(k) for a given grammar and thus determine whether it is SLL(k) The Conjunctive Case We shall employ asically the same idea for the top-down parsing of conjunctive languages. When a multiple-conjunct rule A! 1 & : : : & n (n > 2) is eing applied, the parser should try to derive the same sustring of the input string from each of the conjuncts. This can e achieved y using a tree-structured pushdown to organize the sequential computation so that derivations from each i were carried out independently and in parallel. Every next symol of the input string can e shifted only if the top symols of all the ranches are equal to each other and to the input symol. This ensures the equality of strings eing derived from the conjuncts. The tree-structured pushdown T is a tree with edges laeled with symols from [ N and vertices which can either e marked or not marked. Edges incident to the leaves are the top edges of the pushdown; the top edges are considered ordered. Edges can e thought of as directed from the leaves down to the root, each pointing to its parent. A marked vertex can e thought of as a vertex connected to the top of the pushdown with an imaginary -edge, acting as a `promise' to derive the null string from all the remaining ranches meeting at that vertex. An instantaneous description (id) of a parser with a tree-structured pushdown is a pair (u; T ), where u is the unread portion of the input string and T is the tree-structured pushdown. The initial id is (w; T S ), where w is the input string and T S is a tree comprised of two unmarked vertices connected with an edge laeled with the start symol S. At each step of the computation, the parser does one of the following: 1. Apply a rule. If there are top edges laeled with nonterminals, then consider the leftmost among these edges. Let A 2 N e the nonterminal it is laeled with. Choose (nondeterministically) some rule A! 1 & : : : & n 2 P. If the rule does not contain an epsilon conjunct, i.e. it is of the form A! s 11 : : : s 1l1 & : : : &s n1 : : : s nln (8i l i > 0; 8j s ij 2 V ), then replace the edge with n ranches, as shown in Figure 4(a). If there is an epsilon conjunct, i.e. the rule is of the form A! s 11 : : : s 1l1 & : : : &s n?1;1 : : : s n?1;ln?1 & (n > 1, l i > 0, s ij 2 V ), then top-down.tex; 14/10/2001; 21:09; p.9

10 10 Alexander Okhotin A s 11 s n1 a a a a X X s 1l 1 snl n (a) () Figure 4. One computation step: (a) Apply a rule () Advance. a) If n = 1 (the rule is A! ) and the edge corresponding to A is the only incoming edge at the vertex where it goes, then unmark that vertex (this means that if the parser has earlier promised to derive empty strings from all the ranches meeting at that vertex, then this promise has now een fullled); ) If n > 1 or there are other incoming edges at that vertex, then mark the vertex (to state that all the remaining ranches meeting here should derive ); c) Replace the edge with n? 1 ranches corresponding to nonepsilon conjuncts (see again Figure 4(a)). 2. Advance. If there are no marked vertices in the pushdown, all top edges are laeled with the same terminal symol a 2, and the current input symol is also a (u = av, v 2 ), then do the following: a) Mark all vertices having oth top edges and non-top edges among their incoming edges. ) Remove all top edges and all leaves. c) Assign u := v. This case is illustrated in Figure 4(). The requirement that there should e no marked vertices in the pushdown ensures that nothing ut could e derived from the ranches meeting at marked vertices, ecause the empty string is the only string that can e derived without reading the input. 3. Accept. If oth the input string and the tree-structured pushdown have een exhausted (i.e. the unread portion of the string is and the tree consists of the root vertex), then accept the string. 4. Reject. If none of (i)-(iii) is the case, then reject the string. top-down.tex; 14/10/2001; 21:09; p.10

11 Top-Down Parsing of Conjunctive Languages 11 For each conjunctive grammar G = (; N; P; S) and for each w 2 the corresponding parser with a tree-structured pushdown has an accepting computation on w i w 2 L(G). The following explanation of parser operation also serves as a sketch of the proof. First, we shall use the fact that for any formula s 1 As 2 and for any rule A! 1 & : : : & n 2 P, the network (s 1 ( 1 & : : : & n )s 2 ) can e constructed y replacing the arc corresponding to A in (s 1 As 2 ) y a sugraph (( 1 & : : : & n )). That means that the rst part of the derivation with postponed gluing can e performed on -networks. Second, we note that a formula containing no nonterminal symols generates either the empty language or a single string. This language can e determined y processing (B) from left to right and checking all paths connecting the poles of the network for equality. Thus we see that the second part of the derivation with postponed gluing can also e done y analyzing the -network that was constructed in course of the rst part of the derivation. Then, we oserve that it is possile to do network rewriting (which corresponds to the rst part of the derivation with postponed gluing) at the same time as checking whether the resulting formula will generate exactly the input string (this corresponds to the second part of the derivation with postponed gluing). The network is processed from left to right; all nonterminals encountered are replaced with sugraphs corresponding to rule odies; terminals are passed through in all paths at the same time, provided they all are equal to each other; -arcs are passed through whenever they are encountered. Finally, we note that there is no need to store the left (processed) part of the -network, ecause it is no longer used. It can also e proved that, since we always apply rules to the leftmost nonterminals, the right part of the network will always e a tree with shared leaves. The information aout leaf sharing is not used and therefore may e disregarded. ε ε ε a a ε E A B A B E Figure 5. A splitted -network and its right part as a tree-structured pushdown. We conclude that it suces to store the tree-structured pushdown, which is essentially the right part of the network, as shown in the example in Figure 5. top-down.tex; 14/10/2001; 21:09; p.11

12 12 Alexander Okhotin It can e oserved that the only source of nondeterminism in the topdown parsing is the choice of rule to apply to a nonterminal. In the same way as in the context-free prototype of our parser, we introduce the parsing tale that limits and possily eliminates this nondeterminism using a nite lookahead in the input string. DEFINITION 7 (Conjunctive SLL(k) parsing tale). Let k > 0. Let G = (; N; P; S) e a conjunctive grammar. Dene T &SLL(k), a mapping from N 6k to 2 P. A rule A! 1 & : : : & n 2 P is in T &SLL(k) (B; u) i A = B and there are strings x; y; z 2, such that 1. S G =) xyz. 2. There is a derivation tree of xyz that contains a node laeled with A! 1 & : : : & n. 3. This node is an ancestor of those and only those terminal leaves that correspond to the symols forming y; that implies A =) G ( 1 & : : : & n ) =) G y. 4. F irst k (yz) = u. This case is illustrated in Figure 3(). A conjunctive grammar is said to e strong conjunctive LL(k) (&SLL(k) for short) if for every A 2 N and u 2 6k it holds that jt &SLL(k) (A; u)j 6 1, i.e. the choice of rule is always deterministic. However, the undecidaility of the emptiness prolem for the conjunctive languages [7] easily implies the following result: LEMMA 1. For any k > 0 the tale T &SLL(k) is not computale. It is undecidale whether a given conjunctive grammar is &SLL(k). 4. Computation of Parsing Tales In this section we shall develop a practical method of parsing tale construction for conjunctive top-down parsers. The main idea is to compute some reasonale extension of T &SLL(k), i.e. a function T k : N 6k! 2 P such that for all A 2 N and u 2 6k it would hold that T &SLL(k) (A; u) T k (A; u); (10) top-down.tex; 14/10/2001; 21:09; p.12

13 Top-Down Parsing of Conjunctive Languages 13 and (10) would turn to equality for a suciently large class of grammars. We shall now dene the conjunctive analogs of the First k and Follow k functions; see [1] for the corresponding denitions for the context-free case. DEFINITION 8. Let k > 0. Let G = (; N; P; S) e a conjunctive grammar. Dene First k, a mapping from the set of conjunctive formulae to 6k : First k (A) = F irst k (L G (A)) (11) DEFINITION 9. Let k > 0.Let G = (; N; P; S) e a conjunctive grammar. Dene Follow k, a mapping from N to 6k. A string u 2 6k is in Follow k (A) i there exist strings x; y; z 2, such that 1. S G =) xyz. 2. There is a derivation tree of xyz (see again Figure 3()) that contains a node laeled with some rule A! 1 & : : : & n 2 P. 3. This node is an ancestor of those and only those terminal leaves that correspond to symols forming y. 4. F irst k (z) = u. Both First k and Follow k are clearly incomputale (since First k (S) = ; i First k (S) = ; i L(G) = ;). Again, we shall try to construct some reasonale extensions of these functions. The functions Pfirst k and Pfollow k (which stand for potential First and Follow) dene the terminal strings which either (Pfirst) start all paths in the -network of some formula derived from a given formula, or (Pfollow) follow a given nonterminal in all paths in the -network of some formula derived from S. Memership in Pfirst and Pfollow is estalished on the asis of incomplete derivations; these derivations do not necessarily have continuations leading to terminal strings, ut if they have, then the strings thus proved to e in Pfirst (Pfollow) will e in First (Follow). Pfirst and Pfollow are formally dened y the following algorithms which simulate such incomplete derivations. top-down.tex; 14/10/2001; 21:09; p.13

14 14 Alexander Okhotin DEFINITION 10. Let G = (; N; P; S) e a grammar, let k > 0. Pfirst k is a mapping from F to 6k, dened as follows: Pfirst k (w) = F irst k (fwg) (w 2 ) (12a) Pfirst k (AB) = F irst k (Pfirst k (A) Pfirst k (B)) (A; B 2 F) (12) Pfirst k ((A 1 & : : : &A n )) = n\ i=1 Pfirst k (A i ) (n > 0; A i 2 F) (12c) Pfirst k (A) = (as computed y Algorithm 1) (A 2 N) (12d) ALGORITHM 1. For given G and k > 0 compute Pfirst k (A) for all A 2 N. let Pfirst k (A) = ; for all A 2 N; while new strings can e added to Pfirst k (A) for 8A! s 11 : : : s 1n1 & : : : &s m1 : : : s mnm 2 P Pfirst k (A) = Pfirst k (A) [ [ T m i=1 F irst k (Pfirst k (s i1 ) : : : Pfirst k (s ini )); The function Pfollow k algorithm: : N! 6k is dened y the following ALGORITHM 2. For given G and k > 0 compute Pfollow k (A) for all A 2 N. let Pfollow k (S) = fg; let Pfollow k (A) = ; for all A 2 N n fsg; while new strings can e added to Pfollow k (A) for 8B! 2 conjuncts(p ) for each factorization = A, where ; 2 V and A 2 N Pfollow k (A) = Pfollow k (A) [ [ F irst k (Pfirst k () Pfollow k (B)); The following two lemmas show that Pfirst k extend First k and Follow k respectively. and Pfollow k LEMMA 2. For any grammar G = (; N; P; S) and for any formula A 2 F, First k (A) Pfirst k (A). Proof. It suces to prove that for any A 2 N, w 2 and n > 0, such that A =) G (n) w, it holds that F irst k (w) 2 Pfirst k (A). The proof in an induction on n. Basis n = 2. There is a rule A! w 2 P and the derivation is of the form A =) (w) =) w. Clearly, F irst k (w) is added to Pfirst k (A) at the rst iteration of Algorithm 1. top-down.tex; 14/10/2001; 21:09; p.14

15 Top-Down Parsing of Conjunctive Languages 15 Induction Step. Let A =) (n+2) w, then for some rule A! s 11 : : : s 1p1 & : : : &s m1 : : : s mpm 2 P (where s ij 2 V, p i > 0) A =) (s 11 : : : s 1p1 & : : : &s m1 : : : s mpm ) =) : : : =) =) (u 11 : : : u 1p1 & : : : &u m1 : : : u mpm ) =) w u 11 : : : u 1p1 = : : : = u m1 : : : u mpm = w (13a) (13) s ij =) (l ij ) u ij (8i 2 [1; m] 8j 2 [1; p i ]) (13c) X i;j l ij = n (13d) Consider some i: 1 6 i 6 m. For each j (1 6 j 6 p i ), the statement F irst k (u ij ) 2 Pfirst k (s ij ) (14) follows from the induction hypothesis if s ij Denition 10 if s ij 2. By concatenating (14) for all 1 6 j 6 p i, we otain 2 N and directly from F irst k (u i1 ) : : : F irst k (u ipi ) 2 Pfirst k (s i1 ) : : : Pfirst k (s ipi ) (15) and hence F irst k (w) 2 F irst k (Pfirst k (s i1 ): : :Pfirst k (s ipi )), which implies that F irst k (w) is added to Pfirst k (A) at some iteration of Algorithm 1. For the case of A =2 N, the proof easily follows from Theorem 1 and Denition 10 and is omitted. 2 LEMMA 3. For any grammar G = (; N; P; S) and for any nonterminal A 2 N, Follow k (A) Pfollow k (A). Proof. Let A 2 N and u 2 Follow k (A), then there exist x; y; z 2 that satisfy the conditions listed in Denition 9. Let S = A 0 =) G A G 1 =) : : : G =) A G m =) : : : G =) A n = xyz (16) e the derivation from Denition 9, where A m is the formula where the instance of A mentioned in denition Denition 9 appears for the rst time.our proof is an induction on m. Basis m = 0. Then A = S, z = and thus u =. is added to Pfollow k (S) y the rst statement of Algorithm 2. Induction Step. Let m > 0. Then there exist some B 2 N; B! 1 & : : : & p 2 P ; s 1 ; s 2 2 V ; ; 2 V and i 2 [1; p], such that A m?1 = s 1 Bs 2 A m = s 1 ( 1 & : : : & p )s 2 i = A; (17a) (17) (17c) top-down.tex; 14/10/2001; 21:09; p.15

16 16 Alexander Okhotin where the A in (17c) is the A from Denition 9. Let us change the order of steps in derivation (16), so that the derivation from would e carried out rst. This is possile, ecause the derivation (16) is known to e successful. We get G s 1 ( 1 & : : : &A& : : : & p )s 2 =) s 1 ( 1 & : : : &Az 0 & : : : & p )s 2 (18) for some z 0 2. It can e proved that this implies z = z 0 z 00 for some z By the denition of First, F irst k (z 0 ) 2 First k (); therefore, y Lemma 2, F irst k (z 0 ) 2 Pfirst k (). By the denition of Follow, F irst k (z 00 ) 2 Follow k (B) due to the factorization w = x yz 0 z 00 and the same derivation (16). Since B rst appears at the step m?1 of the derivation, y the induction hypothesis we otain F irst k (z 00 ) 2 Pfollow k (B). We conclude that u 2 F irst k (Pfirst k () Pfollow k (B)), which implies that u is added to Pfollow k (A) at some iteration of Algorithm 2. 2 Let us now consider a parsing tale computed from some functions f and g serving as First and Follow: DEFINITION 11. Let k > 0. Let G e a conjunctive grammar. Let f : F! 6k. Let g : N! 6k. Dene T k;f;g, a mapping from N 6k to 2 P. A rule A! 1 & : : : & n is in T k;f;g (B; u) i A = B and u 2 F irst k (f(( 1 & : : : & n )) g(a)) (19) Lemmas 2 and 3 allow us to estalish the following result, which justies the use of Pfirst k and Pfollow k for the computation of the parsing tales: THEOREM 2. For any conjunctive grammar G = (; N; P; S) and k > 0, T k;pfirst k;pfollowk is a consistent top-down parsing tale, i.e. for all A 2 N and u 2 6n T &SLL(k) (A; u) T k;pfirst (A; u) (20) k;pfollowk Proof. Consider a nonterminal A 2 N, a lookahead string u 2 6k and a rule A! 1 & : : : & m 2 P, such that A! 1 & : : : & m 2 T &SLL(k) (A; u). Let x; y; z 2 e the strings mentioned in Denition 7. Since ( 1 & : : : & m ) =) y, we get that F irst k (y) 2 First k (( 1 & : : : & m )) and thus, y Lemma 2, F irst k (y) 2 Pfirst k (( 1 & : : : & m )) (21) top-down.tex; 14/10/2001; 21:09; p.16

17 Top-Down Parsing of Conjunctive Languages 17 On the other hand, the derivation tree under consideration satises Denition 9. We have F irst k (z) 2 Follow(A) and, y Lemma 3, We concatenate (21) and (22) to get F irst k (z) 2 Pfollow k (A) (22) u 2 F irst k (Pfirst k (( 1 & : : : & m )) Pfollow k (A)); (23) which proves the theorem. 2 EXAMPLE 2. Consider the grammar for L=fa n n c n j n > 0g from Example 1. Its Pfirst 1, Pfollow 1 and the corresponding parsing tale are shown in Tale I; it follows that the grammar is &SLL(1). Tale I. Pfirst 1, Pfollow 1 and T 1;Pfirst1 ;Pfollow1 for grammar from Example 1. T 1;Pfirst1 ;Pfollow1 Pfirst 1 Pfollow 1 a c S f; ag fg S! AP &QC S! AP &QC?? A f; ag f; g A! A! aa A!? P f; g f; cg P!? P! P c P! Q f; ag f; ; cg Q! Q! aq Q! Q! C f; cg fg C!?? C! cc The computation of the top-down parser for this grammar on the input string w = ac is shown in Figure 6. S A P Q C a Q a a A A Q P C P C A P Q C P Q C P c Q C P c C P c C c C c c C C Acc. Figure 6. Example of parsing This example shows that the &SLL(k) grammars can denote some non-context-free languages and thus have greater expressive power than the context-free LL(k) grammars. top-down.tex; 14/10/2001; 21:09; p.17

18 18 Alexander Okhotin Actually, a more general result concerning the scope of &SLL(k) languages can easily e estalished: THEOREM 3. For any k > 0, the family of &SLL(k) languages properly contains all context-free LL(k) languages and is closed under intersection. Let us also note that for each n > 1 there are &SLL(1) languages that cannot e expressed as an intersection of n? 1 context-free languages; that is, for instance, the language L n = fa i 1 1 ai 2 2 : : : ain n ai 1 1 ai 2 2 : : : ain n j 8j i j > 0g; (24) over the alphaet fa 1 ; : : : ; a n g, studied in [5]. Since it is an intersection of n context-free LL(1) languages, it is &SLL(1) y Theorem 3. Therefore, the family of &SLL(1) languages coincides with each sufamily of the innite hierarchy of intersections of context-free languages studied in [5]. It remains an open prolem whether &SLL(k) grammars can generate any conjunctive language, or at least if there exists any &SLL(k) language not in the intersection closure of context-free languages. 5. Implementation The model top-down parser with a tree-structured pushdown, introduced in Section 3, can e implemented in such a way that the numer of machine operations will e proportional to the numer of high-level operations. Each vertex of the tree-structured pushdown could e represented as a quadruple (s; p; n; ), where s 2 [ N is the symol on the arc leading down, p is the pointer to the destination vertex of that arc (for the root, p equals null and s is undened), n is the numer of incoming arcs, and 2 ftrue; falseg indicates whether this vertex is marked. In addition, we should hold the list of top arcs, each represented as a pair (s; p) (s 2 [ N; p is a pointer to some vertex), the pointer to the top arc eing processed and the counter of marked vertices. At each step of computation, the parser does not need to rowse the whole tree-structured pushdown: it suces to oserve the current top arc in the list and either (if it is a nonterminal arc) replace it with the ody of some rule, or (if it is a terminal arc, the terminal equals the current input symol and there are no marked vertices) proceed to the next top arc. When the end of the list is reached, it is already known that all top arcs are equal to the input symol and thus can e removed along with reading the next input symol. top-down.tex; 14/10/2001; 21:09; p.18

19 Top-Down Parsing of Conjunctive Languages 19 We see that each step of the computation (which corresponds to one step of the derivation) requires constant time, with the exception of the removal of the top arcs, which, however, cannot take more time than the application of the grammar rules, ecause every top arc requires an elementary operation to appear. Therefore, the complexity of the algorithm linearly depends on the length of the derivation. We shall also consider an alternative approach to the conjunctive top-down parsing, the recursive descent. This technique, which is frequently used for the top-down parsing of context-free languages, can e straightforwardly extended for the conjunctive case. Let G = (; N; P; S) e an &SLL(k) grammar, let k > 0, let T k : N 6k! 2 P e some function such that T &SLL(k) (A; u) T k (A; u) for all A 2 N and for all u 2 6k. For every terminal symol a 2 dene a procedure a(int &pointer) /* pointer is passed y reference */ f if(w[pointer]==a) pointer++; else reject(); g For every nonterminal symol A 2 N dene a procedure A(int &pointer) /* pointer is passed y reference */ f switch(t k (A; w + pointer)) f case A! s 11 : : : s 1l1 & : : : &s m1 : : : s mlm : int p[1::m]=f pointer, : : :, pointer g; for i = 1 to m /* may e executed in parallel */ for j = 1 to l i s ij (p[i]); if(not (8i; j p[i] == p[j])) reject(); pointer = p[1]; reak; case A! : : : :. default: /* if T k (A; w + pointer) is empty */ reject(); g g top-down.tex; 14/10/2001; 21:09; p.19

20 20 Alexander Okhotin Since the iterations of the outer loops in A() (A 2 N) are data-independent, these loops may e executed in parallel, i.e. their iterations may e distriuted etween processors of a multiprocessor system to speed up the algorithm. Exactly like in the context-free case, this algorithm could e modied to construct the derivation tree of the string, if each procedure creates its sutree and returns it to its caller. This parser is deterministic i jt k (A; u)j 6 1 for all A 2 N and u 2 6k. Its correctness is stated in the following theorem: THEOREM 4. For each grammar G and string w, w 2 L(G) if and only if there is a computation of S() that moves the pointer from 0 to jwj and terminates without making a call to reject(). The proof of Theorem 4 follows from two auxilliary statements: LEMMA 4. Let w = xyz. If there is a computation of s() (s 2 [ N) that moves the pointer from jxj to jxyj without making a call to reject(), then s =) y. Proof. Induction on the height of the tree of procedure calls. Basis: no procedure calls. Either s = a 2 and y = a, or s = A 2 N, y = and we choose the rule A! ; oth cases are trivial. Induction step. If s() makes procedure calls, then s = A 2 N. Some rule A! s 11 : : : s 1l1 & : : : &s n1 : : : s nln 2 P is selected at the rst step of the computation. Then, for each i 2 [1; n], the procedures s i1 () : : : s ili () are consequently called and each of them returns; let p ij e the value of pointer after the return from each s ij (), and let p i0 = jxj. Since our main procedure terminates and moves the pointer to jxyj, we have p 1l1 = : : : = p 1ln = jxyj. Let w e a 1 : : : a jwj. By induction hypothesis, s ij =) a pi;j?1 +1 : : : a pij (25) For all i we concatenate (25) y j to get s i1 : : : s ili =) y, which, using the previously chosen rule for A, allows to construct a derivation from A to y. 2 LEMMA 5. For any k > 0, for any conjunctive grammar G = (; N; P; S), for any w 2 L(G), for any derivation of w, for any symol s 2 [ N, for any node in the corresponding derivation tree that is laeled with s (in case s 2 ) or with some rule for s (in case s 2 N), such that there exist x; y; z 2, where w = xyz, and the mentioned node is an ancestor of those and only those terminal leaves that correspond to symols forming y, there is a computation of s() that moves the pointer from jxj to jxyj without making a call to reject(). top-down.tex; 14/10/2001; 21:09; p.20

21 Top-Down Parsing of Conjunctive Languages 21 Proof. The proof is an induction over the numer of nonepsilon descendants of the mentioned node of the derivation tree. Basis: no nonepsilon descendants. There are two cases to consider: 1. s = a 2 : this case is ovious. 2. s = A 2 N, the node is laeled with the rule A! 2 P and y =. By Denition 7, A! 2 T &SLL(k) (A; F irst k (z)), which proves the case. Induction step. Let s = A 2 N and let A! 1 & : : : & n 2 P e the rule this node is laeled with; like in the previous case, it is immediately estalished that A! 1 & : : : & n 2 T &SLL(k) (A; F irst k (yz)). For all i 2 [1; n] let j i j = l i and i = s i1 : : : s ili. There is a factorizationy = u i1 : : : u ili, where s ij =) u ij. By induction hypothesis, for all i and j there is a computation of s ij () that moves pointer from jxu i1 : : : u i;j?1 j to jxu i1 : : : u i;j?1 u ij j. Therefore, for each i, the corresponding ranch of the computation shall move pointer from jxj to jxyj, which implies that the nal condition will e met and A() shall successfully terminate, setting pointer to jxyj. 2 The time complexity of the recursive descent algorithm is again linearly dependent on the numer of steps in derivation. It is known that for every context-free grammar the numer of steps in the shortest derivation of a given string is ounded y some linear function of the length of the string. However, that is not the case for conjunctive grammars, where the length of derivation can e a higherdegree polynomial or even an exponent, as shown in [6]. For instance, the grammar S! AS & BS j a A! a B! a produces exponential-length derivations (and, esides, it is &SLL(1)). The following sucient condition of linear time complexity could e devised: PROPOSITION 1. Let G = (; N; P; S) e a &SLL(k) grammar (k > 0). If each nonterminal A 2 N that has multiple-conjunct rules is acyclic (i.e. there is no s 1 As 2 2 F, where s i 2 V, such that A =) (s) 1As 2 ), then the computation time of the corresponding &SLL(k) top-down parser is linearly ounded y the length of the input string. top-down.tex; 14/10/2001; 21:09; p.21

22 22 Alexander Okhotin Concluding Remarks It was shown that the context-free SLL(k) algorithm can e extended to the case of conjunctive grammars in a way very similar to how the conjunctive grammars themselves are formed from the context-free grammars: asically, y changing the data structures from strings to serial-parallel graphs, in which parallel composition denotes set-theoretic intersection. Some asic properties of the resulting class of &SLL(k) languages were addressed; however, no technique of proving that some conjunctive language is not &SLL(k) is known so far, which makes it hard to determine the exact scope of this language family and estalish quite proale nonclure results. Many theoretical prolems concerning the family of &SLL(k) languages remain uninvestigated: for instance, do the &SLL(k) languages form an innite hierarchy, as the context-free LL(k) languages do? Can every &SLL(k) language e represented as a nite intersection of context-free languages? Is the family of &SLL(k) languages closed under other operations esides intersection? Are there conjunctive languages that are not &SLL(k) for any k? But, independently of these yet unsolved theoretical issues, the algorithm introduced in this paper allows to extend traditional context-free top-down syntactical analysis to a suclass of conjunctive grammars, thus meeting the needs of the applications that could make use of an additional intersection operation within an existing parsing framework. References 1. A. V. Aho and J. D. Ullman, The Theory of Parsing, Translation and Compiling, Vol. I: Parsing, Prentice-Hall, M. A. Harrison, Introduction to formal language theory, Addison-Wesley, Reading, Mass., M. Latta, R. Wall, \Intersective context-free languages", Lenguajes Naturales y Lenguajes Formales IX, Barcelona, 1993, 15{ P. M. Lewis and R. E. Stearns. Syntax-directed transduction, Journal of the ACM, 15:3 (1968), 465{ L. Y. Liu and P. Weiner, \An innite hierarchy of intersections of context-free languages". Mathematical Systems Theory, 7 (1973) 187{ A. Okhotin, \Conjunctive grammars", in Pre-proceedings of DCAGRS 2000, Dept. of Computer Science, University of Western Ontario, London, Ontario, Canada. Report No. 555 (2000). 7. A. Okhotin, \Conjunctive grammars", full version of [6], Journal of Automata, Languages and Cominatorics,to appear. top-down.tex; 14/10/2001; 21:09; p.22