The Parser. CISC 5920: Compiler Construction Chapter 3 Syntactic Analysis (I) Grammars (cont d) Grammars

Transcription

1 The Parser CISC 5920: Compiler Construction Chapter 3 Syntactic Analysis (I) Arthur. Werschulz Fordham University Department of Computer and Information Sciences Copyright c Arthur. Werschulz, All rights reserved. Spring, 2017 Part of compiler s front end Tasks: Analyzes program structure Syntax error checking Supervise intermediate code generation Controls the lexer Parser design Based on formal language theory Uses context-free grammar, and so doesn t cover all issues 1 / 1 2 / 1 rammars rammars (cont d) enerative/descriptive, rather than proscriptive (Chomsky) Example: The tree has green leaves. Parse tree shows structure Need to have rules, comprising a grammar, to build parse tree: sentence noun phrase verb phrase noun phrase article noun noun phrase adjective noun verb phrase verb noun phrase noun tree, leaves, dog,... verb has, ate,... adjective green, old,... Example: a + b c. rammar: expression expression expression expression expression + expression expression a, b, c,... 3 / 1 4 / 1

2 rammars: Syntax vs. Semantics What s the difference? Syntactically correct semantically correct For natural languages, these grammars don t cover all grammatical sentences. rammars: Formal Definition A grammar has the form = (T, N, S, R), where T : finite set of terminal symbols N: finite set of nonterminal symbols Must have N T =. S N: unique start symbol R: set of productions α β, where α, beta (N T ) (sentential forms). Context-free grammar: Productions x β, where x N. Notation: α β if rule in R that transforms α into β. Omit where possible. If α, β (N T ) and finite sequence α α 1, α 1 α 2, α 2 α 3,... α n 1 β, then we write α β. 5 / 1 6 / 1 rammars: Formal Definition (cont d) Language generated by : { } L() = w T : S w Notational conventions: A, B, C... : non-terminals a, b, c,... : terminals w, x,... : strings of terminals α, β, γ... : sentential forms Boldface: programming language terminals Backus-Naur form (BNF) uses ::= instead of for alternatives Extended Backus-Naur form (EBNF) uses {... }: indefinite repetition (Kleene closure) [... ]: optional item rammars: Parse Trees and Derivations Simple expression grammar E = (T, N, S, R) where T = {i, +,,, /} N = {E} S = E R = {E E + E, E E E, E E E, E E/E, E (E), E i } Analyze (a b) (c + d) (i i) (i + i) E Draw parse tree Shorter E E E E (E) E (E + E) E (E + i) E (i + i) (E) (i + i) (E E) (i + i) (E i) (i + i) (i i) (i + i) 7 / 1 8 / 1

3 rammars: Rightmost, Leftmost Derivations Previous example: rightmost derivation Leftmost derivation of (a b) (c + d) (i i) (i + i)? E E E E E (E) (E E) E (i E) E (i i) E (i i) (E) (i i) (i + E) (i i) (i + i) Left vs. right sentential forms Different kinds of parsers yield the two derivations rammars: Ambiguous rammars Consider i i + i for our grammar E. Possible parsings? E E E i Ei i E + E i i + E i i + i and E E + E E + i E E + i E i + i i i + i Our grammar is ambiguous: both are correct! How to disambiguate? Look outside of grammar (operator hierarchy) Revise the grammar E E T E T T T T F T /F F F (E) i 9 / 1 10 / 1 rammars: The Chomsky Hierarchy rammars: The Chomsky Hierarchy (cont d) rammars of types 0, 1, 2, 3. Let L k = { L() : is of type k } Then L k L k 1. Type 0 (unrestricted, phrase-structure, semi-thue) grammars: Productions α β, where α, β (N T ). Typical form: γaδfebruary28, 2017γβδ for A N. γ, δ: left and right contexts of A Recognizable by Turing machine. Type 1 (context-sensitive) grammars: Productions: as in Type 0, but β ɛ. Recognizable by linear bounded automaton (special TM). Type 2 (context-free) grammars: Productions: A β where A N and β ɛ. Recognizable by non-deterministic stack automaton. NB: NDSA is not equivalent to DSA. Recognize deterministic context-free languages. This includes practical programming languages. Type 3 (regular) grammars: Productions: A a or A Bc, where A, B N, a, c T. Recognizable by FSA. 11 / 1 12 / 1

4 rammars: Some Examples rammars: Some Examples (cont d) L abc = { a n b n c n : n 0 }. Not a context-free language (why not?). rammar for L abc : L P = { legal (balanced) sets of parentheses }. Context-free grammar for L P : S (S) () L P L 2, but L P L 3. Hence L 3 L 2. S asbc abc CB BC bb bb bc bc cc cc 13 / 1 Example: Check aabbcc S asbc aabcbc aabbcc aabbcc aabbcc aabbcc L abc L 1, but L abc L 2. Hence L 2 L / 1 Top-Down Parsers Top-Down Parsers: Left Recursion Consider the grammar: Top-down parsers Start at the root of the parse tree. Leftmost derivation. Bottom-up parsers Start at leaves of the parse tree. Rightmost derivation. Must scan tokens from left to right. Entails some difficulties. Can be done with a set of recursive functions. Eventual goal: Table-driven top-down parser. Parse i i + i Top-down parse: E E + T T T T F F F i (E) E E + T E + T + T E + T + T... Problem? Left recursion, in this case E E + T (also, T T F ). Solution? Remove left-recursive productions from grammar, since top-down parser gets stuck. 15 / 1 16 / 1

5 Top-Down Parsers: Left Recursion (cont d) Top Down Parsers: Left Recursion (cont d) Immediate left-recursion: A Aα Non-immediate left recursion: A Bα, B Aβ et A Bα Aβα Bαβα Aαβα... Removing immediate left recursions? for each A N do begin separate A-left recursions A Aα 1 Aα 2 Aα 3... from non-left recursion A δ 1 δ 2 δ 3... introduce new nonterminal A replace non-left-recursive productions with A δ 1 A δ 2 A δ 3 A... replace left-recursive productions with A ɛ α 1 A α 2 A α 3 A... end Example: Suppose grammar consists of A Ab Ac d Note that L() = d(bc), Separate the productions: A Ab Ac A d Transformed: A da A ɛ ba ca 17 / 1 18 / 1 Top Down Parsers: Left Recursion (cont d) Removing non-immediate left recursion? for each production B whatever... do begin while productions B Aβ where A appears in an earlier-appearing production A γ 1 γ 2 γ 3... do replace B Aβ with B γ 1 β γ 2 β γ 3 β... end Top-Down Parsers: Left Recursion (cont d) Example: S A Sa; A Bb c; B Sd S: first on list. No non-immediate let recursion. But we have S Sa, fix by adding new non-terminal P: S A; S Sa becomes S AP; P ɛ ap A production: ok. B production: S appears earlier in list, so fix: Since we have B Sd with earlier S AP, the former becomes S APd. rammar is now S AP; P ɛ ap; A Bb C; B APd Change B APd to B BbPd; B cpd rammar is now S AP; P ɛ ap; A Bb C; B cpd; B BbPd 19 / 1 20 / 1

6 Top-Down Parsers: Left Recursion (cont d) Example (cont d): S A Sa; A Bb c; B Sd rammar was S AP; P ɛ ap; A Bb C; B APd We changed B APd to B BbPd; B cpd rammar is now S AP; P ɛ ap; A Bb C; B cpd; B BbPd Fix B BbPd: Introduce new non-terminal Q and so B cpd; B BbPd becomes B cpdq; q ɛ bpdq. Final grammar: S AP; P ɛ ap; A Bb c; B cpdq; Q ɛ bpdq Top-Down Parsers: Left Recursion (cont d) Example: Expression grammar E E + T T ; T T F F ; E i (E) E-productions: new non-terminal Q: E TQ; Q ɛ + TQ T -production: new non-terminal R F -production: no change Final grammar: T FR; R ɛ FR E TQ; Q ɛ + TQ; T FR R ɛ FR; F i (E) 21 / 1 22 / 1 Top-Down Parsers: Backtracking Carry out top-down parse by having parser try all possible derivations (brute-force method) This may lead to the wrong tree! Example: Suppose grammar is Token string: bcdc S ee bac baca Exhaustive search (brute force) first tries S bac bcac bcdc We wanted the derivation S bcde. d ca Need to backtrack. How? This could be painful! Need to put back input, deconstruct (part of) tree. Top-Down Parsing: Backtracking (cont d) Better idea: Use a better language design, keeping factor out of the common prefix (left factorization): S ee baq Q c e A d ca We now get proper derivation S baq bcaq bcdq bcde. 23 / 1 24 / 1

7 Recursive-Descent Parsing Try to avoid backtracking. Idea: Devote one function to recognize each nonterminal Example: rammar S aa b A csd e bool S() { if (token_is( a )&& A())) { cout << "S->aA\n"; return true; } if (token_is( b )) { cout << "S->b\n"; return true; } error( S ); return false; } Recursive-Descent Parsing (cont d) Example (cont d): rammar S aa b A csd e bool A() { if (token_is( c ) && S() && token_is( d )) { cout << "A->cSd\n"; return true; } if (tokan_is( e )) { cout << "A->e\n"; return true; } error( A ); return false; } 25 / 1 26 / 1 Recursive-Descent Parsing (cont d) Predictive Parsers Example (cont d): rammar Main program calls S(). S aa b A csd e Parse succeeds iff S() returns true and all input is used up, Do sample parse of token stream acbd (call treee and parse tree). Reamrks: This is a CF. Recognizing CFLs requires a stack. Where is it? In the recursion! Compare grammars S aa b A csd e and S Aa Bb A c da B e fb Second grammar production for S: neither RHS begins with nonterminal. 27 / 1 28 / 1

8 rammar is Suppose input string is feb. S Aa Bb A c da B e fb Proper parse: S Bb fbb feb. If recursive-descent parse starts with S Aa, then parser must... determine that this is a bad decsision, backtrack, start over from S Bb. How to avoid all this? Allow parser to look ahead in input stream. Decide which terminals are (leftmost-)derivaable from each nonterminal on RHS of production. Example (cont d): rammar is Parse tree: S Aa Bb S Aa Bb A c da B e fb daa ca fbb eb Input starts with c or d: use S Aa... FIRST(Aa) = {c, d}. Input starts with e or f : use S Bb... FIRST(Bb) = {e, f }. 29 / 1 30 / 1 Function S tries S Aa if input token is FIRST(Aa) = {c, d}. Function S tries S Bb if input token is FIRST(Bb) = {e, f }. In more detail, let = (T, N, S, R) be a grammar. For any sentential form α generated by : { FIRST(α) = a T : γ (N T ) such that α } aγ. For small grammars, can find FIRST sets by hand. Example: Find FIRST(E) for the grammar E TQ Q ɛ + TQ T FR R ɛ FR F i (E) We find E TQ FRQ (E)RQ... E TQ FRQ irq... and so FIRST(E) = {(, i}. But in general, we d like an algorithm to compute FIRST(α). 31 / 1 32 / 1

9 Predictiive Parsers (cont d) To help derive an algorithm, note the following: 1. If α = aβ, then FIRST(α) = {a}. 2. We say that α is nullable if α ɛ. If α is nullable, then ɛ FIRST(α). 3. FIRST(ɛ) = {ɛ}. 4. If α = Aβ for some A N, then FIRST(A) {ɛ} FIRST(α). Hidden trap in (4): Suppose α = ABδ where A is nullable. THen must follow up withg possibilities from B. Moreover, if B is nullable, must follow possiblities from δ. Example: Let S ABa A b c ɛ B d ɛ Here, A and B are nullable. We have Also, note that FIRST(S) = FIRST(ABa) = {a, b, c, d, e} ɛ FIRST(A) and ɛ FIRST(B), but ɛ FIRST(ABa), since ABa ɛ 33 / 1 34 / 1 Holub s formulation of the general rule: Let α = βx δ, where β N, X T or X is first non-nullable terminal. Then FIRST(α) = (FIRST(β) {ɛ}) FIRST(X ). If α = β (i.e., α is nullable), then FIRST(α) = FIRST(β). Predictive Parser (cont d) Algorithm for computing FIRST(α): if α T N {ɛ} then if α T then FIRST(α) = {α} else if α = ɛ then FIRST(α) = {ɛ} else // α N has form α β 1 β 2 β 3... FIRST(α) = k FIRST(β k) else // α = X 1 X 2... X n FIRST(α) = for (int j = 1; j n X j ɛ; j++) FIRST(α) = FIRST(X j ) Example (cont d): rammar is S ABa, A b c ɛ, B d ɛ. Both A and B are nullable, so FIRST(ABa) = {b, c} {d, e} {a} = {a, b, c, d, e} 35 / 1 36 / 1

10 Predictive Parser (cont d)... uses FIRST sets. Doesn t always work! Algorithm: Suppose A α β, where FIRST(α) FIRST(β). Do we choose α or β? Supppose grammar acquires ɛ-production A ɛ when removing left recursions. When do we choose A ɛ? if A = S then FOLLOW(A) $ for all productions Q xay do if y = qz for some q T then FOLLOW(A) = {q} else FOLLOW(A) = FIRST(y) {ɛ} // typo in book if y = ɛ y ɛ then FOLLOW(A) = FOLLOW(Q) Example: rammar E TQ; Q +TQ ɛ; T FR; R FR ɛ; F (E) i We have Of course FOLLOW sets? FIRST(E) = FIRST(T ) = FIRST(F ) = {(, i} FIRST(Q) = {+, ɛ} FIRST(R) = {, ɛ} FIRST(+TQ) = {+} FIRST((E)) = {(} FIRST( FR) = { } FIRST(i) = {i} FOLLOW(E) = {$, )} FOLLOW(Q) = {$, )} FOLLOW(T ) = {=, $, )} FOLLOW(R) = {+, $, )} FOLLOW(F ) = {+,, $, )} 37 / 1 38 / 1 How to use in a parsing function? Suppose production A rhs 1 rhs 2... rhs n Then bool A() { if next token() FIRST(rhs 1 ) then try rule A rhs 1 else if next token() FIRST(rhs 2 ) then try rule A rhs 2 }. else if next token() FIRST(rhs n ) then try rule A rhs n else if next token() FOLLOW(A) then error( A ); return false return true; 39 / 1 This is an example of an LL(1) grammar: L R token scan, Leftmost derivation, 1-char lookahead. A grammar is LL(1) if for all productions A α β: 1. ( FIRST(α) {ɛ} ) ( FIRST(β) {ɛ} ) =. 2. α ɛ = FIRST(β) FOLLOW(α) =. 40 / 1

11 A Predicitve Recursive-Descent Parser Consider the expression grammar Remove left recursion: E E + T T ; T T F F ; E i (E) Table-Driven Predictive Parsers Problem: Need one function per production. Brittle. Idea: Use general control procedure, specify grammar via table. Example: rammar E E + T T T T F F F (E) i E TQ; Q ɛ + TQ; T FR Remove left recursions: R ɛ FR; F i (E) E TQ; Q +TQ ɛ; T FQ; R FR ɛ; F (E) i See Pascal program on pp A few points: 1. Assume one-letter identifiers, to simplify lexing. 2. BACKUP() is needed if current token is in a FOLLOW set. 3. Functions Q, T (etc.) to recognize nonterminals Q, T, etc. Note the use of forward declarations. 41 / 1 The table for this grammar: Blank: error condition i + ( ) $ E TQ TQ Q +TQ ɛ ɛ T FR FR R ɛ FR ɛ ɛ F i (E) 42 / 1 Table-Driven Predicitve Parser (cont d) Control algorithm uses a stack named symstack: Constructing the Table symstack.push($) str = str + $ (start symbol, stack); while!symstack.empty() x = sysmstack.top(); a = incoming token; if x T then if x == a then sysmstack.pop(); a = next token() else error(); else if table[x,a] blank then symstack.pop() symstack.push(reverse(table[x,a])) else error() Trace execution on the strings i (i + i) and (i+) i. Draw parse tree... compare with Canonical parse tree. 43 / 1 Incoming token must tell us what to ddo next (avoid backtracking) Suppose sysmatck.top() = X and input token is a. Either RHS must begin with a, or RHS must lead to sentential form beginning with a. Example: Suppose input string is (... Need production E (, but no such production exists! However, have derivation E TQ FRQ (E)RQ Only one such path from E to ( Table takes this path Need to choose RHS α if token FIRST(α). But if ɛ FIRST(alpha), this won t work (no column headed ɛ), so use FOLLOW sets. 44 / 1

12 Constructing the Table (con td) Constructing the Table (cont d) Example: Our grammar is E TQ; Q +TQ ɛ; T FQ; R ɛfr ɛ; F (E) i Algorithm: for all productions X β do for all a FIRST() {ɛ} do table[x, a] = ɛ if β = ɛ) (ɛ FIRST(β)) do for all a FOLLOW(X ) do table[x, a] = ɛ. FIRST and FOLLOW sets? FIRST(E) = FIRST(T ) = FIRST(F ) = {i, (} FIRST(Q) = {+, ɛ} FIRST(R) = {, ɛ} FIRST(+TQ) = {+} FIRST( FR) = { } FOLLOW(E) = {$, )} FOLLOW(Q) = FOLLOW(E) = {$, )} FOLLOW(T ) = {+, ), $} FOLLOW(R) = FOLLOW(T ) = {+, ), $}Q FOLLOW(F ) = {+,, ), $} 45 / 1 46 / 1 Constructing the Table (cont d) Conflicts E TQ: FIRST(TQ) = FIRST(T ) = {i, (} = table[e, i] = table[e, (] = TQ Q +TQ ɛ = table[q, +] = +TQ For ɛ: FOLLOW(Q) = {$, )}impliestable[q, $] = table[q, )] = ɛ T FR: FIRST(FR) = FIRST(F ) = {i, (} = table[t, i] = table[t, (] = FR R FR ɛ = table[r, ] = FR For ɛ: FOLLOW(R) = {+, ), $} = table[r, +] = table[r, $] = table[r, )] = ɛ F (E) i = FIRST(F ) = {(, i} = table[r, +] = (E) table[f, i] = i What if table[x, a] has multiple entries? Trouble! Which one to pick? Maybe factorization helps. Example: A abe abf becomes A aq, Q be cf. Maybe not / 1 48 / 1

13 Conflicts (cont d) Example: The if-then-else grammar: S if E then S if E then S else S a b E x y Factorization yields Consider S if E then SQ a b E x y Q else S ɛ if x then if y then a else b Two parse trees, depending on where dangling else is attached: 1. Match else with first if. 2. Match else with most recently unmatched then (standard choice in most programming languages) 49 / 1 Conflicts (cont d) Example (cont d): FIRST, FOLLOW sets reflect this ambiguity Table? FIRST(Q) = {else, ɛ} FOLLOW(S) = {$, else} FOLLOW(Q) = {$, else} if x y then a b else $ S if E then S else S a b E x y Q else S ɛ ɛ 50 / 1