Ambiguity, Precedence, Associativity & Top-Down Parsing. Lecture 9-10

Transcription

1 Ambiguity, Precedence, Associativity & Top-Down Parsing Lecture 9-10 (From slides by G. Necula & R. Bodik) 2/13/2008 Prof. Hilfinger CS164 Lecture 9 1

2 Administrivia Team assignments this evening for all those not listed as having one. HW#3 is now available, due next Tuesday morning (Monday is a holiday). 2/13/2008 Prof. Hilfinger CS164 Lecture 9 2

3 Remaining Issues How do we find a derivation of s? Ambiguity: what if there is more than one parse tree (erpretation) for some string s? rrors: what if there is no parse tree for some string s? Given a derivation, how do we construct an abstract syntax tree from it? 2/13/2008 Prof. Hilfinger CS164 Lecture 9 3

4 Ambiguity Grammar + * ( ) Strings + + * + 2/13/2008 Prof. Hilfinger CS164 Lecture 9 4

5 Ambiguity. xample The string + + has two parse trees is left-associative 2/13/2008 Prof. Hilfinger CS164 Lecture 9 5

6 Ambiguity. xample The string * + has two parse trees + * * + * has higher precedence than + 2/13/2008 Prof. Hilfinger CS164 Lecture 9 6

7 Ambiguity (Cont.) A grammar is ambiguous if it has more than one parse tree for some string quivalently, there is more than one rightmost or leftmost derivation for some string Ambiguity is bad Leaves meaning of some programs ill-defined Ambiguity is common in programming languages Arithmetic expressions IF-THN-LS 2/13/2008 Prof. Hilfinger CS164 Lecture 9 7

8 Dealing with Ambiguity There are several ways to handle ambiguity Most direct method is to rewrite the grammar unambiguously + T T T T * ( ) nforces precedence of * over + nforces left-associativity of + and * 2/13/2008 Prof. Hilfinger CS164 Lecture 9 8

9 Ambiguity. xample The * + has only one parse tree now + T * T + T * 2/13/2008 Prof. Hilfinger CS164 Lecture 9 9

10 Ambiguity: The Dangling lse Consider the grammar if then if then else OTHR This grammar is also ambiguous 2/13/2008 Prof. Hilfinger CS164 Lecture 9 10

11 The Dangling lse: xample The expression if 1 then if 2 then 3 else 4 has two parse trees if if 1 if 4 1 if Typically we want the second form 2/13/2008 Prof. Hilfinger CS164 Lecture 9 11

12 The Dangling lse: A Fix else matches the closest unmatched then We can describe this in the grammar (distinguish between matched and unmatched then ) MIF /* all then are matched */ UIF /* some then are unmatched */ MIF if then MIF else MIF OTHR UIF if then if then MIF else UIF Describes the same set of strings 2/13/2008 Prof. Hilfinger CS164 Lecture 9 12

13 The Dangling lse: xample Revisited The expression if 1 then if 2 then 3 else 4 if if 1 if 1 if A valid parse tree (for a UIF) Not valid because the then expression is not a MIF 2/13/2008 Prof. Hilfinger CS164 Lecture 9 13

14 Ambiguity Impossible to convert automatically an ambiguous grammar to an unambiguous one Used with care, ambiguity can simplify the grammar Sometimes allows more natural definitions But we need disambiguation mechanisms Instead of rewriting the grammar Use the more natural (ambiguous) grammar Along with disambiguating declarations Most tools allow precedence and associativity declarations to disambiguate grammars xamples 2/13/2008 Prof. Hilfinger CS164 Lecture 9 14

15 Associativity Declarations Consider the grammar + Ambiguous: two parse trees of Left-associativity declaration: %left + 2/13/2008 Prof. Hilfinger CS164 Lecture 9 15

16 Precedence Declarations Consider the grammar + * And the string + * * + + * 2/13/2008 Prof. Hilfinger CS164 Lecture 9 16 Precedence declarations: %left + %left *

17 How It s Done I: Intro to Top-Down Parsing Terminals are seen in order of appearance in the token stream: t 1 A B t 4 t 1 t 2 t 3 t 4 t 5 The parse tree is constructed C D From the top From left to right t 2 t 3 t 4 As for leftmost derivation 2/13/2008 Prof. Hilfinger CS164 Lecture 9 17

18 Top-down Depth-First Parsing Consider the grammar T + T T ( ) * T Token stream is: * Start with top-level non-terminal Try the rules for in order 2/13/2008 Prof. Hilfinger CS164 Lecture 9 18

19 Depth-First Parsing. xample * Start with start symbol Try T + T + Then try a rule for T ( ) () + But ( input ; backtrack to T + Try T. Token matches. + But + input *; back to T + Try T * T *T+ But (skipping some steps) + can t be matched Must backtrack to 2/13/2008 Prof. Hilfinger CS164 Lecture 9 19

20 Depth-First Parsing. xample * Try T Follow same steps as before for T And succeed with T * T and T With the following parse tree T * T 2/13/2008 Prof. Hilfinger CS164 Lecture 9 20

21 Depth-First Parsing Parsing: given a string of tokens t 1 t 2... t n, find a leftmost derivation (and thus, parse tree) Depth-first parsing: Beginning with start symbol, try each production exhaustively on leftmost non-terminal in current sentential form and recurse. 2/13/2008 Prof. Hilfinger CS164 Lecture 9 21

22 Depth-First Parsing of t 1 t 2 t n At a given moment, have sentential form that looks like this: t 1 t 2 t k A, 0 k n Initially, k=0 and A is just start symbol Try a production for A: if A BC is a production, the new form is t 1 t 2 t k B C Backtrack when leading terminals aren t prefix of t 1 t 2 t n and try another production Stop when no more non-terminals and terminals all matched (accept) or no more productions left (reject) 2/13/2008 Prof. Hilfinger CS164 Lecture 9 22

23 When Depth-First Doesn t Work Well Consider productions S S a a: In the process of parsing S we try the above rules Applied consistently in this order, get infinite loop Could re-order productions, but search will have lots of backtracking and general rule for ordering is complex Problem here is left-recursive grammar: one that has a non-terminal S S + Sα for some α 2/13/2008 Prof. Hilfinger CS164 Lecture 9 23

24 limination of Left Recursion Consider the left-recursive grammar S S α β S generates all strings starting with a β and followed by any number of α s Can rewrite using right-recursion S β S S α S ε 2/13/2008 Prof. Hilfinger CS164 Lecture 9 24

25 limination of left Recursion. xample Consider the grammar S 1 S 0 ( β = 1 and α = 0 ) can be rewritten as S 1 S S 0 S ε 2/13/2008 Prof. Hilfinger CS164 Lecture 9 25

26 More limination of Left Recursion In general S S α 1 S α n β 1 β m All strings derived from S start with one of β 1,,β m and continue with several instances of α 1,,α n Rewrite as S β 1 S β m S S α 1 S α n S ε 2/13/2008 Prof. Hilfinger CS164 Lecture 9 26

27 General Left Recursion The grammar S A α δ (1) A S β (2) is also left-recursive because S + S β α This left recursion can also be eliminated by first substituting (2) o (1) There is a general algorithm (e.g. Aho, Sethi, Ullman 4.3) But personally, I d just do this by hand. 2/13/2008 Prof. Hilfinger CS164 Lecture 9 27

28 An Alternative Approach Instead of reordering or rewriting grammar, can use top-down breadth-first search. S S a a String: aaa S S a a a S a a S a a (not all matched) a a a a a 2/13/2008 Prof. Hilfinger CS164 Lecture 9 28

29 Summary of Top-Down Parsing So Far Simple and general parsing strategy Left recursion must be eliminated first but that can be done automatically Or can use breadth-first search But backtracking (depth-first) or maaining list of possible sentential forms (breadthfirst) can make it slow Often, though, we can avoid both 2/13/2008 Prof. Hilfinger CS164 Lecture 9 29

30 Predictive Parsers Modification of depth-first parsing in which parser predicts which production to use By looking at the next few tokens No backtracking Predictive parsers accept LL(k) grammars L means left-to-right scan of input L means leftmost derivation k means need k tokens of lookahead to predict In practice, LL(1) is used 2/13/2008 Prof. Hilfinger CS164 Lecture 9 30

31 Recursive Descent: Grammar as Program In recursive descent, we think of a grammar as a program. Here, we ll look at predictive version. ach non-terminal is turned o a procedure ach right-hand side transliterated o part of the procedure body for its non-terminal First, define next() current token of input scan(t) check that next()=t (else RROR), and then read new token. 2/13/2008 Prof. Hilfinger CS164 Lecture 9 31

32 Recursive Descent: xample P S $ S T S S + S ε T ( S ) ($ = end marker) def P (): S(); scan($) def S(): T(); S () def S (): predict if next() == + : scan( + ); S() elif next() in [ ), $]: pass else: RROR def T(): if next () == : scan() elif next() == ( : scan( ( ); S(); scan ( ) ) else: RROR But where do prediction tests come from? 2/13/2008 Prof. Hilfinger CS164 Lecture 9 32

33 Predicting Right-hand Sides The if-tests are conditions by which parser predicts which right-hand side to use. In our example, used only next symbol (LL(1)); but could use more. Can be specified as a 2D table One dimension for current non-terminal to expand One dimension for next token A table entry contains one production 2/13/2008 Prof. Hilfinger CS164 Lecture 9 33

34 But First: Left Factoring With the grammar T + T T * T ( ) Impossible to predict because For T two productions start with For it is not clear how to predict A grammar must be left-factored before use for predictive parsing 2/13/2008 Prof. Hilfinger CS164 Lecture 9 34

35 Left-Factoring xample Starting with the grammar T + T T * T ( ) Factor out common prefixes of productions T X X + ε T ( ) Y Y * T ε 2/13/2008 Prof. Hilfinger CS164 Lecture 9 35

36 Recursive Descent for Real So far, have presented a purist view. In fact, use of recursive descent makes life simpler in many ways if we cheat a bit. Here s how you really handle left recursion in recursive descent, for S S A R: def S(): R () while next() FIRST(A): A() It s a program: all kinds of shortcuts possible. 2/13/2008 Prof. Hilfinger CS164 Lecture 9 36

37 LL(1) Parsing Table xample Left-factored grammar T X T ( ) Y X + ε Y * T ε The LL(1) parsing table ($ is a special end marker): * + ( ) $ T Y ( ) T X T X X + ε ε Y * T ε ε ε 2/13/2008 Prof. Hilfinger CS164 Lecture 9 37

38 LL(1) Parsing Table xample (Cont.) Consider the [, ] entry Means When current non-terminal is and next input is, use production T X T X can generate string with in front Consider the [Y,+] entry When current non-terminal is Y and current token is +, get rid of Y We ll see later why this is right 2/13/2008 Prof. Hilfinger CS164 Lecture 9 38

39 LL(1) Parsing Tables. rrors Blank entries indicate error situations Consider the [,*] entry There is no way to derive a string starting with * from non-terminal 2/13/2008 Prof. Hilfinger CS164 Lecture 9 39

40 Using Parsing Tables ssentially table-driven recursive descent, except We use a stack to keep track of pending nonterminals We reject when we encounter an error state We accept when we encounter end-of-input 2/13/2008 Prof. Hilfinger CS164 Lecture 9 40

41 LL(1) Parsing Algorithm initialize stack = <S,$> repeat case stack of <X, rest> : if T[X,next()] == Y 1 Y n : stack <Y 1 Y n rest>; else: error (); <t, rest> : scan (t); stack <rest>; until stack == < > 2/13/2008 Prof. Hilfinger CS164 Lecture 9 41

42 LL(1) Parsing xample Stack Input Action $ * $ T X T X $ * $ Y Y X $ * $ terminal Y X $ * $ * T * T X $ * $ terminal T X $ $ Y Y X $ $ terminal Y X $ $ ε X $ $ ε $ $ ACCPT 2/13/2008 Prof. Hilfinger CS164 Lecture 9 42

43 Constructing Parsing Tables LL(1) languages are those definable by a parsing table for the LL(1) algorithm such that no table entry is multiply defined Once we have the table Can create table-driver or recursive-descent parser The parsing algorithms are simple and fast No backtracking is necessary We want to generate parsing tables from CFG 2/13/2008 Prof. Hilfinger CS164 Lecture 9 43

44 Predicting Productions I Top-down parsing expands a parse tree from the start symbol to the leaves Always expand the leftmost non-terminal T + * + 2/13/2008 Prof. Hilfinger CS164 Lecture 9 44

45 Predicting Productions II Top-down parsing expands a parse tree from the start symbol to the leaves Always expand the leftmost non-terminal T * * T + + The leaves at any po form a string βaγ β contains only terminals The input string is βbδ The prefix β matches The next token is b 2/13/2008 Prof. Hilfinger CS164 Lecture 9 45

46 Predicting Productions III Top-down parsing expands a parse tree from the start symbol to the leaves Always expand the leftmost non-terminal T * * T + T + The leaves at any po form a string βaγ (A=T, γ=ε) β contains only terminals γ contains any symbols The input string is βbδ (b=) So Aγ must derive bδ 2/13/2008 Prof. Hilfinger CS164 Lecture 9 46

47 Predicting Productions IV Top-down parsing expands a parse tree from the start symbol to the leaves Always expand the leftmost non-terminal T * * T + T + So choose production for T that can eventually derive something that starts with 2/13/2008 Prof. Hilfinger CS164 Lecture 9 47

48 Predicting Productions V Go back to previous grammar, with T X X + ε T ( ) Y Y * T ε Here, YX must match + T Y X Since + doesn t start with *, can t use Y * T + 2/13/2008 Prof. Hilfinger CS164 Lecture 9 48

49 Predicting Productions V Go back to previous grammar, with T X X + ε T ( ) Y Y * T ε T X But + can follow Y, so we choose Y ε Y + 2/13/2008 Prof. Hilfinger CS164 Lecture 9 49

50 FIRST and FOLLOW To summarize, if we are trying to predict how to expand A with b the next token, then either: b belongs to an expansion of A Any A α can be used if b can start a string derived from α In this case we say that b First(α) or b does not belong to an expansion of A, A has an expansion that derives ε, and b belongs to something that can follow A (so S * βabω) We say that b Follow(A) in this case. 2/13/2008 Prof. Hilfinger CS164 Lecture 9 50

51 Summary of Definitions For b T, the set of terminals; α a sequence of terminal & non-terminal symbols, S the start symbol, A N, the set of non-terminals: FIRST(α) T { ε } b FIRST(α) iff α * b ε FIRST(α) iff α * ε FOLLOW(A) T b FOLLOW(A) iff S * A b 2/13/2008 Prof. Hilfinger CS164 Lecture 9 51

52 Computing First Sets Definition First(X) = { b X * bα} {ε X * ε}, X any grammar symbol. 1. First(b) = { b } for b any terminal symbol 2. For all productions X A 1 A n Add First(A 1 ) {ε} to First(X). Stop if ε First(A 1 ) Add First(A 2 ) {ε} to First(X). Stop if ε First(A 2 ) Add First(A n ) {ε} to First(X). Stop if ε First(A n ) Add ε to First(X) 3. Repeat 2 until nothing changes ( compute fixed po ) 2/13/2008 Prof. Hilfinger CS164 Lecture 9 52

53 Computing First Sets, Contd. That takes care of single-symbol case. In general: FIRST(X 1 X 2 X k ) = FIRST(X 1 ) FIRST(X 2 ) if ε FIRST(X 1 ) FIRST(X k ) if ε FIRST(X 1 X 2 X k-1 ) - { ε } unless ε FIRST(X i ) i 2/13/2008 Prof. Hilfinger CS164 Lecture 9 53

54 First Sets. xample For the grammar T X T ( ) Y X + ε Y * T ε First sets First( ( ) = { ( } First( T ) = {, ( } First( ) ) = { ) } First( ) = {, ( } First( ) = { } First( X ) = {+, ε } First( + ) = { + } First( Y ) = {*, ε } First( * ) = { * } 2/13/2008 Prof. Hilfinger CS164 Lecture 9 54

55 Computing Follow Sets Definition Follow(X) = { b S * β X b ω } 1. Compute the First sets for all non-terminals first 2. Add $ to Follow(S) (if S is the start non-terminal) 3. For all productions Y X A 1 A n Add First(A 1 ) {ε} to Follow(X). Stop if ε First(A 1 ) Add First(A 2 ) {ε} to Follow(X). Stop if ε First(A 2 ) Add First(A n ) {ε} to Follow(X). Stop if ε First(A n ) Add Follow(Y) to Follow(X) 4. Repeat 3 until nothing changes. 2/13/2008 Prof. Hilfinger CS164 Lecture 9 55

56 Follow Sets. xample For the grammar T X T ( ) Y Follow sets Follow( ) = {), $} Follow( X ) = {$, ) } Follow( Y ) = {+, ), $} Follow( T ) = {+, ), $} X + ε Y * T ε 2/13/2008 Prof. Hilfinger CS164 Lecture 9 56

57 Constructing LL(1) Parsing Tables Construct a parsing table T for CFG G For each production A α in G do: For each terminal b First(α) do T[A, b] = α If α * ε, for each b Follow(A) do T[A, b] = α 2/13/2008 Prof. Hilfinger CS164 Lecture 9 57

58 LL(1) Parsing Table xample T X T ( ) Y X + ε Y * T ε * + ( ) $ T Y ( ) T X T X X + ε ε Y * T ε ε ε Follow( ) = {), $} Follow( X ) = {$, ) } Follow( Y ) = {+, ), $} Follow( T ) = {+, ), $} First( T ) = {, ( } First( ) = {, ( } First( X ) = {+, ε } First( Y ) = {*, ε } 2/13/2008 Prof. Hilfinger CS164 Lecture 9 58

59 Notes on LL(1) Parsing Tables If any entry is multiply defined then G is not LL(1). This happens If G is ambiguous If G is left recursive If G is not left-factored And in other cases as well Most programming language grammars are not LL(1) There are tools that build LL(1) tables 2/13/2008 Prof. Hilfinger CS164 Lecture 9 59

60 Review For some grammars there is a simple parsing strategy Predictive parsing (LL(1)) Once you build the LL(1) table (or know the FIRST and FOLLOW sets), you can write the parser by hand: recursive descent. Next: a more powerful parsing strategy for grammars that are not LL(1) 2/13/2008 Prof. Hilfinger CS164 Lecture 9 60