Bottom-up Analysis. Theorem: Proof: Let a grammar G be reduced and left-recursive, then G is not LL(k) for any k.

Transcription

1 Bottom-up Analysis Theorem: Let a grammar G be reduced and left-recursive, then G is not LL(k) for any k. Proof: Let A Aβ α P and A be reachable from S Assumption: G is LL(k) A n A S First k (αβ n γ) First k (αβ n+1 γ) = A β n β γ First k ( ) 117 / 292

2 Bottom-up Analysis Theorem: Let a grammar G be reduced and left-recursive, then G is not LL(k) for any k. Proof: Let A Aβ α P and A be reachable from S Assumption: G is LL(k) A n A S First k (αβ n γ) First k (αβ n+1 γ) = A β n β γ α First k ( ) 117 / 292

3 Bottom-up Analysis Theorem: Let a grammar G be reduced and left-recursive, then G is not LL(k) for any k. Proof: Let A Aβ α P and A be reachable from S Assumption: G is LL(k) A n A S First k (αβ n γ) First k (αβ n+1 γ) = A β n β γ A β α First k ( ) 117 / 292

4 Bottom-up Analysis Theorem: Let a grammar G be reduced and left-recursive, then G is not LL(k) for any k. Proof: Let A Aβ α P and A be reachable from S Assumption: G is LL(k) First k (αβ n γ) First k (αβ n+1 γ) = Case 1: β ǫ Contradiction!!! Case 2: β w ǫ == First k (αw k γ) First k (αw k+1 γ) 117 / 292

5 Shift-Reduce Parser Idea: We delay the decision whether to reduce until we know, whether the input matches the right-hand-side of a rule! Donald Knuth Construction: Shift-Reduce parser M R G The input is shifted successively to the pushdown. Is there a complete right-hand side (a handle) atop the pushdown, it is replaced (reduced) by the corresponding left-hand side 118 / 292

6 Shift-Reduce Parser Example: The pushdown automaton: S AB A a B b States: q 0, f, a, b, A, B, S; Start state: q 0 End state: f q 0 a q 0 a a ǫ A A b Ab b ǫ B AB ǫ S q 0 S ǫ f 119 / 292

7 Shift-Reduce Parser Construction: In general, we create an automaton M R G = (Q,T,δ,q 0,F) with: Q = T N {q 0,f} F = {f}; Transitions: (q 0,f fresh); δ = {(q,x,qx) q Q,x T} // Shift-transitions {(qα,ǫ,qa) q Q,A α P} // Reduce-transitions {(q 0 S,ǫ,f)} // finish Example-computation: (q 0, ab) (q 0 a, b) (q 0 A, b) (q 0 A b, ǫ) (q 0 AB, ǫ) (q 0 S, ǫ) (f, ǫ) 120 / 292

8 Shift-Reduce Parser Observation: The sequence of reductions corresponds to a reverse rightmost-derivation for the input To prove correctnes, we have to prove: (ǫ, w) (A, ǫ) iff A w The shift-reduce pushdown automaton MG R non-deterministic is in general also For a deterministic parsing-algorithm, we have to identify computation-states for reduction == LR-Parsing 121 / 292

9 Reverse Rightmost Derivations in Shift-Reduce-Parsers Idea: Observe reverse rightmost-derivations of M R G! Input: + 40 E 1 E 0 + T 1 Pushdown: ( q 0 T F ) T 0 F 2 T 1 F 2 F 1 Generic Observation: In a sequence of configurations of M R G name (q 0 αγ, v) (q 0 αb, v) (q 0 S, ǫ) we call αγ a viable prefix for the complete item [B γ ]. 122 / 292

10 Reverse Rightmost Derivations in Shift-Reduce-Parsers Idea: Observe reverse rightmost-derivations of M R G! Input: E 1 E 0 + T 1 Pushdown: ( q 0 E + F ) T 0 T 1 F 2 F 2 F 1 Generic Observation: In a sequence of configurations of M R G name (q 0 αγ, v) (q 0 αb, v) (q 0 S, ǫ) we call αγ a viable prefix for the complete item [B γ ]. 122 / 292

11 Bottom-up Analysis: Viable Prefix αγ is viable for [B γ ] iff S R αbv A 0 i 0 α 1 A 1 i 1 α 2 A 2 i 2 α m B i γ... with α = α 1... α m Conversely, for an arbitrary valid word α we can determine the set of all later on possibly matching rules / 292

12 Bottom-up Analysis: Admissible Items The item [B γ β] is called admissible for α iff S R αbv with α = αγ : A 0 i 0 α 1 A 1 i 1 α 2 A 2 i 2 α m B i γ β... with α = α 1... α m 124 / 292

13 Characteristic Automaton Observation: The set of viable prefixes from (N T) for (admissible) items can be computed from the content of the shift-reduce parser s pushdown with the help of a finite automaton: States: Items Start state: [S S] Final states: {[B γ ] B γ P} Transitions: (1) ([A α Xβ],X,[A αx β]), X (N T),A αxβ P; (2) ([A α Bβ],ǫ, [B γ]), A αbβ, B γ P; The automaton c(g) is called characteristic automaton for G. 125 / 292

14 Characteristic Automaton For example: E E+T 0 T 1 T T F 0 F 1 F ( E ) 0 2 S E E S E E E+T E + T E E +T E E+ T E E+T E T T E T T T F T F T T F T T F T T F T F F T F ( E ) F ( E ) F ( E ) F ( E ) F ( E ) F F 126 / 292

15 Canonical LR(0)-Automaton The canonical LR(0)-automaton LR(G) is created from c(g) by: 1 performing arbitrarily many ǫ-transitions after every consuming transition 2 performing the powerset construction... for example: E F 0 ( T ( F 3 ( F E 5 ( T * T 11 ) 9 7 * F / 292

16 Canonical LR(0)-Automaton Example: E E+T 0 T 1 Therefore we determine: T T F 0 F 1 F ( E ) 0 2 q 0 = {[S E], q 1 = δ(q 0,E) = {[S E ], {[E E+T], {[E E +T]} {[E T], {[T T F], q 2 = δ(q 0,T) = {[E T ], {[T F], {[T T F]} {[F (E)], {[F ]} q 3 = δ(q 0,F) = {[T F ]} q 4 = δ(q 0,) = {[F ]} 128 / 292

17 Canonical LR(0)-Automaton q 5 = δ(q 0, () = {[F ( E)], q 7 = δ(q 2, ) = {[T T F], {[E E+T], {[F (E)], {[E T], {[F ]} {[T T F], {[T F], q 8 = δ(q 5,E) = {[F (E )]} {[F (E)], {[E E +T]} {[F ]} q 9 = δ(q 6,T) = {[E E+T ], q 6 = δ(q 1,+) = {[E E+ T], {[T T F]} {[T T F], {[T F], q 10 = δ(q 7,F) = {[T T F ]} {[F (E)], {[F ]} q 11 = δ(q 8, )) = {[F (E) ]} 129 / 292

18 Canonical LR(0)-Automaton Observation: The canonical LR(0)-automaton can be created directly from the grammar. Therefore we need a helper function δǫ (ǫ-closure) We define: δ ǫ(q) = q {[B γ] [A α B β ] q, β (N T) : B Bβ} States: Sets of items; Start state: δ ǫ {[S S]} Final states: {q A α P : [A α ] q} Transitions: δ(q,x) = δ ǫ {[A αx β] [A α Xβ] q} 130 / 292

19 LR(0)-Parser Idea for a parser: The parser manages a viable prefix α = X 1...X m on the pushdown and uses LR(G), to identify reduction spots. It can reduce with A γ, if [A γ ] is admissible for α Optimization: We push the states instead of the X i in order not to process the pushdown s content with the automaton anew all the time. Reduction with A γ leads to popping the uppermost γ states and continue with the state on top of the stack and input A. Attention: This parser is only deterministic, if each final state of the canonical LR(0)-automaton is conflict free. 131 / 292

20 LR(0)-Parser... for example: q 1 = {[S E ], {[E E +T]} q 2 = {[E T ], q 9 = {[E E+T ], {[T T F]} {[T T F]} q 3 = {[T F ]} q 10 = {[T T F ]} q 4 = {[F ]} q 11 = {[F (E) ]} The final states q 1,q 2,q 9 contain more then one admissible item non deterministic! 132 / 292

21 LR(0)-Parser The construction of the LR(0)-parser: States: Q {f} (f fresh) Start state: q 0 Final state: f Transitions: Shift: (p,a,pq) if q = δ(p,a) Reduce: (pq 1...q m,ǫ,pq) if [A X 1...X m ] q m, q = δ(p,a) Finish: (q 0 p,ǫ,f) if [S S ] p with LR(G) = (Q,T,δ,q 0,F). 133 / 292

22 LR(0)-Parser Correctness: we show: The accepting computations of an LR(0)-parser are one-to-one related to those of a shift-reduce parser M R G. we conclude: The accepted language is exactly L(G) The sequence of reductions of an accepting computation for a word w T yields a reverse rightmost derivation of G for w 134 / 292

23 LR(0)-Parser Attention: Unfortunately, the LR(0)-parser is in general non-deterministic. We identify two reasons: Reduce-Reduce-Conflict: [A γ ], [A γ ] q with A A γ γ Shift-Reduce-Conflict: [A γ ], [A α aβ] q with a T for a state q Q. Those states are called LR(0)-unsuited. 135 / 292

24 Revisiting the Conflicts of the LR(0)-Automaton What differenciates the particular Reductions and Shifts? Input: E 1 E 0 + T 1 Pushdown: ( q 0 T ) E 1 T 0?? F 2 T 1 F 2 F 1 name E E+T 0 T 1 T T F 0 F 1 F ( E ) / 292

25 Revisiting the Conflicts of the LR(0)-Automaton What differenciates the particular Reductions and Shifts? Input: + 40 T 0? E 1 E 0 + T 1 Pushdown: ( q 0 T )? T 0 F 2 T 1 F 2 F 1 name E E+T 0 T 1 T T F 0 F 1 F ( E ) / 292

26 Revisiting the Conflicts of the LR(0)-Automaton Idea: Matching lookahead with right context matters! Input: E 1 E 0 + T 1 Pushdown: ( q 0 T ) E 1 T 0?? F 2 T 1 F 2 F 1 name E E+T 0 T 1 T T F 0 F 1 F ( E ) / 292

27 Revisiting the Conflicts of the LR(0)-Automaton Idea: Matching lookahead with right context matters! Input: Pushdown: ( q 0 T ) + 40 T 0? E 1? T 0 E 0 + T 1 F 2 T 1 F 2 F 1 name E E+T 0 T 1 T T F 0 F 1 F ( E ) / 292

28 LR(k)-Grammars Idea: Consider k-lookahead in conflict situations. Definition: The reduced contextfree grammar G is called LR(k)-grammar, if for First k (w) = First k (x) with: } S R αaw αβw S R α A w follows: α = α A = A w = x αβx Strategy for testing Grammars for LR(k)-property 1 Focus iteratively on all rightmost derivations S R αxw αβw 2 Identify handle αβ in s. forms αβw (w T, α,β (N T) ) 3 Determine minimal k, such that First k (w) associates β with a unique X β for non-prefixing αβs 137 / 292

29 LR(k)-Grammars for example: (1) S A B A aab 0 B abbb 1... is not LL(k) for any k but LR(0): Let S RαXw αβw. Then αβ is of one of these forms: A, B, a n aab, a n abbb, a n 0, a n 1 (n 0) (2) S aac A Abb b... is also not LL(k) for any k but again LR(0): Let S RαXw αβw. Then αβ is of one of these forms: ab, aabb, aac 138 / 292

30 LR(k)-Grammars for example: (3) S aac A bba b... is not LR(0), but LR(1): Let S R αxw αβw with {y} = First k(w) then αβy is of one of these forms: ab 2n bc, ab 2n bbac, aac (4) S aac A bab b... is not LR(k) for any k 0: Consider the rightmost derivations: S Rab n Ab n c ab n bb n c 139 / 292

31 LR(1)-Parsing Idea: Let s equip items with 1-lookahead Definition LR(1)-Item An LR(1)-item is a pair [B α β, x] with x Follow 1 (B) = {First 1 (ν) S µbν} 140 / 292

32 Admissible LR(1)-Items The item [B α β, x] is admissable for γα if: S RγBw with {x} = First 1 (w) S i 0 γ 0 A 1 i 1 γ 1 A mi m γ m B i α β x w... with γ 0...γ m = γ 141 / 292

33 The Characteristic LR(1)-Automaton The set of admissible LR(1)-items for viable prefixes is again computed with the help of the finite automaton c(g,1). The automaton c(g,1): States: LR(1)-items Start state: [S S, ǫ] Final states: {[B γ, x] B γ P,x Follow 1 (B)} Transitions: (1) ([A α Xβ, x],x,[a αx β, x]), X (N T) (2) ([A α Bβ, x],ǫ, [B γ, x ]), A αbβ, B γ P,x First 1 (β) 1 {x}; This automaton works like c(g) but additionally manages a 1-prefix from Follow 1 of the left-hand sides. 142 / 292

34 The Canonical LR(1)-Automaton The canonical LR(1)-automaton LR(G,1) is created from c(g,1), by performing arbitrarily many ǫ-transitions and then making the resulting automaton deterministic... But again, it can be constructed directly from the grammar; analoguously to LR(0), we need the ǫ-closure δ ǫ as a helper function: δ ǫ(q) = q {[C γ, x] [A α Bβ, x ] q,β (N T) : B Cβ x First 1 (ββ ) 1 {x }} Then, we define: States: Sets of LR(1)-items; Start state: δ ǫ {[S S, ǫ]} Final states: {q A α P : [A α, x] q} Transitions: δ(q,x) = δ ǫ {[A αx β, x] [A α Xβ, x] q} 143 / 292

35 The Canonical LR(1)-Automaton For example: E E+T 0 T 1 T T F 0 F 1 F ( E ) 0 2 First 1 (S ) = First 1 (E) = First 1 (T) = First 1 (F) = name,,( q 0 = {[S E, {ǫ}], q 3 = δ(q 0,F) = {[T F, {ǫ,+, }]} {[E E +T, {ǫ,+}], {[E T, {ǫ,+}], q 4 = δ(q 0,) {[F, {ǫ,+, }]} {[T T F, {ǫ,+, }], {[T F, {ǫ,+, }], q 5 = δ(q 0, () = {[F ( E ), {ǫ,+, }], {[F (E ), {ǫ,+, }], {[E E +T, {),+}], {[F,{ǫ,+, }]} {[E T, {),+}], {[T T F, {),+, }], q 1 = δ(q 0,E) = {[S E, {ǫ}], {[T F, {),+, }], {[E E +T, {ǫ,+}]} {[F (E ), {),+, }], {[F, {),+, }]} q 2 = δ(q 0,T) = {[E T, {ǫ,+}], {[T T F, {ǫ,+, }]} 144 / 292

36 The Canonical LR(1)-Automaton For example: E E+T 0 T 1 T T F 0 F 1 F ( E ) 0 2 First 1 (S ) = First 1 (E) = First 1 (T) = First 1 (F) = name,,( q 5 = δ(q 5, () = {[F ( E ), {),+, }], q 7 = δ(q 2, ) = {[T T F, {ǫ,+, }], {[E E +T, {),+}], {[F (E ), {ǫ,+, }], {[E T, {),+}], {[F, {ǫ,+, }]} {[T T F, {),+, }], {[T F, {),+, }], q 8 = δ(q 5,E) = {[F (E ),{ǫ,+, }]} {[F (E ), {),+, }], {[E E +T, {),+}]} {[F, {),+, }]} q 9 = δ(q 6,T) = {[E E +T, {ǫ,+}], q 6 = δ(q 1,+) = {[E E + T, {ǫ,+}], {[T T F, {ǫ,+, }]} {[T T F, {ǫ,+, }], {[T F, {ǫ,+, }], q 10 = δ(q 7,F) = {[T T F, {ǫ,+, }]} {[F (E ), {ǫ,+, }], {[F, {ǫ,+, }]} q 11 = δ(q 8, )) = {[F (E ), {ǫ,+, }]} 145 / 292

37 The Canonical LR(1)-Automaton For example: E E+T 0 T 1 T T F 0 F 1 F ( E ) 0 2 First 1 (S ) = First 1 (E) = First 1 (T) = First 1 (F) = name,,( q 2 q 3 q 4 = δ(q 5,T) = {[E T, {),+}], q 7 {[T T F, {),+, }]} = δ(q 9, ) = {[T T F, {),+, }], {[F (E ), {),+, }], {[F, {),+, }]} = δ(q 5,F) = {[T F, {),+, }]} q 8 = δ(q 5,E) = {[F (E ),{),+, }]} = δ(q 5,) = {[F, {),+, }]} {[E E +T, {),+}]} q 6 = δ(q 8,+) = {[E E + T, {),+}], q 9 {[T T F, {),+, }], {[T F, {),+, }], {[F (E ), {),+, }], q 10 {[F, {),+, }]} q 11 = δ(q 6,T) = {[E E +T, {),+}], {[T T F, {),+, }]} = δ(q 7,F) = {[T T F, {),+, }]} = δ(q 8, )) = {[F (E ), {),+, }]} 146 / 292

38 The Canonical LR(1)-Automaton E F 0 ( T F 3 ( F 5 ( T 2 6 E + 8 * T 11 ) 9 7 * F / 292

39 The Canonical LR(1)-Automaton E F 0 ( T + T T 6 4 F ( F F 3 E ) 11 5 ( 8 ( F ( T ) E ( *( 7 T 2 * * F 9 * 10 F / 292

40 The Canonical LR(1)-Automaton Discussion: In the example, the number of states was almost doubled... and it can become even worse The conflicts in states q 1,q 2,q 9 are now resolved! e.g. we have for: with: q 9 = {[E E+T, {ǫ,+}], {[T T F, {ǫ,+, }]} {ǫ,+} (First 1 ( F) 1 {ǫ,+, }) = {ǫ,+} { } = 148 / 292

41 The LR(1)-Parser: action goto Output The goto-table encodes the transitions: goto[q,x] = δ(q,x) Q The action-table describes for every state q and possible lookahead w the necessary action. 149 / 292

42 The LR(1)-Parser: The construction of the LR(1)-parser: States: Q {f} (f fresh) Start state: q 0 Final state: f Transitions: Shift: (p,a,pq) if q = goto[q,a], s = action[p,w] Reduce: (pq 1... q β,ǫ,pq) if [A β ] q β, q = goto(p,a), [A β ] = action[q β,w] Finish: (q 0 p,ǫ,f) if [S S ] p with LR(G,1) = (Q,T,δ,q 0,F). 150 / 292

43 The LR(1)-Parser: Possible actions are: shift // Shift-operation reduce(a γ) // Reduction with callback/output error // Error... for example: E E+T 0 T 1 T T F 0 F 1 F ( E ) 0 1 action ǫ ( ) + q 1 S,0 s q 2 E,1 E,1 s q 2 E,1 E,1 s q 3 T,1 T,1 T,1 q 3 T,1 T,1 T,1 q 4 F,1 F,1 F,1 q 4 F,1 F,1 F,1 q 9 E,0 E,0 s q 9 E,0 E,0 s q 10 T,0 T,0 T,0 q 10 T,0 T,0 T,0 q 11 F,0 F,0 F,0 q 11 F,0 F,0 F,0 151 / 292

44 The Canonical LR(1)-Automaton In general: We identify two conflicts: Reduce-Reduce-Conflict: [A γ, x], [A γ, x] q with A A γ γ Shift-Reduce-Conflict: [A γ, x], [A α aβ, y] q with a T und x {a}. for a state q Q. Such states are now called LR(1)-unsuited 152 / 292

45 The Canonical LR(1)-Automaton In general: We identify two conflicts: Reduce-Reduce-Conflict: [A γ, x], [A γ, x] q with A A γ γ Shift-Reduce-Conflict: [A γ, x], [A α aβ, y] q with a T und x {a} k First k (β) k {y}. for a state q Q. Such states are now called LR(k)-unsuited 152 / 292

46 Special LR(k)-Subclasses Theorem: A reduced contextfree grammar G is called LR(k) iff the canonical LR(k)-automaton LR(G,k) has no LR(k)-unsuited states. Discussion: Our example apparently is LR(1) In general, the canonical LR(k)-automaton has much more states then LR(G) = LR(G, 0) Therefore in practice, subclasses of LR(k)-grammars are often considered, which only use LR(G)... For resolving conflicts, the items are assigned special lookahead-sets: 1 independently on the state itself == Simple LR(k) 2 dependent on the state itself == LALR(k) 153 / 292

47 Syntactic Analysis Chapter 5: Summary 154 / 292

48 Parsing Methods deterministic languages = LR(1) =... = LR(k) LALR(k) SLR(k) LR(0) regular languages LL(1) LL(k) 155 / 292

49 Parsing Methods deterministic languages = LR(1) =... = LR(k) LALR(k) SLR(k) LR(0) regular languages Discussion: LL(1) All contextfree languages, that can be parsed with a deterministic pushdown automaton, can be characterized with an LR(1)-grammar. LR(0)-grammars describe all prefixfree deterministic contextfree languages The language-classes of LL(k)-grammars form a hierarchy within the deterministic contextfree languages. LL(k) 156 / 292

50 Lexical and Syntactical Analysis: Concept of specification and implementation: 0 0 [1-9][0-9]* Generator [1 9] [0 9] E E{op}E Generator 157 / 292

51 Lexical and Syntactical Analysis: From Regular Expressions to Finite Automata f t f * f f f a f f f f a b a b a b a 0 b a 1 a b a a 2 a b 3 4 From Finite Automata to Scanners a 02 b 023 w r i t e l n ( " H a l l o " ) ; 1 a 14 A B / 292

52 Lexical and Syntactical Analysis: Computation of lookahead sets: Fǫ(S ) Fǫ(E) Fǫ(E) Fǫ(E) a b c Fǫ(E) Fǫ(T) Fǫ(T) Fǫ(T) Fǫ(T) Fǫ(F) Fǫ(F) {(, name, } a S E T F (,, name 2 From Item-Pushdown Automata to LL(1)-Parsers: S AB i0 S i1 A1 β0 A a B b in An i B β1 β a b γ w First1( ) w First1(First1(γ) 1 First1(β) First1(β0)) w First1(γ) 1 Follow1(B) δ M Output 159 / 292

53 Lexical and Syntactical Analysis: From characteristic to canonical Automata: S E E E+T E T T T T F F E E T T F S E E T T F ( E ) ( F ) F ) F ) F ( ( ( E ) E E E F F + T E +T E T E E E+ E+T F T T F T T F T T F 0 T E F ( ( F F T ( ( 6 E T + 11 ) 8 * 9 7 * F 10 From Shift-Reduce-Parsers to LR(1)-Parsers: S i0 + T E + T 6 γ0 A1 i1 9 4 F + F 4 γ1 Amim * ( F ( F 3 E ) 11 * 5 γm B i ( 8 T ( F ( T E ) 5 8 F 2 ( ( α β T * 7 10 F 2 x w * 7 10 action goto Output 160 / 292