Review for the Midterm

Transcription

1 Review for the Midterm Stephen A. Edwrds Columi University Fll 2018

2 The Midterm Structure of Compiler Scnning Lnguges nd Regulr Expressions NFAs Trnslting REs into NFAs: Thompson s Construction Building DFA from n NFA: Suset Construction Prsing Resolving Amiguity Rightmost nd Reverse-Rightmost Derivtions Building the LR(0) Automton FIRST nd FOLLOW Building n SLR Prsing Tle Shift/Reduce Prsing

3 The Midterm 75 minutes Closed ook One doule-sided sheet of notes of your own devising Anything discussed in clss is fir gme Little, if ny, progrmming Detils of OCml/C/C++/Jv syntx not required

4 Compiling Simple Progrm int gcd(int, int ) { while (!= ) { if ( > ) -= ; else -= ; } return ; }

5 Wht the Compiler Sees int gcd(int, int ) { while (!= ) { if ( > ) -= ; else -= ; } return ; } i n t sp g c d ( i n t sp, sp i n t sp ) nl { nl sp sp w h i l e sp ( sp! = sp ) sp { nl sp sp sp sp i f sp ( sp > sp ) sp sp - = sp ; nl sp sp sp sp e l s e sp sp - = sp ; nl sp sp } nl sp sp r e t u r n sp ; nl } nl Text file is sequence of chrcters

6 Lexicl Anlysis Gives Tokens int gcd(int, int ) { while (!= ) { if ( > ) -= ; else -= ; } return ; } int gcd ( int, int ) { while (!= ) { if ( > ) -= ; else -= ; } return ; } A strem of tokens. Whitespce, comments removed.

7 Prsing Gives n Astrct Syntx Tree func int gcd rgs seq rg rg while return int int!= if int gcd(int, int ) { while (!= ) { if ( > ) -= ; else -= ; } return ; } > -= -=

8 Semntic Anlysis Resolves Symols nd Checks Types func int gcd rgs seq rg rg while return int int!= if Symol Tle > -= -= int int

9 Trnsltion into 3-Address Code L0: sne $1,, seq $0, $1, 0 true $0, L1 # while (!= ) sl $3,, seq $2, $3, 0 true $2, L4 # if ( < ) su,, # -= jmp L5 L4: su,, # -= L5: jmp L0 L1: ret int gcd(int, int ) { while (!= ) { if ( > ) -= ; else -= ; } return ; } elized ssemly lnguge w/ infinite registers

10 Genertion of Assemly gcd: pushl %ep # Sve BP movl %esp,%ep movl 8(%ep),%ex # Lod from stck movl 12(%ep),%edx # Lod from stck.l8: cmpl %edx,%ex je.l3 # while (!= ) jle.l5 # if ( < ) sul %edx,%ex # -= jmp.l8.l5: sul %ex,%edx # -= jmp.l8.l3: leve # Restore SP, BP ret

11 Descriing Tokens Alphet: A finite set of symols Exmples: { 0, 1 }, { A, B, C,..., Z }, ASCII, Unicode String: A finite sequence of symols from n lphet Exmples: (the empty string), Stephen, αβγ Lnguge: A set of strings over n lphet Exmples: (the empty lnguge), { 1, 11, 111, 1111 }, ll English words, strings tht strt with letter followed y ny sequence of letters nd digits

12 Opertions on Lnguges Let L = {, wo }, M = { mn, men } Conctention: Strings from one followed y the other LM = { mn, men, womn, women } Union: All strings from ech lnguge L M = {, wo, mn, men } Kleene Closure: Zero or more conctentions M = {} M MM MMM = {, mn, men, mnmn, mnmen, menmn, menmen, mnmnmn, mnmnmen, mnmenmn,... }

13 Regulr Expressions over n Alphet Σ A stndrd wy to express lnguges for tokens. 1. is regulr expression tht denotes {} 2. If Σ, is n RE tht denotes {} 3. If r nd s denote lnguges L(r ) nd L(s), (r ) (s) denotes L(r ) L(s) (r )(s) denotes {tu : t L(r ),u L(s)} (r ) denotes i=0 Li (L 0 = {} nd L i = LL i 1 )

14 Nondeterministic Finite Automt All strings contining n even numer of 0 s nd 1 s A 1 1 C B D 1 1. Set{ of sttes S : A B C D 2. Set of input symols Σ : {0,1} 3. Trnsition function σ : S Σ 2 S stte 0 1 A {B} {C} B {A} {D} C {D} {A} D {C} {B} 4. Strt stte s 0 : A 5. Set{ of ccepting } sttes F : A }

15 The Lnguge induced y n NFA An NFA ccepts n input string x iff there is pth from the strt stte to n ccepting stte tht spells out x. A 1 1 C B 1 1 D Show tht the string is ccepted A B D C D B A

16 Trnslting REs into NFAs Symol r 1 r r 1 r 2 2 Sequence r 1 r 1 r 2 r 2 Choice (r ) r Kleene Closure

17 Trnslting REs into NFAs Exmple: Trnslte ( ) into n NFA. Answer: Show tht the string is ccepted. Answer:

18 Simulting NFAs Prolem: you must follow the right rcs to show tht string is ccepted. How do you know which rc is right? Solution: follow them ll nd sort it out lter. Two-stck NFA simultion lgorithm: 1. Initil sttes: the -closure of the strt stte 2. For ech chrcter c, New sttes: follow ll trnsitions leled c Form the -closure of the current sttes 3. Accept if ny finl stte is ccepting

19 Simulting n NFA:, Strt

20 Simulting n NFA:

21 Simulting n NFA:

22 Simulting n NFA:

23 Simulting n NFA:, Done

24 Deterministic Finite Automt Restricted form of NFAs: No stte hs trnsition on For ech stte s nd symol, there is t most one edge leled leving s. Differs sutly from the definition used in COMS W3261 (Sipser, Introduction to the Theory of Computtion) Very esy to check cceptnce: simulte y mintining current stte. Accept if you end up on n ccepting stte. Reject if you end on non-ccepting stte or if there is no trnsition from the current stte for the next symol.

25 Deterministic Finite Automt { } type token = ELSE ELSEIF rule token = prse "else" { ELSE } "elseif" { ELSEIF } e l s e i f

26 Deterministic Finite Automt { type token = IF ID of string NUM of string } rule token = prse "if" { IF } [ - z ] [ - z 0-9 ]* s lit { ID(lit) } [ 0-9 ]+ s num { NUM(num) } i ID f eg z0 9 IF z0 9 hj z ID z NUM 0 9

27 Building DFA from n NFA Suset construction lgorithm Simulte the NFA for ll possile inputs nd trck the sttes tht pper. Ech unique stte during simultion ecomes stte in the DFA.

28 Suset construction for ( )

29 Suset construction for ( )

30 Suset construction for ( )

31 Suset construction for ( )

32 Suset construction for ( )

33 Result of suset construction for ( ) Is this miniml? (Suset construction does not involve minimiztion)

34 Amiguous Arithmetic Amiguity cn e prolem in expressions. Consider prsing 3-4 * with the grmmr e e + e e e e e e /e N * * * * *

35 Opertor Precedence Defines how sticky n opertor is. 1 * * 4 * t higher precedence thn +: (1 * 2) + (3 * 4) * * 3 4 * + t higher precedence thn * : 1 * (2 + 3) * 4 *

36 Associtivity Whether to evlute left-to-right or right-to-left Most opertors re left-ssocitive ((1 2) 3) 4 1 (2 (3 4)) left ssocitive right ssocitive

37 Fixing Amiguous Grmmrs A grmmr specifiction: expr : expr PLUS expr expr MINUS expr expr TIMES expr expr DIVIDE expr NUMBER Amiguous: no precedence or ssocitivity. Ocmlycc s complint: 16 shift/reduce conflicts.

38 Assigning Precedence Levels Split into multiple rules, one per level expr : expr PLUS expr expr MINUS expr term term : term TIMES term term DIVIDE term tom tom : NUMBER Still miguous: ssocitivity not defined Ocmlycc s complint: 8 shift/reduce conflicts.

39 Assigning Associtivity Mke one side the next level of precedence expr : expr PLUS term expr MINUS term term term : term TIMES tom term DIVIDE tom tom tom : NUMBER This is left-ssocitive. No shift/reduce conflicts.

40 Rightmost Derivtion of + 1 :e t + e 2 :e t 3 : t t 4 : t e t + e t + t t + t + + At ech step, expnd the rightmost nonterminl. nonterminl hndle : The right side of production Fun nd interesting fct: there is exctly one rightmost expnsion if the grmmr is unmigious.

41 Rightmost Derivtion: Wht to Expnd 1 :e t + e 2 :e t 3 : t t 4 : t e t + e t + t t + t + + e t + e t + t t + t + + Expnd here Terminls only

42 Reverse Rightmost Derivtion 1 :e t + e 2 :e t 3 : t t 4 : t e t + e t + t t + t t + t + t + t t + e e vile prefixes terminls t t t + e e

43 Shift/Reduce Prsing Using n Orcle 1 :e t + e 2 :e t 3 : t t 4 : t e t + e t + t t + t + + stck + shift + shift + shift + reduce 4 t + reduce 3 t + shift t + shift t + reduce 4 t + t reduce 2 t + e e reduce 1 ccept input

44 Hndle Hunting Right Sententil Form: ny step in rightmost derivtion Hndle: in sententil form, RHS of rule tht, when rewritten, yields the previous step in rightmost derivtion. The ig question in shift/reduce prsing: When is there hndle on the top of the stck? Enumerte ll the right-sententil forms nd pttern-mtch ginst them? Usully infinite in numer, ut let s try nywy.

45 The Hndle-entifying Automton Mgicl result, due to Knuth: An utomton suffices to locte hndle in right-sententil form. e e t t + t + + t + e t + t + + t+ t + t + + t + t t + t + + t t t t e t + e + t t

46 Building the Initil Stte of the LR(0) Automton 1 :e t + e 2 :e t 3 : t t 4 : t e e Key ide: utomt identify vile prefixes of right sententil forms. Ech stte is n equivlence clss of possile plces in productions. At the eginning, ny vile prefix must e t the eginning of string expnded from e. We write this condition e e

47 Building the Initil Stte of the LR(0) Automton 1 :e t + e 2 :e t 3 : t t 4 : t e e e t + e e t Key ide: utomt identify vile prefixes of right sententil forms. Ech stte is n equivlence clss of possile plces in productions. At the eginning, ny vile prefix must e t the eginning of string expnded from e. We write this condition e e There re two choices for wht n e my expnd to: t + e nd t. So when e e, e t + e nd e t re lso true, i.e., it must strt with string expnded from t.

48 Building the Initil Stte of the LR(0) Automton 1 :e t + e 2 :e t 3 : t t 4 : t e e e t + e e t t t t Key ide: utomt identify vile prefixes of right sententil forms. Ech stte is n equivlence clss of possile plces in productions. At the eginning, ny vile prefix must e t the eginning of string expnded from e. We write this condition e e There re two choices for wht n e my expnd to: t + e nd t. So when e e, e t + e nd e t re lso true, i.e., it must strt with string expnded from t. Similrly, t must e either t or, so t t nd t.

49 Building the LR(0) Automton e e e t + e S0 : e t t t t The first stte suggests vile prefix cn strt s ny string derived from e, ny string derived from t, or.

50 Building the LR(0) Automton Just pssed string derived from e S7 : e e e e e e t + e S0 : e t t t t t t S1 : t Just pssed prefix tht ended in n Just pssed prefix ending in string derived from t t S2 : e t +e e t The first stte suggests vile prefix cn strt s ny string derived from e, ny string derived from t, or. The items for these three sttes come from dvncing the cross ech thing, then performing the closure opertion (vcuous here).

51 Building the LR(0) Automton S7 : e e e e e e t + e S0 : e t t t t t S2 : e t +e e t + S4 : e t + e t t S1 : t In S2, + my e next. This gives t + e. t t S3 : In S1, my e next, giving t

52 Building the LR(0) Automton S7 : e e e e e e t + e S0 : e t t t t t S2 : e t +e e t + e t + e e t + e S4 : e t t t t t t S1 : t t t S3 : t t t In S2, + my e next. This gives t + e. Closure dds 4 more items. In S1, my e next, giving t nd two others.

53 Building the LR(0) Automton S7 : e e e e e e t + e S0 : e t t t t t S2 : e t +e e t + t e t + e e t + e S4 : e t t t t e t t S1 : t S6 : e t + e t t S3 : t t t t S5 : t t

54 The first function If you cn derive string tht strts with terminl t from sequence of terminls nd nonterminls α, then t first(α). 1. If X is terminl, first(x ) = {X }. 2. If X, then dd to first(x ). 3. If X Y 1 Y k nd first(y 1 ), first(y 2 ),..., nd first(y i 1 ) for i = 1,...,k for some k, dd first(y i ) {} to first(x ) X strts with nything tht ppers fter skipping empty strings. Usully just first(y 1 ) first(x ) 4. If X Y 1 Y K nd first(y 1 ), first(y 2 ),..., nd first(y k ), dd to first(x ) If ll of X cn e empty, X cn e empty 1 :e t + e 2 :e t 3 : t t 4 : t first() = {} first(t) = {} ecuse t t nd t first(e) = {} ecuse e t + e, e t, nd first(t) = {}.

55 First nd first(α) mens α cn derive the empty string. 1. If X is terminl, first(x ) = {X }. 2. If X, then dd to first(x ). 3. If X Y 1 Y k nd first(y 1 ), first(y 2 ),..., nd first(y i 1 ) for i = 1,...,k for some k, dd first(y i ) {} to first(x ) 4. If X Y 1 Y K nd first(y 1 ), first(y 2 ),..., nd first(y k ), dd to first(x ) X Y Z Y Y Z c Z W W W d first() = {} first(c) = {c} first(d) = {d} (1) first(w ) = {} first(d) = {,d} (2,3) first(z ) = first(c) (first(w ) {}) {} = {,c,d} (3,3,4) first(y ) = {} {} = {,} (2,3) first(x ) = (first(y ) {}) (first(z ) {}) first() = {,,c,d} (3,3,3)

56 The follow function If t is terminl, A is nonterminl, nd At cn e derived, then t follow(a). 1. Add $ ( end-of-input ) to follow(s) (strt symol). End-of-input comes fter the strt symol 2. For ech prod. Aα, dd first(α) {} to follow(a). A is followed y the first thing fter it 3. For ech prod. A B or A Bα where first(α), then dd everything in follow(a) to follow(b). If B ppers t the end of production, it cn e followed y whtever follows tht production 1 :e t + e 2 :e t 3 : t t 4 : t first(t) = {} first(e) = {} follow(e) = {$} follow(t) = { } 1. Becuse e is the strt symol

57 The follow function If t is terminl, A is nonterminl, nd At cn e derived, then t follow(a). 1. Add $ ( end-of-input ) to follow(s) (strt symol). End-of-input comes fter the strt symol 2. For ech prod. Aα, dd first(α) {} to follow(a). A is followed y the first thing fter it 3. For ech prod. A B or A Bα where first(α), then dd everything in follow(a) to follow(b). If B ppers t the end of production, it cn e followed y whtever follows tht production 1 :e t + e 2 :e t 3 : t t 4 : t first(t) = {} first(e) = {} follow(e) = {$} follow(t) = { + } 2. Becuse e t + e nd first(+) = {+}

58 The follow function If t is terminl, A is nonterminl, nd At cn e derived, then t follow(a). 1. Add $ ( end-of-input ) to follow(s) (strt symol). End-of-input comes fter the strt symol 2. For ech prod. Aα, dd first(α) {} to follow(a). A is followed y the first thing fter it 3. For ech prod. A B or A Bα where first(α), then dd everything in follow(a) to follow(b). If B ppers t the end of production, it cn e followed y whtever follows tht production 1 :e t + e 2 :e t 3 : t t 4 : t first(t) = {} first(e) = {} follow(e) = {$} follow(t) = { +,$} 3. Becuse e t nd $ follow(e)

59 The follow function If t is terminl, A is nonterminl, nd At cn e derived, then t follow(a). 1. Add $ ( end-of-input ) to follow(s) (strt symol). End-of-input comes fter the strt symol 2. For ech prod. Aα, dd first(α) {} to follow(a). A is followed y the first thing fter it 3. For ech prod. A B or A Bα where first(α), then dd everything in follow(a) to follow(b). If B ppers t the end of production, it cn e followed y whtever follows tht production 1 :e t + e 2 :e t 3 : t t 4 : t first(t) = {} first(e) = {} follow(e) = {$} follow(t) = { +,$} Fixed-point reched: pplying ny rule does not chnge ny set

60 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s1 7 2 From S0, shift n nd go to S1; or cross t nd go to S2; or cross n e nd go to S7.

61 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 From S1, shift nd go to S3; or, if the next input could follow t, reduce y rule 4. According to rule 1, + could follow t; from rule 2, $ could.

62 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 From S2, shift + nd go to S4; or, if the next input could follow n e (only the end-of-input $), reduce y rule 2.

63 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 From S3, shift n nd go to S1; or cross t nd go to S5.

64 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s1 6 2 From S4, shift n nd go to S1; or cross n e or t.

65 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 From S5, reduce using rule 3 if the next symol could follow t (gin, + nd $).

66 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r1 From S6, reduce using rule 1 if the next symol could follow n e ($ only).

67 Converting the LR(0) Automton to n SLR Tle S7: e e S0 S1: t S3 e t t S5: t t S2: e t t 1 :e t + e 2 :e t 3 : t t 4 : t S4 e + S6: e t + e Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r1 7 If, in S7, we just crossed n e, ccept if we re t the end of the input.

68 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r $ Shift, goto 1 Look t the stte on top of the stck nd the next input token. Find the ction (shift, reduce, or error) in the tle. In this cse, shift the token onto the stck nd mrk it with stte 1.

69 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r $ Shift, goto $ Shift, goto 3 Here, the stte is 1, the next symol is, so shift nd mrk it with stte 3.

70 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t 0 +$ Shift, goto $ Shift, goto $ Shift, goto 1 Stte Action Goto $ Reduce 4 + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r1 7 Here, the stte is 1, the next symol is +. The tle sys reduce using rule 4.

71 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t Stte Action Goto + $ e t 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r $ Shift, goto $ Shift, goto $ Shift, goto $ Reduce $ Remove the RHS of the rule (here, just ), oserve the stte on the top of the stck, nd consult the goto portion of the tle.

72 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t 0 +$ Shift, goto $ Shift, goto $ Shift, goto 1 Stte Action Goto + $ e t $ Reduce 4 3 t 5 +$ Reduce 3 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r1 7 Here, we push t with stte 5. This effectively cks up the LR(0) utomton nd runs it over the newly dded nonterminl. In stte 5 with n upcoming +, the ction is reduce 3.

73 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t 0 +$ Shift, goto $ Shift, goto $ Shift, goto 1 Stte Action Goto + $ e t $ Reduce 4 3 t 5 +$ Reduce 3 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r1 7 0 t 2 +$ Shift, goto 4 This time, we strip off the RHS for rule 3, t, exposing stte 0, so we push t with stte 2.

74 Shift/Reduce Prsing with n SLR Tle Stck Input Action 1 :e t + e 2 :e t 3 : t t 4 : t 0 +$ Shift, goto $ Shift, goto $ Shift, goto 1 Stte Action Goto + $ e t $ Reduce 4 3 t 5 +$ Reduce 3 0 s r4 s3 r4 2 s4 r2 3 s1 5 4 s r3 r3 6 r t +$ Shift, goto t + 4 $ Shift, goto t $ Reduce t t $ Reduce t e $ Reduce 1 0 e 7 $ Accept

75 Shift/Reduce nd Reduce/Reduce Conflicts Shift/reduce conflicts re cused y stte like t Else s t If the next token is Else, do you reduce it since Else my follow t, or shift it? Reduce/reduce conflicts re cused y stte like t t e t Do you reduce y t t or y e t?

76 A Reduce/Reduce Conflict S1 : 1 : 2 : 3 : S0 : S3 : S2 : S4 :