arxiv: v2 [cs.lo] 24 Feb 2017

Transcription

1 Lol Lexing teven Ou University of dinurgh Phil ott University of dinurgh Jques Fleuriot University of dinurgh rxiv: v2 [s.lo] 24 Fe 2017 strt We introdue novel prsing onept lled lol lexing. It integrtes the lssilly seprted stges of lexing nd prsing y llowing lexing to e dependent upon the prsing progress nd y proving simple mehnism for onstrining lexil miguity. This mes it possile for lnguge design to e omposle not only t the level of ontext-free grmmrs, ut lso t the lexil level. It lso mes it possile to inlude lightweight error-hndling diretly s prt of the lnguge speifition insted of leving it up to the implementtion. We present high-level lgorithm for lol lexing, whih is n extension of rley s lgorithm. We hve formlly verified the orretness of our lgorithm with respet to its lol lexing semntis in Iselle/HOL. 1. Introdution The trditionl pproh to speifying the syntx of omputer lnguge is to define two omponents, lexer (lso lled snner nd prser. The lexer prtitions the input doument onsisting of sequene of hrters into sequene of toens. h toen is uniquely ssoited with terminl, suh tht the sequene of toens n e viewed s sequene of terminls. The prser is typilly defined y ontext-free grmmr (CFG, nd hes if the sequene of terminls is in the lnguge generted y the CFG. CFGs re powerful lnguge design tool. most importnt property of CFGs is omposility. Given two or more CFGs, it is esy to omine them into single CFG in vrious wys in order to speify lnguge s mshup of severl other lnguges, whih is ommon senrio in modern progrmming environments. The lexer omponent of this trditionl setup is prolemti though in suh mshup senrio. eyword in one lnguge might e n entifier in nother lnguge, rendering the lexers of these two lnguges inomptile with eh other. The prolem is tht lexers re not supposed to e omposed in the trditionl setup. Prtil solutions to this prolem usully involve some form of d-ho ommunition etween prser nd lexer stges, thus shifting n issue This wor hs een funded y PRC grnt P/L011794/1. whih should e delt with t the lnguge design level to the implementtion level. The min resons for the trditionl split of syntx reognition into lexing nd prsing re speed, expressivity nd onveniene: peed Typilly terminls re speified vi regulr expressions tht n e reognized with deterministi finite stte mhines. This is usully muh fster thn prsing with respet to CFG. xpressivity Despite CFGs eing more expressive thn regulr expressions, two ommon lexing rules nnot e expressed with CFGs: longest-mth, nd priority. The longest-mth rule sttes tht if multiple terminls re ssoited with regulr expressions tht ould ll mth prefixes of the rest of the sequene of hrters to e snned, then the terminls whih mth the longest prefix re to e preferred. The priority rule omes into ply if fter pplition of the longest-mth rule there re still t lest two different terminls s possile ndtes: then liner priority order mong terminls is ssumed, nd the terminl with the highest priority mong ll ndtes is pied. Conveniene In this setup, whitespe nd omments re usully speil terminls whih do not pper in ny grmmr rule, ut whih re filtered out of the sequene of terminls during the lexing stge. nnerless prsing [3] proposes to solve our lexing prolem y relinquishing seprte lexing, effetively entifying hrters, toens nd terminls. This negtively ffets ll three mentioned dvntges of seprte lexing stge, the most severely ffeted eing expressivity: in order to pproximte the missing longest-mth nd priority rules, snnerless prsing introdues follow restritions nd rejet produtions, whih re wrd dditions to the elegnt formlism of ontext-free grmmrs euse they destroy the nie omposility properties of ontext-free grmmrs. snnerless prsing tehnique lled prt prsing goes even further nd does wy with ontext-free grmmrs entirely, employing prsing expression grmmrs insted [2], gin to the detriment of omposility. Insted, we propose new prsing semntis whih we ll lol lexing. Lol lexing eeps the distintion etween

2 lexing nd prsing, nd etween hrters, toens nd terminls. But insted of deterministilly onverting sequene of hrters into sequene of toens, lol lexing onverts sequene of hrters into set of toen sequenes, pplying lexing nd prsing in n intertwined mnner. The ontriutions of this pper re s follows: We first define the novel onept of lol lexing in etion 2. We then present exmples of pplitions of lol lexing in etion 3. These exmples show tht lol lexing is generlistion of the trditionl setup, nd tht lol lexing redily llows integrted ess t the lnguge design level to issues suh s lexil omposility nd error-hndling. In etion 4 we desrie high-level lgorithm whih implements lol lexing s n extension of rley s lgorithm. We hve formlly verified the orretness of this lgorithm in Iselle/HOL [6], nd prove n outline of this orretness proof in etion 5. The full proof ( totl of lines or out 230 pges is ville in [12]. We lso prove prtil lirry for lol lexing [13], written in l/l.js [7, 8]. The lirry ontins exmples of its pplition to the exmples in etion 3. Before onluding, we disuss further relted wor in etion Definition of Lol Lexing Before defining lol lexing, we remind the reder of few si notions. For set U we let U denote the set of sequenes with elements in U, ε U denotes the empty sequene, nd for two sequenes α U nd β U we let αβ U denote their ontention. Given sequene α U, we denote its length y α nd let α i U denote the i-th element of α for i {0,..., α 1}. ontextfree grmmr is qudruple (N, T, R,, where N is the set of nonterminls, T the set of terminls (suh tht N nd T re disjoint, R N (N T the rules of the grmmr nd N the strt. Insted of (N, α R we often write N α. For α, β (N T we sy α β iff there re α 0, N, α 1 nd γ suh tht α = α 0 Nα 1, β = α 0 γα 1 nd N γ. Furthermore, we write for the reflexive nd trnsitive losure of. We define the lnguge L of grmmr G s the set of ll sequenes of terminls derivle from the strt, L = {w T w}. Furthermore we define the set of prefixes L prefix of G vi { } L prefix = w T α (N T. wα. Definition 2.1 (Toen. Given set of terminls T, nd set of hrters Σ, toen x is pir x = (t, T Σ. In exmples we will often use the nottion t for the toen x. We ll the toen empty iff = 0. We define [x] = t nd x =. We lift these nottions in the ovious mnner to sequenes of toens: given toen sequene q = x 0... x r (T Σ we define [q] = [x 0 ]... [x r ] T nd q = x 0... x r Σ. Definition 2.2 (Lol Lexing. Given set of terminls T, nd set of hrters Σ, we ll pir (Lex, el lol lexing (with respet to T nd Σ iff: The lexer Lex ssigns to eh terminl t T lexing funtion Lex(t whih, given sequene of hrters D Σ nd position {0,..., D }, returns set onsisting of toens (t, suh tht D nd i = D i for ll i suh tht 0 i 1. The seletor el tes two toen sets nd B suh tht B nd returns toen set el(, B suh tht el(, B B. We usully define el indiretly y defining strit prtil order < el on toens nd setting el(, B = {x B y B. x < el y}. emntis of Lol Lexing Given lol lexing ll, wht is its semntis, i.e. how does it onvert sequene D of hrters into set ll(d of toen sequenes? Note tht while for defining prtiulr lol lexing ll we do not need ontext-free grmmr, just its sets of terminls nd hrters, the semntis of ll is dependent upon grmmr. In the following we will often ll toen sequenes pths. The onversion proess wors s follows: We go through D from to right, produing sets of pths P u (T Σ long the wy. The index {0,..., D } denotes the position in D we re looing t, nd we use the index u {0, 1,...} to tr itertively generted P 0, P1,... t this position. Let s ssume now tht we hve rehed position in D, hving so fr produed the set of pths P 0; in se we re t the very eginning of D suh tht = 0 we ssume P0 0 = {ε}. We will hve een reful to only produe p P 0 suh tht [p] L prefix. Now, whih toen should we produe next? For sure, the toen to produe next must e memer of the set X = {x T Σ x Lex([x](D, } euse ording to Lex these re the only toens whih strt t position in D. Furthermore, we re only interested in the suset W 0 of those memers of X whih n ontinue sequene in P 0 to sequene of terminls in L prefix: W 0 = { x X p P 0. p = [px] L prefix }. The seletor dees whether nd how to onstrin ny remining miguity in the toen seletion: The set Z 1 = el(, W 0 ontins those toens we use to rete the next set of pths P 1 İf Z 1 does not ontin ny empty toens, then P1 is just P 1 = ppend Z 1 P 0,

3 where ppend T P = P {pt p P p = t T [pt] L prefix }, nd euse we only dded pths q with q > we n simply proeed to position 1 vi P1 0 = P1. But if Z 1 does indeed ontin empty toens, then there might e newly dded pths q with q =, nd we might e le to extend these pths even further. Therefore we define P 1 more generlly s where P 1 = limit (ppend Z 1 P 0, limit f X = f n (X. n=0 The ft tht now P 1 \ P0 my ontin pths q with q = mens tht potentilly more toens in X eome eligile to extends pths whih stop t position. We therefore eep repeting the ove proedure (potentilly infinitely often until we re sure to hve produed ll eligile toens t position y forming monotone hins W 0 W 1 W 2... Z 0 Z 1 Z 2... P 0 P 1 P 2... where for u {0, 1, 2,...} we reursively define W u = {x X p P u. p = [px] L prefix} Z u1 = el(z u, Wu1 P u1 = limit (ppend Z u1 P u. We finlize the genertion of toen sequenes t position y defining P = u=0 nd move on to position 1 vi P1 0 = P. To omplete the given set of reursive equtions, we define Z 0 = for ll positions. One we hve rrived t the end of the hrter sequene D, we finish the onversion of D into the set ll(d of toen sequenes vi defining P = P D nd P u ll(d = {p P p = D [p] L}. Note tht the lol lexing semntis does not depend on the prtiulr form of the grmmr, ut only on L nd L prefix. If the grmmr ontins no unprodutive nonterminls, i.e. if for ll X N there is w Σ suh tht X w, then L prefix is uniquely determined y L nd thus in this se the lol lexing semntis only depends on L lone. Let us lso omment on why the seletor tes two rguments. It is tempting (nd indeed this ws the first thing we tried to let the seletor t only on single rgument, s in Z u1 = el W u1, ut this destroys the property tht the Z u form monotone hin, whih turned out to e importnt for the orretness of our lgorithm for lol lexing (see etion pplitions of Lol Lexing In this setion we explore lol lexing nd its pplitions through rnge of exmples. xmple 3.1 (Trditionl Lexil peifitions. Turning trditionl lexil speifition into the definition of lol lexing is strightforwrd. Let the trditionl speifition e defined over n input lphet Σ nd given y n pirs (r 1, t 1... (r n, t n. The r i re regulr expressions over Σ, none of whih mth ε Σ, nd the set of terminls T onsists of n different terminls t 1,..., t n. For eh t i T we then define Lex(t i (D, = {(t i, }, where is the longest prefix of D... D D 1 suh tht r i mthes. If r i mthes no prefix of D... D D 1 then we define Lex(t i (D, =. We define strit prtil order < el on the set of toens whih enodes the longest-mth nd priority rules: (t i, < el (t j, d iff < d ( = d i < j. This implies tht el(, X will e either empty or singleton euse < el is totl on ll possile X. To endow the lol lexing ll = (Lex, el with the equivlent semntis to trditionl lexing, we define the ontextfree grmmr G = ({, T}, T, R, where R = T, ε, T t 1,. T t n. Beuse of L prefix = L = T this grmmr poses no dditionl onstrints on the lol lexing proess, nd therefore ll(d = iff trditionl lexing of D would led to n error, nd ll(d = {p} iff trditionl lexing of D would yield the toen sequene p. xmple 3.2 (Infinite et of Toen equenes. We hnge the previous exmple slightly nd llow the r i to lso mth the empty hrter sequene ε. s onrete exmple onser Σ = {}, n = 1 nd let r 1 e regulr expression whih mthes ny (possily empty sequene of s. Then

4 using the sme grmmr G s in the previous exmple, we otin for exmple { ll( =, ε, ε ε, } ε ε ε,..., t 1 t 1 t 1 t 1 whih is n infinite set. t 1 t 1 t 1 t 1 t 1 t 1 Beuse lol lexing intertwines lexing nd prsing, determining for hrter sequene D Σ ll possile toen sequenes ll(d involves finding vl prses of [p] for ll p ll(d. The following exmples demonstrte this interply etween lexing nd prsing nd the role of the seletor el in it. xmple 3.3. Conser the grmmr H = (N, T, R, where N = {,, }, T = {, minus,, }, R = {, minus,,,,, }, =. The input hrters re Σ = {, -,,, }. s previously, we speify Lex simply y ssoiting the terminls with regulr expressions: The terminl reognizes the hrter, minus reognizes the hrter -, reognizes ny nonempty sequene of letters nd reognizes ny nonempty sequene of letters nd hyphens -. Let D denote the hrter sequene -. We hoose the simplest option for < el nd define < el =, i.e. no toen hs higher priority thn nother toen. This implies el(, B = B for ll toen sets B. The result of determining ll(d is shown in Figure 1. Beuse overlps with oth entifier nd minus, D n e interpreted in eight different wys s toen sequene. Beuse of the longest-mth rule for regulr expressions, - minus / ll(d, exluded together with severl other toen sequenes whih would otherwise qulify. For ll p ll(d the terminl sequene [p] yields vl prse tree, i.e. there re 8 different wys to prse D lthough H y itself is n unmiguous grmmr; ll of them re jointly depited in the prse grph in Figure 2. miguities re depited s dshed lines in the grph: whenever there re multiple wys to proeed with the derivtion of nonterminl, eh lterntive is onneted with the nonterminl y dshed line. xmple 3.4. We use the sme grmmer H nd the sme lexer Lex s in xmple 3.3, ut hoose seletor el suh tht hs lower priority thn nd minus: x < el y iff [x] = [y] {, minus}. This results in unique lexing of -, { } - ll(- =, minus ll(- = - minus, - minus, - minus, - minus, -, -, -, - Figure 1. Lexing - in xmple minus - - Figure 2. Prsing - in xmple 3.3

5 - minus Figure 3. Prsing - in xmple 3.4 nd thus lso in unique prsing s shown in Figure 3. xmple 3.5. We gin leve H nd Lex fixed. This time we define < el suh tht longer toens hve higher priority: x < el y iff x < y. This yields two possile lexings for -, -, ll(- = -, nd leds to the miguous prsing shown on the hnd se of Figure 4. xmple 3.6. We hoose to modify the previous exmple suh tht its remining miguity is resolved in fvour of y defining x < el y iff x < y ( x = y [x] = [y] =. This leds to unique lexing for -, { } - ll(- =, nd yields the unique prsing depited on the right hnd se of Figure 4. xmple 3.7 (The Lexer H. Conser the expression (* given in the progrmming lnguge C. The mening of this expression depends on whether is type entifier or vrile entifier. If it is type entifier, then the expression is to e interpreted s type st, otherwise s - - Figure 4. Prsing - in x. 3.5 ( nd x. 3.6 (right multiplition. Beuse type entifiers nnot e distinguished from vrile entifiers y their loo, in trditionl prsing prolem rises, euse the lexer phse is supposed to hppen efore the prsing phse nd ny semnti nlysis, ut the terminl type of relly depends on informtion from the semnti nlysis. The trditionl solution, to mnully diret feed from the semnti nlysis into the lexer, is nown s the lexer h [4]. With lol lexing, the lexer / prser n offer oth lterntives, nd let lter stges of the nlysis pi the right one, thus deoupling prsing nd semnti nlysis. The grmmr C demonstrtes this, where T = {type,, steris,, right} nd R = xpr, xpr Cst, xpr Deref, xpr, xpr xpr right, xpr steris xpr, Cst Type right xpr, Deref steris xpr, Type type. The lexer Lex is hosen in the ovious wy, together with n empty < el. The resulting miguous prse grph for (* is shown in Figure 5, nd ll((* = ( ( right type * steris right, * steris. hrödinger s Toen xmple 3.7 is speil se of senrio where the onversion from hrter sequenes to toen sequenes is miguous, ut where the toen oundries re lwys the sme in ll lterntive toen sequenes. The toens ppering in suh senrio hve een hristened hrödinger s toens [5]. Lol lexing is more power-

6 xpr {, mul,, num,, right}, nd the rules y ( xpr xpr Cst Type type * steris right xpr Figure 5. Prsing (* in C xpr Deref xpr um, um um, um, mul tom, tom, tom um right, tom, tom num. The input hrters re Σ = {, *, (,, ,... z}, nd Lex is speified suh tht reognizes the hrter, mul reognizes the hrter *, reognizes the opening ret (, right reognizes the losing ret, reognizes ny nonempty sequene of letters nd digits strting with letter, nd num reognizes ny nonempty sequene of digits. Conser now the following string w whih is invl with respet to the lnguge we just speified: 2(*(1 ful thn the hrödinger s toen pproh nd (t lest oneptully susumes it. Ruy lippers The Mrp prser [10] is n rley-sed prsing lirry whih dvotes the use of tehnique lled Ruy lippers [11]. This tehnique tes dvntge of the ft tht the rley prser nows whih toens it expets t ny given stge of the prse progress. Mrp hs n interfe through whih the prser n ommunite with the snner to negotite whih toen to sn next, thus llowing for sophistited error hndling. Ruy lippers nd lol lexing re oth hildren of the sme insight, nmely tht the snning nd prsing stges should ommunite euse the prser hs informtion out whih toens it expets next. Lol lexing though tes this insight to new level whih in priniple is independent from prtiulr prsing lgorithm lie rley. In this sense, lol lexing n e seen s proving rigorous semntis for ertin uses of the Ruy lippers tehnique. The next exmple demonstrtes how lol lexing n e used to speify lightweight error reovery s prt of the lnguge design. xmple 3.8 (rror Reovery. Conser grmmr for simple rithmeti expressions, where the nonterminls re given y {xpr, um,, tom}, the terminls y Insted of simply dignosing tht there is some prse error t position = 1, we would lie to reover some of the struture of w, for exmple for proving etter error messges or for rih intertive editing experiene. To simplify this ts, we first me our grmmr more permissive y dding the rule tom. This hs the effet tht string lie 2x eomes legl, representing the multiplition of 2 nd x. uh nottion is ommon mthemtil prtie nd thus seems lie justifile design hoie. We then introdue three new terminls: e-tom nd e-right oth reognize the empty string ε only, nd e-superfluous reognizes the losing ret. We inorporte these new terminls into the grmmr y dding the following rules: e-superfluous, tom um e-right, tom e-tom Finlly, we hoose the seletor el suh tht the three dded error terminls hve lower priority thn ll other terminls. Pths in ll(d tht ontin error terminls will only e onsered y us if ll(d ontins no pths without error terminls. Figure 6 shows the result of pplying the updted grmmr to w. The orresponding pth is 2 num ( * mul ε e-tom e-tom right ( e-superfluous ε 1 num ε e-right. Testing onvines us tht the updted grmmr n indeed suessfully prse ll D Σ, nd does so unmiguously. Whitespe In our exmples we hve voed to me use of whitespe. In trditionl prsing there re essentilly two different wys of deling with whitespe:

7 tom 2 num tom ( um * mul tom um e-superfluous um tom " e-tom xpr um right tom ( " e-tom tom um tom 1 num Figure 6. Prsing 2(*(1 in xmple 3.8 " e-right 1. One pproh is to expliitly inorporte whitespe terminls into the grmmr rules. This n eome umersome nd error-prone if done mnully, euse usully whitespe n leglly pper lmost everywhere. 2. The other pproh is to hndle whitespe t the lexil stge exlusively. The first pproh pplies to lol lexing s well. The seond pproh does not diretly pply, s there is no single lexil stge nymore with lol lexing. Nevertheless, it seems tht lol lexing llows to omine the onveniene of the seond pproh with the fine-grined ontrol of the first one y using extended ttriute grmmrs to speify lyout onstrints, ming it possile to tle lyout-sensitive lnguges. The omintion of lol lexing with lyoutsensitivity is wor in progress nd eyond the sope of this pper. 4. Implementing Lol Lexing Given grmmr G nd lol lexing ll, let us define the hrter lnguge L Σ of G nd ll y L Σ = {D Σ ll(d }. How do we uild reognizer for L Σ? Lexing Driving Prsing Our first ttempt might e to diretly pply the semntis of lol lexing fter hving pied prsing lgorithm whih is ple of reognizing oth the prefixes L prefix nd the lnguge L of G. Most of the populr prsing lgorithms would e suitle for this, suh s LL, LR or rley prsing. While this pproh will wor in mny ses, it is ineffiient to tret the prsing lgorithm s l ox whih is repetedly sed whether given sequene of terminls is in L prefix or not. More importntly, s xmple 3.2 shows there re finite grmmrs whih nevertheless produe infinite sets of toen sequenes nd for whih this pproh would therefore fil y getting stu in non-terminting pth generting loop. Prsing Driving Lexing etter pproh seems to e to inverse ove pproh nd to let the prsing progress drive the lexing proess. Often it will e possile to predit from the internl prser stte whih terminl is expeted next. For this to wor, we need to modify the prsing lgorithm so tht it not only nows out terminls T, ut lso out hrters Σ. The rley lgorithm seems to e est suited to e dpted to suh purpose, s it wors on ll ontext-free grmmrs, opes grefully with miguity, nd is esily extended with top-down, -right, nd ottom-up prmetriity. This is why rley-sed prsing is our min fous here. Nevertheless, studying how to modify other prsing lgorithms for lol lexing is interesting nd of potentilly gret prtil interest s well; experiments indite tht in prtiulr the LR(1 prsing lgorithm n e modified to filitte lol lexing in nturl nd simple wy. rley s lgorithm We first desrie ( high-level version of rley s originl lgorithm, ssuming Σ = T. To reognize n input D Σ s elonging to L, it omputes items; n item is qudruple (r, d, i, j where r = (N αβ R is rule of the grmmr, d = α is position within tht rule demring the urrent prsing progress, i {0,..., D } is the origin of the item, nd j {i,..., D } is the in of the item. n lterntive wy to write the item is s (N α β, i, j. rley s lgorithm uilds monotone hin of item sets I 0 I 1 I 2... I D = I.

8 Init = {( α, 0, 0 α R} Predit I = I {(M γ,, N α β i. Complete I = I (N α Mβ, i, I (M γ R} {(N αm β, i, j γ. (N α Mβ, i, j I (M γ, j, I} n I = I {(N αx β, i, 1 < D X = D (N α Xβ, i, I} Figure 7. Building Blos of rley s lgorithm To this end, we define n initil item set Init nd monotone opertors Predit, Complete nd n whih ll te position {0,..., D } nd n item set I nd return n ugmented item set (Figure 7. We then define for {0,..., D } the opertor π y π I = limit (n Complete Predit I, reusing the limit opertor introdued in etion 2, nd use π to reursively define the sets I y I 0 = π 0 Init, I = π I 1 for > 0. Theorem 4.1 (Corretness of rley s lgorithm. rley s lgorithm is oth sound nd omplete, i.e. D iff α. ( α, 0, D I. Proof. This is overed y Theorem 4.2 for the speil se Σ = T, { {(t, t} Lex(t(D, = < el =. for < D D = t, otherwise rley s lgorithm with Lol Lexing We now ssume tht we hve lol lexing ll nd thus Σ nd T do not neessrily oine nymore. We leve Init, Predit nd Complete unhnged, ut we need n updted n opertor tht wors on toens insted of hrters, nd new opertor Toens (Figure 8. The opertion Toens T I first determines ll the ndte terminls tht ould possily pper next t position Toens T I = el T {x X N α β i. X T (N α Xβ, i, I x Lex(X(D, } n T I = I {(N αx β, i, (X, T (N α Xβ, i, I} Figure 8. New nner in D ording to I. It then determines whih of those ndte terminls n tully e lexed s toens t tht position. It pplies the seletor el to it nd returns the resulting set of toens. Finlly, the n opertor is esily dpted to wor on toens insted of single hrters. We itertively ompute the sets T 0, T 1,... of toens t position, nd these t s rguments T to oth Toens nd n. ordingly, we need to updte the definition of π to te the dditionl rgument T into ount: π T I = limit (n T Complete Predit I. Furthermore, it is now possile tht snning t position might dd new items to in due to the existene of empty toens, therefore enlrging the set of eligile terminls t position. To ope with this we eep pplying the opertor π with updted toen sets until it onverges: J 0 0 = π 0 Init J u1 = π T u1 I = u=0 J u J 0 1 = π 1 I T 0 T u1 = J u = Toens T u J u. Note tht in se of J u1 = J u we hve I = J u, thus the omputtion of I n stop t tht point. In prtiulr, if T 1 does not ontin ny empty toens, then the omputtion of I simplifies to I = J 1. ove equtions for omputing I = I D show n ovious orrespondene to the equtions we used for defining P in etion 2. nd indeed, rley s lgorithm with lol lexing is orret with respet to the lol lexing semntis: Theorem 4.2 (Corretness of rley s lgorithm with Lol Lexing. rley s lgorithm with lol lexing is oth sound nd omplete, i.e. D L Σ iff α. ( α, 0, D I. Proof. ee etion 5.

9 The ove orretness result omes with vet: It my e the se tht the omputtion requires n infinite mount of time nd spe. If the grmmr is finite though, then for n input D Σ the size of I is ounded y (( D 1 D 1 1 α, 2 N α R nd thus the omputtion will require only finite mount of time nd spe (ssuming el nd Lex require only finite mount in the first ple. Prtil Implementtion We hve developed prtil lirry for lol lexing written in l/l.js whih n e used to try out the exmples in etion 3 [13]. It is firly nive proof-of-onept implementtion nd not optimized for dt strutures t ll. For future versions of the lirry we pln to exmine whih of the mny estlished es for ming rley prsing fster lso pply (t lest prtilly to our se. 5. Proof of Theorem 4.2 In this setion we give short outline of the proof of Theorem 4.2. The full forml proof is ville s Iselle/HOL 2016 theory files nd hs een fully mhine-heed for orretness [12]. To onvine yourself tht the forml proof relly proves Theorem 4.2 we reommend first studying theories CFG, LolLexing nd LLrleyPrsing. These ontin the si definitions in (lmost the sme nottion s presented in this pper. You should then proeed to loo t theory MinTheorems. It ontins the theorem Corretness whih is the mhine-heed ounterprt of Theorem 4.2. Proof Outline There is n intuitive orrespondene etween the sets P u nd the sets J u. Clrifying this orrespondene is the most importnt step towrds proving the orretness of the lgorithm. Definition 5.1 (Vl nd Generted Items. We ll n item (N α β, i, j I p-vl for some toen sequene p P iff there is u {0,..., p } suh tht p = j, p 0... p u 1 = i, [p 0... p u 1 ]Nγ for some γ, α [p u... p p 1 ]. For P P we sy tht P genertes P, where P = {x I p P. x is p-vl}. This notion of vlity hs een inspired y the one introdued in [9] where it hs een defined s n solute property of n item. To me it wor in our ontext, we hd to define it not solutely, ut reltively with respet to pth. The ul of the proof onsists then in proving I = P. We will not delve into the rther tehnil proof of this here, ut we wnt to point out the following supporting theorem out pths: Theorem 5.1. For ll inputs D, for ll {0,..., D } nd for ll u {0, 1, 2,...} the following holds: Given p, q P u suh tht p = q 0... q n 1 for some n {0,..., q } nd [p q n... q q 1 ] L prefix, it follows tht p q n... q q 1 P u. Intuitively this mens tht when there re two pths p P nd q P whih meet t some position, i.e. p = nd q = d with = =, then they n rossover, i.e. oth d nd will e in P s long s this mes sense with respet to the grmmr. But p nd q re not gurnteed to rrive t the sme time u t position, nd so it might e tht toens tht were round when p rrived re not there nymore when q rrives, nd vie vers. The ft tht the toen sets Z u form monotone hin t position mens tht this nnot hppen. The semntis of lol lexing is defined y mutully reursive equtions whih intertwine lexing nd prsing, ut one ll toen sets Z = u=0 Zu hve een estlished it is possile to disentngle lexil nd grmmtil mtters gin s nother supporting theorem out pths shows: Theorem 5.2. Let p e sequene of toens. Then p P iff p i Z p 0...p i 1 nd [p] L prefix. 0 i< p From I = P nd with the help of Theorem 5.2 it is then strightforwrd to prove Theorem 4.2: Proof. Let us first ssume D L Σ. This implies ll(d, whih implies tht there is p P with p = D nd [p]. This mens there is n α suh tht α nd α [p]. Therefore, ( α, 0, D is p-vl nd thus ( α, 0, D P = I. On the other hnd, let us ssume tht there is n α with ove property. This mens tht ( α, 0, D is p-vl for some p P, i.e. there is u {0,..., p } with p = D, p 0... p u 1 = 0, [p 0... p u 1 ]γ for some γ, α [p u... p p 1 ]. ove fts together with Theorem 5.2 show tht y dropping the first u empty toens from p we otin pth p u... p p 1 ll(d.

10 6. Further Relted Wor strong influene on our wor hs een the e of lox rley prsing presented in [1]. lox is (possily third-prty prser omponent plugged into the rley prsing frmewor y ssoiting the lox with nonterminl. Our Lex omponent n silly e viewed s olletion of loxes, ut insted of ssoiting them with nonterminls, we ssoite them with terminls. This mes it possile to tret loxes s onept tht is in priniple independent from rley prsing. Unlie the originl wor on loxes we lso prove method for dismigution, vi the seletor el. 7. Conlusion With hindsight lol lexing is simple onept, ut it hs ten us over two yers to rrive t the onept s it is presented in this pper. Our lgorithm for lol lexing nd the semntis of lol lexing developed se-y-se during this time. There hve een enough missteps long the wy to finlly me us formlly verify our lgorithm. Despite its simpliity, the exmples from etion 3 show tht lol lexing is verstile nd unifying onept for designing syntx. We hope tht you my find it useful, too. Referenes [1] Trevor Jim, Yitzh Mndelum, Dv Wler. emntis nd lgorithms for Dt-dependent Grmmrs, POPL [2] Brin Ford. Prsing xpression Grmmrs: Reognition Bsed yntti Foundtion, POPL [3] elo Visser. nnerless Generlized-LR Prsing, Report P9707, University of msterdm [4] The lexer h, [5] John yo, R. Nigel Horspool. hrödinger s toen, oftwre Prtie nd xperiene, [6] Iselle, [7] l, [8] éstien Doerene. l.js, [9] Cliff B. Jones. Forml Development Of Corret lgorithms: n xmple Bsed On rley s Reogniser, Proving ssertions out Progrms, [10] Jeffrey Kegler. The Mrp prser, jeffreyegler.githu.io/mrp-we-site/. [11] Jeffrey Kegler. Mrp nd the Ruy lippers, logs.perl.org/users/jeffrey_egler/2011/ 11/mrp-nd-the-ruy-slippers.html. [12] teven Ou. Iselle Theories for Lol Lexing (v1.1.0, doi: /zenodo [13] teven Ou. Lol Lexing Prototype Implementtion (v1.0.1, doi: /zenodo