DReX: A Declarative Language for Efficiently Evaluating Regular String Transformations

Size: px

Start display at page:

Download "DReX: A Declarative Language for Efficiently Evaluating Regular String Transformations"

Baldric Craig
5 years ago
Views:

1 DReX: A Declarative Language for Efficiently Evaluating Regular String Transformations Rajeev Alur Loris D Antoni Mukund Raghothaman POPL

2 DReX is a DSL for String Transformations { Book1, title = {Title0}, author = { Author1 }, year = { Year1 }, { Book2, title = {Title1}, author = { Author2 }, year = { Year2 }, { Book1, title = {Title1}, author = { Author1 }, year = { Year1 }, }

3 Describing align-bibtex Using DReX The simpler issue of make-entry Given two entries, Entry 1 and Entry 2, make-entry outputs the title of Entry 2 and the remaining body of Entry 1 Entry 1 Entry 2 All but title Title only 3

4 Describing align-bibtex Using DReX align-bibtex = chain(make-entry, R Entry ) Entry 1 Entry 2 Entry 3 Entry k 1 Entry k make-entry (Entry 1 Entry 2 ) make-entry (Entry 2 Entry 3 ) make-entry (Entry 3 Entry 4 ) make-entry (Entry k 1 Entry k ) Function combinators such as chain combine smaller functions into bigger ones 4

5 Why DReX? DReX is declarative Languages, Σ bool Regular expressions Tranformations, Σ Γ DReX DReX is fast: Streaming evaluation algorithm for well-typed expressions Based on robust theoretical foundations Expressively equivalent to regular string transformations Multiple characterizations: two-way finite state transducers, MSO-definable graph transformations, streaming string transducers Closed under various operations: function composition, regular look-ahead etc. DReX supports algorithmic analysis Is the transformation well-defined for all inputs? Does the output always have some nice property? σ, is it the case that f (σ) L? Are two transformations equivalent? 5

6 DReX is publicly available! Go to drexonline.com 6

7 Function Combinators 7

8 Base functions: σ γ Map input string σ to γ, and undefined everywhere else.c.cpp σ Σ and γ Γ are constant strings Analogue of basic regular expressions: {σ}, for σ Σ 8

9 Conditionals: try f else g If f (σ) is defined, then output f (σ), and otherwise output g(σ) try [0-9] Number else [a-z] Name Analogue of unambiguous regex union 9

10 Split sum: split(f, g) Split σ into σ = σ 1 σ 2 with both f (σ 1 ) and g(σ 2 ) defined. If the split is unambiguous then split(f, g)(σ) = f (σ 1 )g(σ 2 ) σ 1 σ 2 f g f (σ 1 ) g(σ 2 ) Analogue of regex concatenation If title maps a BibTeX entry to its title, and body maps a BibTeX entry to the rest of its body, then make-entry = split(body, title) 10

11 Iterated sum: iterate(f ) Split σ = σ 1 σ 2... σ k, with all f (σ i ) defined. If the split is unambiguous, then output f (σ 1 )f (σ 2 )... f (σ k ) σ 1 σ 2 σ k f f f f (σ 1 ) f (σ 2 ) f (σ k ) Kleene-* If echo echoes a single character, then id = iterate(echo) is the identity function 11

12 Left-iterated sum: left-iterate(f ) Split σ = σ 1 σ 2... σ k, with all f (σ i ) defined. If the split is unambiguous, then output f (σ k )f (σ k 1 )... f (σ 1 ) σ 1 σ k 1 σ k f (σ k ) f (σ k 1 ) f (σ 1 ) Think of string reversal: left-iterate(echo) 12

13 Repeated sum: combine(f, g) combine(f, g)(σ) = f (σ)g(σ) σ f g f (σ) g(σ) No regex equivalent σ σσ: combine(id, id) 13

14 Chained sum: chain(f, R) σ 1 L(R) σ 2 L(R) σ 3 L(R) σ k L(R) f (σ 1 σ 2 ) f (σ 2 σ 3 ) f (σ 3 σ 4 ) f (σ k 1 σ k ) And similarly for left-chain(f, R) 14

15 Summary of Function Combinators Purpose Regular Transformations Regular Expressions Base, σ γ, {σ} Concatenation split(f, g), left-split(f, g) R 1 R 2 Union try f else g R 1 R 2 Kleene-* iterate(f ), left-iterate(f ) R Repetition combine(f, g) Chained sum chain(f, R), left-chain(f, R) New! 15

16 Regular String Transformations Or, why our choice of combinators was not arbitrary Languages, Σ bool DFA Tranformations, Σ Γ? 16

17 Historical Context Regular languages Beautiful theory Regular expressions DFA Analysis questions (mostly) efficiently decidable Lots of practical implementations 17

18 String Transducers One-way transducers: Mealy machines a/babc Folk knowledge [Aho et al 1969] Two-way transducers strictly more powerful than one-way transducers Gap includes many interesting transformations Examples: string reversal, copy, substring swap, etc. 18

19 String Transducers Two-way finite state transducers Known results Closed under composition [Chytil, Jákl 1977] Decidable equivalence checking [Gurari 1980] Equivalent to MSO-definable string transformations [Engelfriet, Hoogeboom 2001] Streaming string transducers: Equivalent one-way deterministic model with applications to the analysis of list-processing programs [Alur, Černý 2011] Two-way finite state transducers are our notion of regularity 19

20 Function Combinators are Expressively Complete Theorem (Completeness, Alur et al 2014) All regular string transformations can be expressed using the following combinators: Basic functions:, σ γ, split(f, g), left-split(f, g), try f else g, iterate(f ), left-iterate(f ), combine(f, g), chained sums: chain(f, R), and left-chain(f, R). 20

21 Evaluating DReX Expressions 21

22 The Anatomy of a Streaming Evaluator (a, 1) (b, 2) (b, 3) (Result, γ) (a, 4) (b, 5) (Result, γ ) (σ n, n) (Result, γ) (σ i, i) Evaluator for f 22

23 The Case of split(f, g) f defined g defined 1 i j n (σ i, i) T f (Result, γ) (σ i, i) T g (Result, γ) 23

24 The Case of split(f, g) f defined g defined 1 i j n (Start, i) (σ i, i) T f (Result, γ) (Start, i) (σ i, i) T g (Result, γ) 23

25 The Case of split(f, g) f defined g defined 1 f defined i j n (Start, i) (σ i, i) T f (Result, γ) (Start, i) (σ i, i) T g (Result, γ) 23

26 The Case of split(f, g) f defined g defined 1 f defined i j n (Start, i) (σ i, i) T f (Result, j, γ) (Start, i) (σ i, i) T g (Result, j, γ) 23

27 The Case of split(f, g) f defined g defined 1 f defined i j n (Start, i) (σ i, i) T f (Result, j, γ) (Start, i) (σ i, i) T g (Result, j, γ) Thread starting at index Index at which T f responded Result by T f 2 9 aaab 3 7 abbab reported 23

28 The Case of split(f, g) f defined g defined 1 f defined i j n (Start, i) (σ i, i) T f (Result, j, γ) (Kill, j) (Start, i) (σ i, i) T g (Result, j, γ) (Kill, j) Thread starting at index Index at which T f responded Result by T f 2 9 aaab 3 7 abbab reported 23

29 The Case of split(f, g) What if two threads of T g report results simultaneously? f defined g defined f defined g defined Statically disallow! split(f, g) is well-typed iff both f and g are well-typed, and their domains are unambiguously concatenable 24

30 Main Result Theorem 1. All regular string transformations can be expressed as well-typed DReX expressions. 2. DReX expressions can be type-checked in O(poly( f, Σ )). 3. Given a well-typed DReX expression f, and an input string σ, f (σ) can be computed in time O( σ, poly( f )). 25

31 Summary of Typing Rules, σ γ are always well-typed split(f, g) and left-split(f, g) are well-typed iff f and g are well-typed, and Dom(f ) and Dom(g) are unambiguously concatenable try f else g is well-typed iff f and g are well-typed, and Dom(f ) and Dom(g) are disjoint iterate(f ) and left-iterate(f ) are well-typed iff f is well-typed, and Dom(f ) is unambiguously iterable chain(f, R) and left-chain(f, R) are well-typed iff f is well-typed, R is an unambiguous regular expression, Dom(f ) is unambiguously iterable, and Dom(f ) = R R 26

32 Experimental Results 27

33 Experimental Results Streaming evaluation algorithm for well-typed expressions 8 7 seconds 6 delete-comm 5 insert-quotes 4 get-tags reverse 3 swap-bibtex 2 align-bibtex 1 0 characters align-bibtex has 3500 nodes in syntax tree, typechecks in half a second Type system did not get in the way 28

34 Conclusion Introduced a DSL for regular string transformations Described a fast streaming algorithm to evaluate well-typed expressions 29

35 Conclusion Summary of operators Purpose Regular Transformations Regular Expressions Base, σ γ, {σ} Concatenation split(f, g), left-split(f, g) R 1 R 2 Union try f else g R 1 R 2 Kleene-* iterate(f ), left-iterate(f ) R Repetition combine(f, g) Chained sum chain(f, R), left-chain(f, R) New! 30

36 Future Work Implement practical programmer assistance tools Static: Precondition computatation, equivalence checking Runtime: Debugging aids Theory of regular functions Automatically learn transformations from teachers (L*), from input / output examples, etc. Trees to trees / strings (Processing hierarchical data, XML documents, etc.) ω-strings to strings Non-regular extensions Count number of a-s in a string 31

37 Thank you! Questions? drexonline.com 32

38 What About Unrestricted DReX Expressions? 33

39 Evaluating Unrestricted DReX Expressions is Hard Or, why the typing rules are essential With function composition, it is PSPACE-complete combine(f, g) is defined iff both f and g are defined Flavour of regular expression intersection The best algorithms for this are either Non-elementary in regex size, or Cubic in length of input string 34

regular programming over data streams Mukund Raghothaman April 27, 2015

regular programming over data streams Mukund Raghothaman April 27, 2015 an introduction to drex the case of swapbibtex @book{gal1638, publisher={elzevir}, place={leiden}, year={1638}, title={two New Sciences},