A Universal Calculus for Stream Processing Languages

Transcription

1 1 A Universal Calculus for Stream Processing Languages Robert Soulé, Martin Hirzel, Robert Grimm, Buğra Gedik, Henrique Andrade, Vibhore Kumar, and Kun-Lung Wu New York University and IBM Research

2 2 Stream Processing Is Everywhere Move large amounts of data through several computational steps

3 3 Stream Processing Has Many Flavors StreamIt: synchronous CQL: SQL + streams processing Sawzall: functional data processing in the large

4 4 Stream Processing Has Many Implementations StreamIt: synchronous CQL: SQL + streams processing Sawzall: functional data processing in the large

5 5 Variety Breeds Confusion We want to understand and compare streaming languages What is their expressiveness? How to optimize the data processing steps? How to scale the overall applications? Especially across clusters? Enter our universal calculus: Brooklet Formal foundation for answering the above questions Provably correct optimizations and translations

6 6 Outline of This Talk Motivation Requirements The Brooklet Core Calculus Generality: Translating StreamIt, CQL, and Sawzall to Brooklet Utility: Optimizing Brooklet to Brooklet Outlook and Conclusions

7 7 Elements of a Streaming App State Queue Operator Queue

8 Elements of a Streaming App 8

9 9 Elements of a Streaming App Not all operators have state

10 10 Elements of a Streaming App Operators may share state

11 11 Requirements for Calculus Make trigger Make nondeterministic nondeterministic Make explicit Treat functions as opaque Make explicit and 1-to-1

12 12 Brooklet Syntax $total trades volume Sum (volume, $total) Sum(trades, $total)

13 13 Function Environment $total trades volume Sum Sum F: The function implementations

14 14 Queue Store $total trades trades Sum volume volume Q: The contents of the queues

15 15 Variable Store $total trades volume Sum V: The contents of the variables

16 16 Brooklet Operational Semantics $total trades volume Sum F <Q, V> <Q, V >

17 17 Complete Calculus Brooklet syntax: P b ::= out in op Brooklet program out ::= output q ; Output declaration in ::= input q ; Input declaration op ::= ( q, v ) f ( q, v ); Operator q ::= id Queue identifier v ::= $ id Variable identifier f ::= id Function identifier Brooklet example: IBM market maker. output result; input bids, asks; (ibmbids) SelectIBM(bids); (ibmasks) SelectIBM(asks); ($lastask) Window(ibmAsks); (ibmsales) SaleJoin(ibmBids,$lastAsk); (result,$cnt) Count(ibmSales,$cnt); Brooklet semantics: F b V, Q V,Q d, b = Q(q i ) op = (_, _) f(q, v); (b, d )=F b (f)(d, i, V (v)) V = updatev (op,v,d ) Q = updateq(op, Q, q i, b ) F b V, Q V,Q (E-FireQueue) op = (_, v) f(_, _); updatev (op, V, d) =[v d]v op = (q, _) f(_, _); d f,b f = Q(q f ) Q =[q f b f ]Q Q =[ q i q : q i Q(q i ),b i ]Q updateq(op, Q, q f, b) =Q (E-UpdateV) (E-UpdateQ)

18 18 Example: A Fannie Mae Bid/Ask Join $lastask $total asks SaleJoin trades Sum volume bids $lastbid

19 19 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,0, > $total = 0 <FNM,1, 2> asks $lastask $total bids SaleJoin trades Sum volume <FNM,1, 1> <FNM,1, 2> $lastbid $lastbid = <FNM,0,0>

20 20 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,0, > $total = 0 <FNM,1, 2> asks $lastask $total bids SaleJoin trades Sum volume <FNM,1, 1> <FNM,1, 2> $lastbid $lastbid = <FNM,0,0>

21 21 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,0, > $total = 0 <FNM,1, 2> asks $lastask $total bids SaleJoin trades Sum volume <FNM,1, 2> $lastbid $lastbid = <FNM,1, 1>

22 22 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,0, > $total = 0 <FNM,1, 2> asks $lastask $total bids SaleJoin trades Sum volume <FNM,1, 2> $lastbid $lastbid = <FNM,1, 1>

23 23 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,0, > $total = 0 <FNM,1, 2> asks $lastask $total bids SaleJoin trades Sum volume <FNM,1, 2> $lastbid $lastbid = <FNM,1, 1>

24 24 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 0 $lastask $total asks SaleJoin trades Sum volume bids <FNM,1, 2> $lastbid $lastbid = <FNM,1, 1>

25 25 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 0 $lastask $total asks SaleJoin trades Sum volume bids <FNM,1, 2> $lastbid $lastbid = <FNM,1, 1>

26 26 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 0 $lastask $total asks SaleJoin trades Sum volume bids <FNM,1, 2> $lastbid $lastbid = <FNM,1, 1>

27 27 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 0 $lastask $total asks bids SaleJoin <FNM,1, 2> trades Sum volume $lastbid $lastbid = <FNM,1, 1>

28 28 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 0 $lastask $total asks bids SaleJoin <FNM,1, 2> trades Sum volume $lastbid $lastbid = <FNM,1, 2>

29 29 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $lastask $total = 0 $total asks bids SaleJoin <FNM,1, 2> trades Sum volume $lastbid $lastbid = <FNM,1, 2>

30 30 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 0 $lastask $total asks bids SaleJoin trades <FNM,1> Sum volume $lastbid $lastbid = <FNM,1, 2>

31 31 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 1 $lastask $total asks bids SaleJoin trades <FNM,1> Sum volume $lastbid $lastbid = <FNM,1, 2>

32 32 Example: A Fannie Mae Bid/Ask Join $lastask = <FNM,1, 2> $total = 1 $lastask $total asks bids SaleJoin trades <FNM,1> Sum volume $lastbid $lastbid = <FNM,1, 2>

33 33 Translations Demonstrating Brooklet s generality by translating three rather diverse streaming languages

34 34 CQL, StreamIt, Sawzall: One Translation Approach Do not model local computations Expose graph topology Expose implicit and explicit state Make queues explicit and 1-to-1 Wrap original operators in higher-order functions Make state explicit <, > Functions Queues Variables

35 35 Example: CQL to Brooklet CQL Brooklet select Sum(shares) from trades where trades.ticker = FNM Make queues explicit and 1-to-1 $total Make state explicit trades Filter FNM-trades Sum volume

36 36 Example: CQL to Brooklet CQL Brooklet select Sum(shares) from trades where trades.ticker = FNM Dynamically adapt runtime arguments $total Statically bind the original function trades Filter FNM-trades volume

37 37 Translation Correctness Theorem CQL/StreamIt Input execute CQL/StreamIt Output translate translate Brooklet Input execute Brooklet Output Results under CQL and StreamIt semantics are the same as the results under Brooklet semantics after translation First formal semantics for Sawzall

38 38 Optimizations Demonstrating Brooklet s utility by realizing three essential optimizations

39 39 Operator Fusion: Eliminate Queueing Delays before after Look for connected operators, whose state isn t used anywhere else

40 40 Operator Fission: Process More Data in Parallel before after Split Join Look for stateless operators

41 41 Operator Reordering: Filter Data Early Filter before after Filter Look for operators whose read/write sets don t overlap [Ghelli et al., SIGMOD 08]

42 42 Outlook More optimizations dynamic operator placement, load balancing, subquery sharing, eliminating spurious synchronization Richer extended calculus Types, verify progress, time constraints Common execution platform Practical challenges: data types, library of operators, serialization, process management, error handling

43 43 Conclusions Streaming is everywhere Media, finance, web applications Need a calculus to understand (distributed) implementations Minimal, non-deterministic, makes state and communication explicit Provide a formal and practical foundation for stream programming Mappings from CQL, StreamIt, and Sawzall Formalizing of Fusion, Fission, and Reordering

44 44

45 45

46 46 CQL Translation Rules CQL program translation: [ F c,p c ] p c = F b,p b [ F c, SName ] p c =, outputsname;inputsname; (T p c -SName) [ F c, RName ] p c =, outputrname;inputrname; (T p c -RName) F b, output q o ; input q; op = [F c,p cs ] p c q o = freshid() v = freshid() F b =[S2R wraps2r(f c(s2r))]f b op = op, (q o,v) S2R(q o,v); [ F c, S2R(P cs ) ] p c = F b, output q o ; input q; op (T p c -S2R) F b, output q o ; input q; op = [F c,p cr ] p c q o = freshid() v = freshid() F b =[R2S wrapr2s(f c(r2s))]f b op = op, (q o,v) R2S(q o,v); [ F c, R2S(P cr ) ] p c = F b, output q o ; input q; op (T p c -R2S) F b, output q o ; input q; op = [ F c,p cr ] p c n = P cr q o = freshid() q = q 1,...,q n i 1...n : v i = freshid() op = op 1,...,op n F b =[R2R wrapr2r(f c(r2r))]( F b ) op = op, (q o, v) R2R(q o, v); [ F c, R2R(P cr ) ] p c = F b, output q o ;input q ;op (T p c -R2R) T T CQL operator wrappers: σ, τ = d q s = d v s = s {e, τ : e σ} σ = f(s, τ) wraps2r(f)(d q, _,d v )=σ, τ,s σ, τ = d q σ = d v σ = f(σ, σ ) (W c -S2R) wrapr2s(f)(d q, _,d v )=σ, τ, σ (W c -R2S) σ, τ = d q d i = d i {σ, τ} j = i 1...n : d j = d j j 1...n : σ : σ, τ d j wrapr2r(f)(d q,i,d) =, d (W c -R2R-Wait) σ, τ = d q d i = d i {σ, τ} j = i 1...n : d j = d j j 1...n : σ j = aux(d j, τ) wrapr2r(f)(d q,i,d) =f(σ), τ, d σ, τ d aux(d, τ) =σ (W c -R2R-Ready) (W c -R2R-Aux)

47 47 Operator Fission i 1...n: q i = freshid() op = (q out ) f(q in ); i 1...n: q i = freshid() F b, op s =[, split roundrobin, q, q in ] p s i 1...n: op i = (q i) f(q i ); F b, op j =[, join roundrobin,q out, q ] p s F b, op N split F b F b F b, op s op op j