Flow Interfaces Compositional Abstractions of Concurrent Data Structures. Siddharth Krishna, Dennis Shasha, and Thomas Wies

Transcription

1 Flow Interfaces Compositional Abstractions of Concurrent Data Structures Siddharth Krishna, Dennis Shasha, and Thomas Wies

2 Background Verifying programs, separation logic, inductive predicates Slides courtesy of Thomas Wies

3 Verifying Data Structures procedure delete(x: Node) { if (x!= null) { var y := x.next; delete(y); free(x); } x } y null

4 Proof by Hand-Waving { x }... null x = null procedure delete(x: Node) { { x } if (x!= null) { x... y null var y := x.next; { x } null delete(y); { } y free(x); } } { } {... }

5 Proof by Hand-Waving { x } null procedure delete(x: Node) { if (x!= null) { var y := x.next; delete(y); free(x); } } { }

6 Separation Logic by Example Points-to predicate x! y x y Stack x 0 y 42 Heap ? A partial heap consisting of one allocated cell

7 Separation Logic by Example Points-to predicate x! y x y Points-to predicate expresses permission to access (i.e. read/write/deallocate) heap location x and nothing else! SL assertions describe the part of the heap that a program is allowed to work with.

8 Separation Logic by Example Points-to predicate y! x x y Stack x 0 y 42 Heap 0? 42 0 A partial heap consisting of one allocated cell

9 Separation Logic by Example Separating conjunction x! y y! x x y Stack x 0 y 42 Heap Composition of disjoint partial heaps

10 Separation Logic by Example Equalities x! y x = z x,z y Stack x 0 y 42 z 0 Heap ? Equalities only constrain the stack

11 Separation Logic by Example Separating conjunction x! y x! z unsatisfiable? Subheaps must be disjoint (x can't be at two different places at once)

12 Separation Logic by Example Classical conjunction x! y y! x x,y?

13 Separation Logic by Example Separating conjunction x! z y! z 2 x = y still unsatisfiable? Convention: has higher precedence than

14 Inductive Predicates ls x x = null emp y. x y ls y ls x y. x y ls y y, z. x y y z ls z y, z, w. y, z, w. y, z, w. x y y z z w ls(w) x y y z z w w = null emp x y y z z null x y z null

15 Proof by SL ls(x) procedure delete(x: Node) { if (x!= null) { var y := x.next; delete(y); free(x); } } emp ls x x null x y ls(y) x y emp emp

16 Background The Trouble with Inductive Predicates Concurrent data structures are complex

17 Harris' Non-blocking List mh Z Z 8

18 Harris' Non-blocking List mh Z Z fh Z Z

19 Limitations of Inductive Predicates Traversals need to visit each node exactly once. ls mh, null ls fh, null mh fh 2 5 Overlays

20 Limitations of Inductive Predicates Traversals need to visit each node exactly once. harris mh, fh, null... harris( ) mh fh Unbounded sharing

21 Limitations of Inductive Predicates harris mh, fh, null ls(mh, null) mh fh 2 5 Threads can enter main list at arbitrary points

22 Other Approaches Overlapping conjunction: mh fh 2 5 Potentially cyclic complex predicate Updates: ramifications, reason about

23 Other Approaches Iterated separating conjunction: C E φ(x) mh fh 2 5 Can t do better than closed set of nodes Every node reachable memory leaks

24 Other Issues Entailment lemmas are specific to data structure. Example: but ls x, y ls y, z ls(x, z) sls x, y sls y, z sls x, z Solution: add lower and upper bounds sls(x, y, l, u) sls x, y, l, v sls y, z, w, u v w sls(x, z, l, u) Different composition lemma!

25 The Problem An abstraction mechanism that can handle data structures like the Harris list? (i.e. handle overlays, sharing, arbitrary traversals & have easy reasoning)

26 Background The Trouble with Inductive Predicates Flow Interfaces A new abstraction for concurrent data structures

27 The Idea Inductive predicates: Pro: inductive properties Con: fixed traversals Iterated : C E φ(x) Pro: easy reasoning Con: only local properties Best of both? Inductive properties local conditions But allow dependence on inductive quantity: flow

28 The Idea Express all invariants w.r.t. local condition on nodes and flow. Introduce a graph composition that preserves local invariants on global flows. Introduce a generic predicate to abstract a heap region satisfying the local flow condition (the flow interface).

29 Local Data Structure Invariants with Flows root Path counting! Can we express the property that root points to a tree as a local condition of each node in the graph?

30 Flows Step : Define the graph root 0 Graph G = (N, e) N finite set of nodes e: N N D D is a flow domain For example: D = N { } Label each edge in the graph with.

31 Flows Step 2: Calculate the flow root 0 Given an inflow in: N D Here in n = ITE(n =root,, 0) 0 0 0

32 Flows Step 2: Calculate the flow root 0 0 Start with flow(n) = in(n) And iterate until fixpoint flow n = in n + flow n Y e(n Y, n) \Y

33 Flows Step 2: Calculate the flow root 0 0 Different inflows result in different flows

34 Flows Step 3: Define invariant on flow root If every node satisfies the good condition γ^ flow(in, G) n = 0 0 for in n = ITE(n = root,, 0) 0 0 then G is a tree rooted at root

35 Flows Step 3: Define invariant on flow root 0 For lists, enforce at most outgoing edge γ` flow in, G n = (e \ = { n, n Y } e \ = ε) e \ : edge function restricted to nonzero edges leaving n 0

36 Flows Alternate view Flow graph G = (N, E) Viewing E as a matrix, E c \\Y = d E \d E d\y = number of 2-length paths from n to n E ` \\Y = number of l-length paths from n to n Capacity C = I + E + E c + (converges if D is a flow domain) cap G n, n Y C \\Y = the number of paths from n to n flow in, G (n) in n Y cap(g)(n Y, n) \Y

37 Data Invariants 3 root 3 Flow domain: ω-cpo & positive semiring (D,,, +,, 0, ) Another Example: lower-bound domain (Z {, },, min, min, max,, ) Label each edge with value of souce node Use in n = ITE(n = root,, ) flow in, G n = max value in some path from root to n These paths are sorted if γ s flow in, G where a is value at n n a

38 Stacking Flows (, ) (, 3) (, 3) (0, ) root (0, ) 3 3 (0, ) (, 5) (, 5) 5 5 (0, ) Want shape & data invariants? Flow of product is product of flows! Example: min-heap Use product of path-counting and lowerbound domain Use in n = ITE(n = root, (, ), (0, )) And good condition γ t flow in, G where a is value at n n =, l l a

39 Stacking Flows (, 3) 3 (, 5) 5 (0, ) root (0, ) (, 5) 5 (, ) (, 3) (0, ) 3 (0, ) Data invariants are decoupled from shape invariants Min-heap γ t flow in, G Sorted list γ s` flow in, G (e \ = { n, n Y n =, l l a n =, l l a } e \ = ε)

40 Harris List We can now describe Harris List Flow domain: two path-counting flows One from mh and one from fh Every node is on at least one of these lists Nodes labelled: marked/unmarked All nodes in free list are marked mh fh 2 5

41 Expressivity of Flows Flows can describe: Lists (singly and doubly linked, cyclic) Trees (n-ary, arbitrary arity) Nested combinations Sorted lists, binary heaps, BSTs Overlaid structures (threaded and B-link trees) Irregular structures (DAGs, graphs) Unbounded sharing & irregular traversals (Harris) But inductive predicates easier for: Simple inductive structures/abstractions

42 Compositional Reasoning But can we reason compositionally about flows and graphs à la SL?

43 Graph Composition Standard SL Composition (disjoint union) is too weak: r r x * x = x y y y a tree a tree not a tree Need a composition that preserves flow!

44 Flow Graphs (in, G) is a flow graph iff G = (N, N o, e) is a partial graph with N the set of internal nodes of the graph r N o the set of sink nodes of the graph x e: N x (N N o ) D is an edge function in: N D is an inflow y Inflow in specifies rely of G on its context.

45 Flow Graph Composition (in, G) root

46 Flow Graph Composition (in, G) = (in, G ) (in 2, G 2 ) root G G If in x n = in n + flow in, G)(n Y e(n Y, n) \ y z {

47 Flow Graph Composition H H 2 is commutative: H H 2 = H 2 H associative : (H H 2 ) H 3 = H (H 2 H 3 ) cancelative: H H = H H 2 H = H 2 Flow graphs form a separation algebra. We can use them to give semantics to SL assertions. How do we abstract flow graphs?

48 Modifying the Graph (in, G) = (in, G ) (in 2, G 2 ) (in, G ) (in, G ) (in 2, G 2 ) root G G G Need to preserve the flow into G 2

49 Modifying the Graph (in, G) = (in, G ) (in 2, G 2 ) (in, G ) = (in, G ) (in 2, G 2 ) root G G G Preserve the number of paths going through G?

50 Flow Map of a Flow Graph n n o flm in, G n, n G flm in, G n, n = e n, n x e n }, n over all paths in G flow in, G n = in n flm in, G (n, n ) \ z

51 Flow Map: Example

52 Flow Map: Example Flow map abstracts from internal structure of the graph

53 Flow Map: Example Flow map abstracts from internal structure of the graph

54 Flow Map: Example 0 0 Flow map abstracts from internal structure of the graph

55 Flow Interfaces I = (in, a, f) is a flow interface in: N D is an inflow a A is the abstraction of node labels f: N N o D is a flow map r ({r, }, _, { r, y }) x y

56 Flow Interfaces Flow interfaces induce a congruence on flow graphs: H, H Y I H x H c I H x Y H c Y I Lift flow graph composition to interfaces: (Flow Interfaces, ) also a separation algebra!

57 Reasoning about Modifications congruence can replace G x with any G x Y with same interface r r x y x y Showing equivalence: requires fixpoint reasoning But concurrent algos modify small regions

58 Separation Logic with Flow Interfaces Good graph predicate Gr γ (I) γ: SL predicate that defines good node condition and abstraction of heap onto nodes of flow graph I: flow interface Good node predicate N γ (x, I) like Gr but denotes a single node definable in terms of Gr Dirty region predicate [P] γ,i P: SL predicate denotes heap region that is expected to satisfy interface I but may currently not

59 Example Sorted Linked List root (, ) (, 3) 3 (, 5) γ x, in, a, f n k, n Y in n =, l a = {k} l k f = ITE(n Y = null, ε, { n, n Y }) Global invariant: Gr I I \ + 0 = {r, } + 0 I = ε 5

60 Harris List Global invariant: Φ I. Gr I I \ + 0 = {mh, 0 } + {fh, 0 } + 0 I = ε

61 Data-Structure-Agnostic Proof Rules Decomposition Gr I x I I x, I c. N x, I x Gr I c I = I x I c Step I = I x I c x, y I x I = ε y I c Composition Gr I x Gr I c Gr I x I c

62 Harris: Insert procedure insert() { var l := mh; Φ var r := unmarked(l.next); while (r!= null &&?? ) { l := r; r := unmarked(l.next); } } Gr I mh I \ (Decomp) N l, I` Gr I c I = I` I c

63 Harris: Insert procedure insert() { } var l := mh; Φ var r := unmarked(l.next); while (r!= null &&?? ) { l := r; r := unmarked(l.next); } N l, I` Gr I c I = I` I c I = I` I c l, r I` I = ε (Step), (Decomp) N l, I` N r, I Gr I Ž I = I` I I Ž

64 Harris: Insert procedure insert() { } var l := mh; var r := unmarked(l.next); while (r!= null &&?? ) { } l := r; Φ r := unmarked(l.next); N l, I` Gr I c I = I` I c N l, I` N r, I Gr I Ž I = I` I I Ž (Comp) N r, I Gr I I = I I

65 Harris: Insert procedure insert() { var l := mh; Φ var r := unmarked(l.next); while (r!= null &&?? ) { l := r; r := unmarked(l.next); } } N l, I` Gr I c I = I` I c N l, I` Gr I I = I` I

66 Flow Interfaces Data structure abstractions that can handle unbounded sharing and overlays treat structural and data constraints uniformly do not encode specific traversal strategies provide data-structure-agnostic composition and decomposition rules remain within general theory of separation logic

67 Background The Trouble with Inductive Predicates Flow Interfaces Concurrent Dictionaries A memory-safe, linearizable, abstract template

68 Concurrent Dictionaries Dictionary: key-value store Examples: sorted linked lists, BSTs, B-trees, hash maps, With flows: generic template + proof

69 Inset Flows root 6 {v v < 6} {v v > 6} 3 8 KS: the set of all search keys e.g. KS = Int Inset flow domain: (2,,,,,, KS) {v v < 3} {v v > 3} {v v > 8} 4 9 Label each edge with the set of keys that follow that edge in a search (edgeset).

70 Inset Flows root KS {v v < 6} {v v > 6} KS: the set of all search keys e.g. KS = Int Inset flow domain: (2,,,,,, KS) {v v < 3} {v v > 3} {v v > 8} Set inflow in of root to KS and to for all other nodes.

71 Inset Flows root {v v < 6} KS {v v > 6} I = {v 3 < v} I 2 = {v 3 < v < 6} {v v < 3} {v v > 3} {v v > 8} I 3 = {v 8 < v} I I 2 I 3 flow(in, G)(n) is the inset of node n, i.e., the set of keys k such that a search for k will traverse node n.

72 From Insets to Keysets outset(g) n = š e(n, n Y ) \ v v < 5 v } 5 {5} 7 {v v > 5} keyset(in, G) n = inset(in, G) n outset(g)(n) 3 v < v < 5} keyset(in, G)(n) is the set of keys that could be in n

73 Verifying Concurrent Dictionaries Good state conditions edgesets are disjoint for each n: {e(n,n')} n' N are disjoint keyset of each n covers n's contents: C(G)(n) keyset(in, G)(n) Keyset Theorem [Shasha and Goodman, 988] If all ops preserve good state, and methods search/insert/delete k at node s.t. k keyset(n) The implementation is linearizable

74 Encoding Using Flows Description of good node Description of locked node Contents are in keyset Edgesets leaving node are disjoint Global invariant:

75 Give-up Template

76 Actions

77 Keyset Theorem with Flows Theorem: An implementation of the template with appropriate γ, γ that satisfy the spec below (with some side conditions) is memory safe and linearizable.

78 Background The Trouble with Inductive Predicates Flow Interfaces Concurrent Dictionaries Conclusion A new way to reason about data structures

79 Conclusion Radically new approach for building compositional abstractions of data structures. Fits in existing (concurrent) separation logics. Enables simple correctness proofs of concurrent data structure algorithms. Proofs can abstract from the details of the specific data structure implementation. Siddharth Krishna, Dennis Shasha, and Thomas Wies. Go with the Flow: Compositional Abstractions for Concurrent Data Structures. POPL 208.