I Can t Believe It s Not Causal! Scalable Causal Consistency with No Slowdown Cascades

Transcription

1 I Can t Believe It s Not Causal! Scalable Causal Consistency with No Slowdown Cascades Syed Akbar Mehdi 1, Cody Littley 1, Natacha Crooks 1, Lorenzo Alvisi 1,4, Nathan Bronson 2, Wyatt Lloyd 3 1 UT Austin, 2 Facebook, 3 USC, 4 Cornell University

2 Causal Consistency: Great In Theory Higher Perf. Eventual Consistency Causal Consistency Strong Consistency Stronger Guarantees Lots of exciting research building scalable causal data-stores, e.g., Ø COPS [SOSP 11] Ø Eiger [NSDI 13] Ø Cure [ICDCS 16] Ø Bolt-On [SIGMOD 13] Ø Orbe [SOCC 13] Ø TARDiS [SIGMOD 16] Ø Chain Reaction [EuroSys 13] Ø GentleRain [SOCC 14]

3 Causal Consistency: But In Practice The middle child of consistency models Reality: Largest web apps use eventual consistency, e.g., Espresso TAO Manhattan

4 Key Hurdle: Slowdown Cascades Implicit Assumption of Current Causal Systems Reality at Scale

5 Key Hurdle: Slowdown Cascades Wait Enforce Consistency Wait Implicit Assumption of Current Causal Systems Reality at Scale Slowdown Cascade

6 Replicated and sharded storage for a social network

7 W 1 Writes causally ordered as W " W $ W %

8 W 1 W 1 Applied? W 2 Buffered W 2 Applied? W 3 Buffered Current causal systems enforce consistency as a datastore invariant

9 Slowdown Cascade W 1 W 1 Delayed Buffered consistency inside the data store Applied? W 1 W 2 Applied? Slowdown cascades affect all previous W causal systems because they 2 enforce W 3 Buffered Alice s advisor unnecessarily waits for Justin Bieber s update despite not reading it

10 Buffered Replicated Writes Slowdown Cascades in Eiger (NSDI 13) Replicated write buffers grow arbitrarily because Eiger enforces consistency inside the datastore Replicated writes received Normal Slowdown

11 OCCULT Observable Causal Consistency Using Lossy Timestamps

12 Observable Causal Consistency Causal Consistency guarantees that each client observes a monotonically non-decreasing set of updates (including its own) in an order that respects potential causality between operations Key Idea: Don t implement a causally consistent data store Let clients observe a causally consistent data store

13 How do clients observe a causally consistent datastore?

14 Writes accepted only by master shards and then replicated asynchronously in-order to slaves

15 Each shard keeps track of a shardstamp which counts the writes it has applied

16 Client Client 3 Client 2 Causal Timestamp: Vector of shardstamps which identifies a global state across all shards

17 78 7 a Client Client 3 Client 2 Write Protocol: Causal timestamps stored with objects to propagate dependencies

18 8 7 a Client Client 3 Client 2 Write Protocol: Server shardstamp is incremented and merged into causal timestamps

19 8 7 a Client Client 3 Client 2 Read Protocol: Always safe to read from master

20 a Client Client 3 Client 2 Read Protocol: Object s causal timestamp merged into client s causal timestamp

21 8 7 a Client b Client 3 Client 2 Read Protocol: Causal timestamp merging tracks causal ordering for writes following reads

22 8 7 a Client b Client 3 Client 2 Replication: Like eventual consistency; asynchronous, unordered, writes applied immediately

23 8 7 a a Delayed! Client Client b b Client 3 Replication: s increment their shardstamps using causal timestamp of a replicated write

24 8 7 a a Client Client Delayed! 8 8 b b 8 5 5? Client 3 Read Protocol: Clients do consistency check when reading from slaves

25 8 7 a a Client Client 2 Delayed! 5 5 b b b s dependencies are delayed, but we can read it anyway! ? Client 3 Read Protocol: Clients do consistency check when reading from slaves

26 ? a a Stale Shard! Delayed! Client Client b b Client 3 Read Protocol: Clients do consistency check when reading from slaves

27 ? a a Stale Shard! Delayed! Client Client b b Client 3 Options: 1. Retry locally 2. Read from master Read Protocol: Resolving stale reads

28 Causal Timestamp Compression What happens at scale when number of shards is (say) 100,000? Size(Causal Timestamp) == 100,000?

29 Causal Timestamp Compression: Strawman To compress down to n, conflate shardstamps with same ids modulo n Problem: False Dependencies Compress

30 Causal Timestamp Compression: Strawman To compress down to n, conflate shardstamps with same ids modulo n Compress Problem: False Dependencies Solution: Use system clock as the next value of shardstamp on a write Decouples shardstamp value from number of writes on each shard

31 Causal Timestamp Compression: Strawman To compress from N to n, conflate shardstamps with same ids modulo n Compress Problem: Modulo arithmetic still conflates unrelated shardstamps

32 Causal Timestamp Compression Insight: Recent shardstamps more likely to create false dependencies Use high resolution for recent shardstamps and conflate the rest Shardstamps Shard IDs * Catch-all shardstamp

33 Causal Timestamp Compression Insight: Recent shardstamps more likely to create false dependencies Use high resolution for recent shardstamps and conflate the rest Shardstamps Shard IDs * Catch-all shardstamp 0.01 % false dependencies with just 4 shardstamps and 16K logical shards

34 Transactions in OCCULT Scalable causally consistent general purpose transactions

35 Properties of Transactions A. Atomicity B. Read from a causally consistent snapshot C. No concurrent conflicting writes

36 Properties of Transactions A. Observable atomicity B. Observably read from a causally consistent snapshot C. No concurrent conflicting writes

37 Properties of Transactions A. Observable Atomicity B. Observably read from a causally consistent snapshot C. No concurrent conflicting writes Properties of Protocol 1. No centralized timestamp authorities (e.g. per-datacenter) Transactions ordered using causal timestamps

38 Properties of Transactions A. Observable Atomicity B. Observably read from a causally consistent snapshot C. No concurrent conflicting writes Properties of Protocol 1. No centralized timestamp authority (e.g. per-datacenter) Transactions ordered using causal timestamps 2. Transaction commit latency is independent of number of replicas

39 A. Observable Atomicity B. Observably read from causally consistent snapshot C. No concurrent conflicting writes 1. Read Phase Buffer writes at client 2. Validation Phase Client validates A, B and C using causal timestamps 3. Commit Phase Properties of Transactions Three Phase Protocol Buffered writes committed in an observably atomic way

40 0 a = [] 0 a = [] b = [Bob] b = [Bob] c = [Cal] c = [Cal] Alice and her advisor are managing lists of students for three courses

41 0 a = [] 0 a = [] b = [Bob] b = [Bob] c = [Cal] c = [Cal] Observable atomicity and causally consistent snapshot reads enforced by same mechanism

42 0 a = [] 0 a = [] Start T 1 r(a) = [] w(a = [Abe]) 1 1 b = [Bob] b = [Bob] c = [Cal] c = [Cal] Transaction T 1 : Alice adding Abe to course a

43 1 0 a = [Abe] a = [] Start T 1 r(a) = [] w(a = [Abe]) Commit T b = [Bob] b = [Bob] c = [Cal] c = [Cal] Transaction T 1 : After Commit

44 1 0 a = [Abe] a = [] Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) 1 1 b = [Bob] b = [Bob] c = [Cal] c = [Cal] Transaction T 2 : Alice moving Bob from course b to course c

45 1 a = [Abe] Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) 12 1 b = [Bob] 12 1 c = [Cal] 0 a = [] Atomicity through causality: b = [Bob] Make writes dependent on each other c = [Cal] Observable Atomicity: Make writes causally dependent on each other

46 1 a = [Abe] Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) 1 1 b = [Bob] 1 1 c = [Cal] 0 a = [] b = [Bob] c = [Cal] Observable Atomicity: Same commit timestamp makes writes causally dependent on each other

47 1 a = [Abe] Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] 2 1 c = [Bob, Cal] 0 a = [] b = [Bob] c = [Cal] Observable Atomicity: Same commit timestamp makes writes causally dependent on each other

48 1 0 a = [Abe] a = [] Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] c = [Bob, Cal] c = [Cal] Transaction writes replicate asynchronously

49 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Transaction writes replicate asynchronously

50 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Alice s advisor reads the lists in a transaction Start T 3

51 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Alice s advisor reads the lists in a transaction Start T 3 r(b) = [Bob]

52 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Start T 3 r(b) = [Bob] T 3 Read Set b = [Bob] Transactions maintain a Read Set to validate atomicity and causal snapshot reads

53 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Start T 3 r(b) = [Bob] r(c) = [Bob,Cal] T 3 Read Set b = [Bob] c = [Bob, Cal] Transactions maintain a Read Set to validate atomicity and read from causal snapshot

54 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Start T 3 r(b) = [Bob] r(c) = [Bob,Cal] T 3 Read Set b = [Bob] c = [Bob, Cal] Validation failure: c knows more writes from grey shard than applied at the time b was read

55 1 0 a = [Abe] a = [] Delayed! Start T 1 r(a) = [] w(a = [Abe]) Commit T 1 Start T 2 r(b) = [Bob] r(c) = [Cal] w(b = []) w(c = [Bob, Cal]) Commit T b = [] b = [Bob] Delayed! c = [Bob, Cal] c = [Bob, Cal] Start T 3 r(b) = [Bob] r(c) = [Bob,Cal] r(a) = [] T 3 Read Set b = [Bob] c = [Bob, Cal] Ordering Violation: Detected in the usual way. Red Shard is stale!

56 A. Observable Atomicity B. Observably read from causally consistent snapshot C. No concurrent conflicting writes 1. Read Phase Buffer writes at client 2. Validation Phase Client validates A, B and C using causal timestamps 3. Commit Phase Properties of Transactions Three Phase Protocol Buffered writes committed in an observably atomic way

57 A. Observable Atomicity B. Observably read from causally consistent snapshot C. No concurrent conflicting writes 1. Read Phase Buffer writes at client 2. Validation Phase 2. Validation Phase a. Validate Read Set to verify A and B Client validates A, B b. and Validate C using Overwrite causal timestamps Set to verify C 3. Commit Phase Properties of Transactions Three Phase Protocol Buffered writes committed in an observably atomic way

58 Evaluation

59 Evaluation Setup Occult implemented by modifying Redis Cluster (baseline) Evaluated on CloudLab Two datacenters in WI and SC 20 server machines (4 server processes per machine) 16K logical shards YCSB used as the benchmark For graphs shown here read-heavy (95% reads) workload with zipfian distribution

60 Evaluation Setup Occult implemented by modifying Redis Cluster (baseline) Evaluated on CloudLab Two datacenters in WI and SC 20 server machines (4 server processes per machine) 16K logical shards YCSB used as the benchmark For graphs shown here read-heavy (95% reads) workload with zipfian distribution We show cost of providing consistency guarantees

61 Goodput Comparison Goodput (million ops/s) Occult Transactions Occult Single-Key Redis Cluster Num Ops per Transaction (T size )

62 Goodput Comparison Goodput (million ops/s) % Occult Transactions Occult Single-Key Redis Cluster Num Ops per Transaction (T size ) 4 shardstamps per causal timestamp 8.7% 39.6%

63 Log 10 (Latency us) 280us 390us 800us 1.6ms 3.7ms Effect of slow nodes on Occult Latency ms th 75th 90th 95th 99th Percentiles slow nodes

64 Conclusions Enforcing causal consistency in the data store is vulnerable to slowdown cascades Sufficient to ensure that clients observe causal consistency: Use lossy timestamps to provide the guarantee Avoid slowdown cascades Observable enforcement can be extended to causally consistent transactions Make writes causally dependent on each other to observe atomicity Also avoids slowdown cascades