A Piggybacking Design Framework for Read- and- Download- efficient Distributed Storage Codes. K. V. Rashmi, Nihar B. Shah, Kannan Ramchandran

Transcription

1 A Piggybacking Design Framework for Read- and- Download- efficient Distributed Storage Codes K V Rashmi, Nihar B Shah, Kannan Ramchandran

2 Outline IntroducGon & MoGvaGon Measurements from Facebook s Warehouse cluster The Piggybacking framework Via the Piggybacking framework Best known codes for several senngs Comparison with other codes Preliminary pracgcal experiments Summary & future work

3 Outline IntroducGon & MoGvaGon Measurements from Facebook s Warehouse cluster The Piggybacking framework Via the Piggybacking framework Best known codes for several senngs Comparison with other codes Preliminary pracgcal experiments Summary & future work

4 Redundancy: replicagon, erasure codes Redundancy for reliability & availability ReplicaGon: expensive for large scale data Erasure codes: storage- efficient a a a b b a+b b Replication a+2b (4, 2) Reed-Solomon code

5 However RS codes increase disk IO and download during repair

6 Repair: increased disk IO & download Replication RS code a a b a a IO & Download 1x a b a+b b a+b a IO & Download 2x b a+2b Typical RS repair: IO & download = size of message

7 [Rashmi et al, USENIX HotStorage 2013]! A solution to the network challenges of data recovery in erasure-coded distributed storage systems: A study on the Facebook warehouse cluster! MoGvaGon: Facebook s Warehouse Cluster Measurements MulGple tens of PBs and growing MulGple thousands of nodes

8 [Rashmi et al, USENIX HotStorage 2013]! A solution to the network challenges of data recovery in erasure-coded distributed storage systems: A study on the Facebook warehouse cluster! MoGvaGon: Facebook s Warehouse Cluster Measurements MulGple tens of PBs and growing MulGple thousands of nodes Reducing storage requirements is of high importance Uses (14, 10) RS code for storage efficiency on less- frequently accessed data MulGple PBs of RS coded data

9 Breakdown of repairs # repairs % of repairs x x 10-9 Dominant scenario: Single repairs Rashmi et al, USENIX HotStorage 2013!

10 Amount of transfer Median of 180 TB transferred across racks per day for repair Around 5 Gmes that under 3x replicagon Rashmi et al, USENIX HotStorage 2013!

11 Outline IntroducGon & MoGvaGon Measurements from Facebook s Warehouse cluster The Piggybacking framework Via the Piggybacking framework Best known codes for several senngs Comparison with other codes Preliminary pracgcal experiments Summary & future work

12 Piggybacking RS codes: Toy Example Step 1: Take 2 stripes of (4, 2) Reed- Solomon code systemagc 1 a 1 b 1 systemagc 2 a 2 b 2 parity 1 parity 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2

13 Piggybacking RS codes: Toy Example Step 2: Add piggybacks a 1 b 1 a 2 b 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2 +a 1 No addigonal storage!

14 Fault- Tolerance Same fault tolerance as RS code: can tolerate any 2 failures a 1 b 1 a 2 b 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2 +a 1

15 Fault- Tolerance Same fault tolerance as RS code: can tolerate failure of any 2 nodes a 1 b 1 a 2 b 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2 +a 1 a 1 a 2

16 Fault- Tolerance Same fault tolerance as RS code: can tolerate failure of any 2 nodes a 1 b 1 a 2 b 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2 +a 1 subtract a 1 a 2

17 Fault- Tolerance Same fault tolerance as RS code: can tolerate failure of any 2 nodes a 1 b 1 a 2 b 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2 +a 1 a 1 a 2 b 1 b 2

18 Repair a 1 b 1 a 2 b 2 a 1 +a 2 a 1 +2a 2 b 1 +b 2 b 1 +2b 2 +a 1

19 Repair IO & Download = 3 (instead of 4 as in RS) a 1 a 2 a 1 +a 2 a 1 +2a 2 b 1 b 2 b 1 +b 2 b 1 +2b 2 +a 1 b 2 b 1 +b 2 b 1 +2b 2 +a 1

20 Repair IO & Download = 3 (instead of 4 as in RS) b 2 subtract a 1 a 2 a 1 +a 2 a 1 +2a 2 b 1 b 2 b 1 +b 2 b 1 +2b 2 +a 1 b 1 +b 2 b 1 +2b 2 +a 1

21 Repair IO & Download = 3 (instead of 4 as in RS) a 1 a 2 a 1 +a 2 a 1 +2a 2 b 1 b 2 b 1 +b 2 b 1 +2b 2 +a 1 b 2 b 1 +b 2 b 1 +2b 2 +a 1 subtract

22 General Piggybacking Framework Step 1: Take 2 (or more) stripes of (n, k) code C a 1 a k f 1 (a 1,,a k ) f n-k (a 1,,a k ) b 1 b k f 1 (b 1,,b k ) f n-k (b 1,,b k )

23 General Piggybacking Framework Step 2: Add `Piggybacks a 1 a k f 1 (a 1,,a k ) f n-k (a 1,,a k ) b 1 b k f 1 (b 1,,b k ) + p 1 (a 1,,a k ) f n-k (b 1,,b k ) + p n-k (a 1,,a k )

24 General Piggybacking Framework Decoding: use decoder of C a 1 a k f 1 (a 1,,a k ) f n-k (a 1,,a k ) b 1 b k f 1 (b 1,,b k ) + p 1 (a 1,,a k ) f n-k (b 1,,b k ) + p n-k (a 1,,a k ) recover a 1,,a k as in C subtract piggybacks; recover b 1,,b k as in C

25 General Piggybacking Framework Data- reconstrucgon: peeling decoder using C block 1 block k Piggybacking a does not reduce 1 b 1 minimum distance a k b k Can choose f 1 (arbitrary 1,,a k ) funcgons f 1 (b 1 for,,b k ) piggybacking + p 1 (a 1,,a k ) block k+1 block k+2 f 2 (a 1,,a k ) f 2 (b 1,,b k ) + p 2 (a 1,,a k ) block k+3 block k+r f 3 (a 1,,a k ) f r (a 1,,a k ) f 3 (b 1,,b k ) + p 3 (a 1,,a k ) f r (b 1,,b k ) + p r (a 1,,a k ) recover a 1,,a k as in C subtract piggybacks recover b 1,,b k as in C

26 block 1 block k block k+1 rent block block k+2 lengths & repair- efficiency block k+3 block k+r General Piggybacking Framework a 1 such that they can be used for repair a k f 1 (a 1,,a k ) f 2 (a 1,,a k ) f 3 (a 1,,a k ) f r (a 1,,a k ) - details in the paper Efficient repair Piggybacking funcgons should be designed eg, ropose 3 piggyback funcgon designs a 2 b 1 b k f 1 (b 1,,b k ) + p 1 (a 1,,a k ) f 2 (b 1,,b k ) + p 2 (a 1,,a k ) f 3 (b 1,,b k ) + p 3 (a 1,,a k ) f r (b 1,,b k ) + p r (a 1,,a k ) We propose 3 designs of piggyback funcgons Use the Piggybacking funcgons! a 1 a 1 +a 2 a 1 +2a 2 b 1 b 2 b 1 +b 2 b 1 +2b 2 +a 1

27 Outline IntroducGon & MoGvaGon Measurements from Facebook s Warehouse cluster The Piggybacking framework Via the Piggybacking framework Best known codes for several senngs Comparison with other codes Preliminary pracgcal experiments Summary & future work

28 Via the Piggybacking framework 1 PracGcal High- rate MDS codes: Lowest known IO & download during repair

29 Storage constrained systems: MDS & high- rate Then, why not high- rate Minimum Storage RegeneraGng (MSR) codes? - Require block length exponengal in k (Tamo et al 2011) Block length: number of sub- divisions of data units need high granularity of data low read efficiency block length

30 Comparison with High- rate MSR n k Block length IO & Download (% of message size) RS Piggy- RS MSR RS Piggy- RS MSR

31 Comparison With Other Codes Code MDS Parameters Block length (in k) High- rate MSR Y all exponengal Product- matrix MSR etc Y low rate linear Rotated- RS Y 3 pariges constant EVENODD/RDP Y 2 pariges linear MBR N all linear Local repair N all constant Piggyback Y all constant / linear

32 Comparison With Other Codes Code MDS Parameters Block length (in k) High- rate MSR Y all exponengal Product- matrix MSR etc Y low rate linear Rotated- RS Y 3 pariges constant EVENODD/RDP Y 2 pariges linear MBR N all linear Local repair N all constant Piggyback Y all constant / linear

33 Comparison With Other Codes

34 Via the Piggybacking framework 2 Binary MDS (vector) codes lowest known IO & download during repair for all parameters where binary MDS (vector) codes exist (lowest when #parity 3; En Gad et al ISIT 2013 for #parity=2)

35 Via the Piggybacking framework 3 Enabling parity repair in regeneragng codes designed for only systemagc repair efficiency in systemagc repair retained parity repair improved Example

36 RegeneraGng code that repairs systemagc nodes efficiently Parity node repair performed by downloading all data a c a c b 2a+b+2c d 2b+c+d RC systematic download = 15x a+b+2c a+2b+4d

37 Take two stripes of this code a c a c b d b d 2a+b+2c 2b+c+d 2a +b +2c 2b +c +d a+b+2c a+2b+4d a +b +2c a +2b +4d

38 Add Piggybacks of pariges from first stripe onto second stripe a c a c b d b d 2a+b+2c 2b+c+d 2a +b +2c 2b +c +d + a+b+2c +a+2b+4d a+b+2c a+2b+4d a +b +2c a +2b +4d

39 SystemaGc repair: same efficiency as original code (Piggyback can be subtracted off in the downloaded data) a c a c b d b d 2a+b+2c 2b+c+d 2a +b +2c 2b +c +d + a+b+2c +a+2b+4d RC systematic download = 15x a+b+2c a+2b+4d a +b +2c a +2b +4d

40 Parity repair as in the original code requires 2x download a b c d a c Original RC parity repair download = 2x b d 2a+b+2c 2b+c+d 2a +b +2c 2b +c +d + a+b+2c +a+2b+4d a+b+2c a+2b+4d a +b +2c a +2b +4d

41 Using the Piggybacks, need only 15x download a b 2a+b+2c c d 2b+c+d a c Second parity repair using b d Piggyback: download = 15x 2a +b +2c 2b +c +d + a+b+2c +a+2b+4d a+b+2c a+2b+4d a +b +2c a +2b +4d

42 Via the Piggybacking framework 4 Currently implemengng (14, 10) Piggyback- RS in HDFS 30% reducgon in IO and download same storage & fault tolerance

43 (14,10) Piggybacked- RS code Step 1: Take a (14, 10) Reed- Solomon code a 1 a 10 b 1 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) f 3 (b 1,,b 10 ) f 4 (b 1,,b 10 )

44 (14, 10) Piggybacked- RS code Step 2: Add `Piggybacks a 1 a 10 b 1 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 4 (0,,0,a 4,a 5,a 6,0,,0) f 4 (b 1,,b 10 ) + f 4 (0,,0,a 7,a 8,a 9,0)

45 (14, 10) Piggybacked- RS code Tolerates any 4 block failures a 1 a 10 b 1 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 4 (0,,0,a 4,a 5,a 6,0,,0) f 4 (b 1,,b 10 ) + f 4 (0,,0,a 7,a 8,a 9,0)

46 (14, 10) Piggybacked- RS code Tolerates any 4 block failures a 1 a 10 b 1 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 4 (0,,0,a 4,a 5,a 6,0,,0) f 4 (b 1,,b 10 ) + f 4 (0,,0,a 7,a 8,a 9,0) recover a 1,,a 10 like in RS

47 (14, 10) Piggybacked- RS code Tolerates any 4 block failures a 1 a 10 b 1 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 4 (0,,0,a 4,a 5,a 6,0,,0) f 4 (b 1,,b 10 ) + f 4 (0,,0,a 7,a 8,a 9,0) recover a 1,,a 10 like in RS

48 (14, 10) Piggybacked- RS code Tolerates any 4 block failures a 1 a 10 b 1 b 10 recover a 1,,a 10 like in RS f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 1 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 1 (0,,0,a 4,a 5,a 6,0,,0) subtract piggybacks (funcgons of a 1,,a 10 ) f 4 (b 1,,b 10 ) + f 1 (0,,0,a 7,a 8,a 9,0)

49 (14, 10) Piggybacked- RS code Tolerates any 4 block failures a 1 a 10 b 1 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 1 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 1 (0,,0,a 4,a 5,a 6,0,,0) f 4 (b 1,,b 10 ) + f 1 (0,,0,a 7,a 8,a 9,0) recover a 1,,a 10 like in RS subtract piggybacks (funcgons of a 1,,a 10 ) recover b 1,,b 10 like in RS

50 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 3 (a 1,,a 10 ) f 4 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) f 3 (b 1,,b 10 ) + f 4 (0,,0,a 4,a 5,a 6,0,,0) f 4 (b 1,,b 10 ) + f 4 (0,,0,a 7,a 8,a 9,0)

51 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0)

52 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) recover b 1,,b 10 like in RS

53 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) recover b 1,,b 10 like in RS

54 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) recover b 1,,b 10 like in RS subtract f 2 (b 1,,b 10 )

55 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) recover b 1,,b 10 like in RS subtract f 2 (b 1,,b 10 )

56 (14, 10) Piggybacked- RS code Efficient data- recovery a 1 b 1 a 2 b 2 a 3 b 3 a 10 b 10 f 1 (a 1,,a 10 ) f 2 (a 1,,a 10 ) f 1 (b 1,,b 10 ) f 2 (b 1,,b 10 ) + f 4 (a 1,a 2,a 3,0,,0) recover b 1,,b 10 like in RS subtract f 2 (b 1,,b 10 ) remove effect of a 2 and a 3 to get a 1

57 (14, 10) Piggyback- RS in HDFS 30% reducgon in IO and download on average same storage & fault tolerance However, requires connecgvity > in RS - is this a concern?

58 Is connecgng to more nodes a concern? We performed measurements for various data- sizes in the Facebook Warehouse cluster in producgon Piggyback- RS codes: Reduce primary metrics of IO & download Time to repair also reduces upon connecgng to more Locality/ConnecGvity NOT an issue in this senng

59 Outline IntroducGon & MoGvaGon Measurements from Facebook s Warehouse cluster The Piggybacking framework Via the Piggybacking framework Best known codes for several senngs Comparison with other codes Preliminary pracgcal experiments Summary & future work

60 Summary Piggybacking code design framework 3 piggyback funcgon designs Best known codes for several senngs MDS + high- rate + small block length binary MDS (vector) parity repair in regeneragng codes

61 Future work & open problems Other Piggybacking designs / applicagons Bounds for Piggybacking approach?

62 Future work & open problems Other Piggybacking designs / applicagons Bounds for Piggybacking approach? High- rate MDS: Tradeoff between block length & IO/download IO and download x RS x x 1 2 (constant) Piggyback x x x x??????? (linear) Block length MSR x (exponengal)

63 Thanks!