Scalable Failure Recovery for Tree-based Overlay Networks

Transcription

1 Scalable Failure Recovery for Tree-based Overlay Networks Dorian C. Arnold University of Wisconsin Paradyn/Condor Week April 30 May 3, 2007 Madison, WI

2 Overview Motivation Address the likely frequent failures in extreme-scale systems State Compensation: Tree-Based Overlay Networks (TBŌNs) failure recovery Use surviving state to compensate for lost state Leverage TBŌN properties Inherent information redundancies Weak data consistency model: convergent recovery Final output stream converges to non-failure case Intermediate output packets may differ Preserves all output information 2007 Dorian C. Arnold 2

3 HPC System Trends 60% larger than 10 3 processors 10 systems larger than 10 4 processors System Location Size Time Frame RoadRunner LANL ~3.2x Jaguar ORNL ~4.2x SunFire x64 TACC ~5.2x Cray XT4 ORNL ~2x BlueGene/P ANL ~5x BlueGene/Q ANL/LLNL ~ Dorian C. Arnold 3

4 Large Scale System Reliability BlueGene/L (~42 days) (~417 days) (~4,167 days) (~41,667 days) (~416,667 days) MTTF (days) 4 3 Purple ,000 10, ,000 1,000,000 10,000, Dorian C. Arnold Node Count 4

5 Current Reliability Approaches Fail-over (hot backup) Replace failed primary w/ backup replica Extremely high overhead: 100% minimum! Rollback recovery Rollback to checkpoint after failure May require dedicated resources and lead to overloaded network/storage resources 2007 Dorian C. Arnold 5

6 Our Approach: State Compensation Leverage inherent TBŌN redundancies Avoid explicit replication No overhead during normal operation Rapid recovery Limited process participation General recovery model Applies to broad classes of computations 2007 Dorian C. Arnold 6

7 Background: TBŌN Model TBŌN Output FE CP 0 Application Front-end Filter State Packet Filter CP CP 1 CP 2 CP Tree of Communication Processes TBŌN Input CP CP CP CP BE BE BE BE BE BE BE BE Application Back-ends 2007 Dorian C. Arnold 7

8 A Trip Down Theory Lane Formal treatment provides confidence in recovery model Reason about semantics before/after recovery Recovery model doesn t change computation Prove algorithmic soundness Understand recovery model characteristics System reliability cannot depend upon intuition and ad-hoc reasoning 2007 Dorian C. Arnold 8

9 Theory Overview TBŌN end-to-end argument: output only depends on state at the end-points channel state filter state CP i Can recover from lost of any internal filter and channel states channel state CP k CP j CP l 2007 Dorian C. Arnold 9

10 Theory Overview (cont d) TBŌN Output Theorem Output depends only on channel states and root filter state channel state CP i All-encompassing Leaf State Theorem State at leaves subsume channel state (all state throughout TBŌN) filter state channel state CP j Result: only need leaf state to recover from root/internal failures CP k CP l 2007 Dorian C. Arnold 10

11 Theory Overview (cont d) TBON Output Theorem All-encompassing Leaf State Theorem Builds on Inherent Redundancy Theorem 2007 Dorian C. Arnold 11

12 Background: Notation fs n (CP i ) cs n,p ( CP i ) CP i fs p (CP j ) cs( CP j ) CP j CP k CP l 2007 Dorian C. Arnold 12

13 Background: Data Aggregation Filter function: f (in n (CP i ); f s n (CP i ))! f out n (CP i ); f s n + 1 (CP i )g Packets from input channels Current filter state Output packet Updated filter state 2007 Dorian C. Arnold 13

14 Background: Filter Function Built on state join and t difference operators State join operator, Update current state by merging inputs in n (CP i ) t f s n (CP i )! f s n + 1 (CP i ) Commutative: Associative: Idempotent: a t b = bt a (a t b) t c = a t (bt c) a t a = a 2007 Dorian C. Arnold 14

15 Background: Descendant Notation CP i desc 0 (CP i ) CP CP desc 1 (CP i ) CP CP desc k-1 (CP i ) CP CP CP CP desc k (CP i ) fs( desc k (CP i ) ): join of filter states of specified processes cs( desc k (CP i ) ): join of channel states of specified processes 2007 Dorian C. Arnold 15

16 TBŌN Properties: Inherent Redundancy Theorem The join of a CP s filter state with its pending channel state equals the join of the CP s children s filter states. t t t fs n (CP 0 ) cs n,p (CP 0 ) cs n,q (CP 0 ) = fs p (CP 1 ) fs q (CP 2 ) CP 0 CP 1 CP Dorian C. Arnold 16

17 TBŌN Properties: Inherent Redundancy Theorem The join of a CP s filter state with its pending channel state equals the join of the CP s children s filter states. fs( desc 0 (CP 0 )) t cs( desc 0 (CP 0 )) = fs( desc 1 (CP 0 )) CP 0 CP 1 CP Dorian C. Arnold 17

18 TBŌN Properties: All-encompassing Leaf State Theorem The join of the states from a sub-tree s leaves equals the join of the states at the sub-tree s root and all in-flight data CP 0 CP 1 CP 2 CP 3 CP 4 CP 5 CP Dorian C. Arnold 18

19 TBŌN Properties: All-encompassing Leaf State Theorem The join of the states from a sub-tree s leaves equals the join of the states at the sub-tree s root and all in-flight data From Inherent Redundancy Theorem: f s(desc1(cp 0 )) = f s(desc0(cp 0 )) t cs(desc0(cp 0 )) f s(desc2(cp 0 )) = f s(desc1(cp 0 )) t cs(desc1(cp 0 )) f s(desck (CP 0 )) = f s(desck 1(CP 0 )) t cs(desck 1(CP 0 )) f s(desck (CP 0 )) = f s(cp 0 ) t cs(desc0(cp 0 )) t : : : t cs(desck 1(CP 0 )) 2007 Dorian C. Arnold 19

20 State Composition Motivated by previous theory State at leaves of a sub-tree subsume state throughout the higher levels Compose state below failure zones to compensate for lost state Addresses root and internal failures State decomposition for leaf failures Generate child state from parent and sibling s 2007 Dorian C. Arnold 20

21 If CP j fails, all state associated with CP j is lost State Composition TBŌN Output Theorem: Output depends only on channel states and root filter state All-encompassing Leaf State Theorem: State at leaves subsume channel state (all state throughout TBŌN) fs(cp j ) cs( CP i, m) cs( CP j ) CP j CP i Therefore, leaf states can replace lost channel state without changing computation s semantics CP k CP l 2007 Dorian C. Arnold 21

22 State Composition Algorithm if detect child failure remove failed child from input list resume filtering from non-failed children endif if detect parent failure do determine/connect to new parent while failure to connect propagate filter state to new parent endif 2007 Dorian C. Arnold 22

23 Summary: Theory can be Good! Allows us to make recovery guarantees Yields sensible, understandable results Better-informed implementation What needs to be implemented What does not need to be implemented 2007 Dorian C. Arnold 23

24 References Arnold and Miller, State Compensation: A Scalable Failure Recovery Model for Tree-based Overlay Networks, UW-CS Technical Report, February Other papers and software: Dorian C. Arnold 24

25 Bonus Slides! 2007 Dorian C. Arnold 25

26 State Composition Performance r ecover y latency = (connecti on establi shment max adoptees)+ output over head LAN connection establishment: ~1 millisecond 2007 Dorian C. Arnold 26

27 Failure Model Fail-stop Multiple, simultaneous failures Failure zones: regions of contiguous failure Application process failures May view as sequential data sources/sink Amenable to basic reliability mechanisms Simple restart, sequential checkpointing 2007 Dorian C. Arnold 27