On the Quality of Service of Failure Detectors. Sam Toueg Wei Chen, Marcos K. Aguilera (part of Wei ChenÕs PhD Thesis)

Transcription

1 On the Quality of Service of Failure etectors Sam oueg Wei Chen, Marcos K. Aguilera part of Wei ChenÕs Ph hesis

2 Abstract We study the quality of service QoS of failure detectors. By QoS, we mean a specification that quantifies a how fast the failure detector detects actual failures, and b how well it avoids false detections. We first propose a set of QoS metrics to specify failure detectors for systems with probabilistic behaviors,i.e., for systems where message delays and message losses follow some probability distributions. We then give a new failure detector algorithm and analyze its QoS in terms of the proposed metrics. We show that, among a large class of failure detectors, the new algorithm is optimal with respect to some of these QoS metrics. Given a set of failure detector QoS requirements, we show how to compute the parameters of our algorithm so that it satisfies these requirements, and we show how this can be done even if the probabilistic behavior of the system is not known. We then present some simulation results that show that the new failure detector algorithm provides a better QoS than an algorithm that is commonly used in practice. Finally, we briefly explain how to make our failure detector adaptive, so that it automatically reconfigures itself when there is a change in the probabilistic behavior of the network. Full paper available at:

3 What is the QoS of Failure etectors A specification that quantifies Ð speed: how fast an F detects a crash Ð accuracy: how well an F avoids erroneous detections

4 Our Results Metrics for the QoS specification esign and analysis of a new failure detector algorithm Ð An optimality result Ð he analysis on the QoS metrics Ð Satisfying the QoS requirements given by applications ealing with unknown system behavior and unsynchronized clocks Simulation results

5 About Failure etectors F trust trust trust suspect suspect suspect Process p up down

6 Part I On the QoS Specification of Failure etectors

7 Our Results on QoS Metrics Identify 7 QoS metrics Quantify the relations between these metrics Propose 3 metrics as the primary metrics for the QoS specification

8 Metric 1: etection ime Measure how fast a failure detector detects a process crash. F trust Process p up suspect down

9 What should be the accuracy metrics? Accuracy: how well an F avoids erroneous suspicions For accuracy metrics, consider runs in which process p does not crash etermining a good set of accuracy metrics is not that simple

10 he First Accuracy Metric: Query Accuracy Probability P A Application random query S F Process p up P A is the probability that the failure detector output is correct at a random time.

11 Query Accuracy Probability is not Sufficient F1 F2 Process p up

12 Another Accuracy Metric: Average Mistake Rate λ M λ M measures the rate at which a failure detector makes mistakes. λ M itself is not sufficient either. F1 F2 Process p up

13 P A and λ M together are not sufficient F1 is better than F2 in both P A and λ M, but F2 is faster in correcting each mistakes F1 F2 Process p up

14 More Accuracy Metrics Mistake uration M Good Period uration G Mistake Recurrence ime MR F M G MR

15 Forward Good Period uration FG More relevant to short-lived applications monitor request Application F FG interrupt Process p up

16 Relations among Accuracy Metrics + = = + = > = Pr 1 Pr G G G G G FG k G k G k FG x G G FG E V E E E E E k E E dy y E x MR G A E P = E 1 MR M E = λ MR M G = +

17 Primary Metrics etection time F Mistake recurrence time MR Mistake duration M F Process p Process p M MR

18 Part II he esign and Analysis of the New Failure etector Algorithm

19 he Probabilistic Network Model message loss probability p L message delay time we first assume clocks are synchronized; then only that clock drifts are negligible

20 A Simple F Algorithm Process p η η η Process q O O O F at q iming out depends on two consecutive messages

21 Large etection ime epends on the delay of the last message sent by p Process p Process q crash O O O F at q max + O

22 New Algorithm w/ synchronized clocks η Process p m i-1 m i m i+1 m i+2 Process q Freshness points: τ i-1 F at q δ η τ i τ i+1 τ i+2 At any time t [τ i,τ i+1, F trusts p iff q has received heartbeat message m i or higher.

23 he Core Property At any time t [τ i,τ i+1, F trusts p iff q has received heartbeat message m i or higher. Process p Process q m i-1 m i m i+1 m i+2 t t t F at q Freshness points: τ i τ i+1 τ i+2

24 etection ime Process p Process q F at q m i crash δ η τ i τ i+1 δ +η

25 he QoS Analysis of the New Failure etector Algorithm

26 An Optimality Result Among all F algorithms such that the monitored process p sends a message every η, the detection time is always less than a given bound, our new algorithm provides the best query accuracy probability.

27 Summary of the QoS Analysis P A where 1 η = 1 dx u x p S u x η 0 = j = 0 = 1 p Pr < δ + η u0 By decreasing η or increasing δ linearly: Ð P A increases exponentially towards 1 L E = η p MR S δ η [ p + 1 p Pr > δ + x jη ] L Ð E MR increases exponentially we also derived E M, E G, λ M from P A and E MR L

28 Satisfying QoS Requirements Given a set of QoS requirements as a tuple such that Find η and δ to achieve these requirements U M M L MR MR U E E,, U M L MR U

29 Suppose p L and the distribution Pr t is known Problem to solve: such that max η δ + η η p S η 0 S L MR U u x dx p U M

30 Configuration Procedure Step 1: computeq 0 = 1 p Pr < and let U η max = q 0 M U L Step 2: let f η = η U η 1 U q 0 [ pl + 1 pl Pr > jη] j= 1 find the largest η η max that satisfies Step 3: set δ η = U L f η MR

31 Heartbeat Probabilistic Behavior QoS Requirements U L,, MR U M P L Configurator η P[ x] δ Failure etector

32 Part III ealing with: Unknown System Behavior Unsynchronized Clocks

33 Unknown System Behavior Message loss probability p L is not known Message delay distribution Pr x is not known Clocks are synchronized Need to modify the configuration procedure to satisfy the given QoS requirements

34 Main Idea Bound Pr x using E and V Modify configuration procedure to use E and V instead of Pr x Estimate E, V and p L using heartbeats Use estimates to run configuration procedure

35 Estimator of Heartbeat Probabilistic Behavior QoS Requirements U L,, MR U M P L Configurator η E δ V Failure etector

36 etails 1. Bound Pr > t using the one-sided inequality: For all t > E, Pr > t V V + t E 2

37 γ η β η M MR E E where ,, 0 η δ η δ γ η δ η δ η δ β = = + + = = E V E p E k j E V j E p V L k j L etails contõd 2. Bound the QoS accuracy metrics:

38 etails contõd Step 3: set η δ = U Step 1: compute and let, min max E U U M = γ η E V E p U U L + = γ Step 2: let find the largest η η max that satisfies L f MR η = + + = η η η η η E j U L U U j E p V j E V f 3. Obtain the following configuration procedure:

39 Estimate p L, E and V Estimating p L : using the sequence numbers associated with heartbeat messages Estimating E and V: Ð p timestamps the heartbeats using the sending time S i Ð q records the receipt times A i of heartbeats Ð taking the average and the variance of A i -S i

40 When Clocks are Not Sychronized Problem: Freshness points cannot be set as shifts of the sending times of heartbeats Solution: Ð shift the freshness points with respect to expected arrival times EA i Õs of heartbeats Ð estimate the expected arrival times

41 Algorithm with non-synch clocks Process p m i m i+1 m i+2 Process q E EA i δ α τ i τ i+1 τ i+2 EA i is the expected arrival time of m i. Parameter α accounts for the variation of message delay. δ = E +α

42 QoS analysis of new algorithm remains the same with δ replaced by E+α Given some QoS requirements, we can compute F parameters δ and α using only p L and V

43 QoS Requirements U L,, MR U M Estimator of Heartbeat Probabilistic Behavior P L V Configurator η α EA i Õs Failure etector

44 Estimating Expected Arrival imes Using n most recently received heartbeat messages With appropriate n, the estimates can be very accurate

45 Estimating Expected Arrival imes Process q m 1 m 2 m 3... m n arrival times: A 1 A 2 A... 3 A n EA n+1? known A naive idea: compute the average of interarrival time, and add it to A n to get EA n+1 his does not work: it depends too much on A n

46 EA he Estimator n 1 1 Ai iη + + 1η n n+ n i= 1 ÒnormalizeÓ each A i by shifting it backward in time by iη compute the normalized A i Õs shift forward the computed average by n+1η

47 Spectrum of Algorithms Simple algorithm New algorithm with known expected arrival times 1 n = number of messages used to estimate the expected arrival times

48 Part IV Simulation Results

49 Algorithms We Simulated New algorithm with synchronized clocks Ð matches the analytical results New algorithm with unsynchronized clocks Ð matches the one with synchronized clocks Simple algorithm Ð he new algorithm provides better QoS than the simple algorithm

50 How to Compare Send heartbeat messages at the same rate Satisfy the same bound on the worst-case detection time Compare the average time between mistakes E MR

51 Simulation Settings intersending time η = 1 message loss probability p L = 0.01 message delay has exponential distribution E = σ = 0.02

52 Comparing E MR

53 Comparing E MR

54 Summary Proposed a set of QoS metrics and quantified the relation between them Presented a new failure detector algorithm and analyzed its QoS Showed how to compute the parameters of the new algorithm to satisfy some given QoS requirements Showed how to use the algorithm when: a the system behavior is not known, and b clocks are not synchronized Presented simulation results showing that the new algorithm provides a better QoS than a simple algorithm

55 Related Work Vogels [1996] Gouda and McGuire [1998] Van Renesse, Minsky and Hayden [1998] Raynal and ronel [1999] Ver ssimo and Raynal [2000]