Integrating External and Internal Clock Synchronization. Christof Fetzer and Flaviu Cristian. Department of Computer Science & Engineering

Transcription

1 Integrating External and Internal Clock Synchronization Christof Fetzer and Flaviu Cristian Department of Computer Science & Engineering University of California, San Diego La Jolla, CA 9093? fcfetzer, aviug June 4, 1996 Abstract We address the problem of how to integrate faulttolerant external and internal clock synchronization. In this paper we propose a new external/internal clock synchronization algorithm which provides both external and internal clock synchronization for as long as a majority of the reference time servers (servers with access to reference time) stay correct. When half or more of the reference time servers are faulty, the algorithm degrades to a fault-tolerant internal clock synchronization algorithm. We prove that at least F +1 reference time servers are necessary for achieving external clock synchronization when up to F reference time servers can suer arbitrary failures, thus the proposed algorithm provides maximum fault-tolerance. In this paper we also derive lower bounds for the best maximum external deviation achievable in standard mode and the best drift rate achievable in degraded mode. Our algorithm is optimal with respect to these two bounds: (1) the maximum external deviation is optimal in standard mode, and () the drift rate of the clocks is optimal in standard and degraded mode. 1 Introduction Synchronized clocks are crucial for many faulttolerant real-time systems. We assume the existence of a global time base called real-time. Reference time To appear in the Journal of Real-Time Systems, An earlier version of this paper was published in the Proceedings of the 15th International Conference on Distributed Systems, May, This work was partially supported by grants from the Air Force Oce of Scientic Research, the German Academic Exchange Service (DAAD), the Powell Foundation, Sun Microsystems, and the Microelectronics Innovation and Computer Research Opportunities in California. is a granular representation of real-time and is typically provided by a standard source of time such as NIST. Clocks can be externally or internally synchronized [1]. A clock is externally synchronized if at any point in real-time the distance between its value and reference time is bounded by an a priori given constant called maximum external deviation. A set of clocks is internally synchronized if at any point in real-time the distance between the values of two correct clocks in the set is bounded by an a priori given constant called the maximum internal deviation and each clock runs within a linear envelope of real time. Externally synchronized clocks are always internally synchronized by two times the maximum external deviation, but internally synchronized clocks are not always externally synchronized. The systems we consider in this paper consist of a set of time servers which provide time information to local clients. We assume that each time server has access to a local hardware clock. Hardware clocks can be viewed as a counter which is incremented by the ticks of an oscillator. Because of imprecisions of the oscillator and temperature changes, hardware clocks drift apart from reference time. The rate of this drift is typically very small. However, even when the hardware clocks would be initially synchronized with reference time, eventually the distance between a hardware clock and reference time can be greater than any a priori given constant. To achieve external synchronization, at least one time server must have access to reference time. Such servers can use GPS or radio receivers to receive the time signal broadcasts by a standard source of time. A time server with a local access to reference time is called a reference time server, the other servers are non-reference time servers. The failure model assumed in this paper allows that 1

2 time servers suer arbitrary failures. To illustrate the eects of arbitrary failures, let us assume that the correct non-reference time servers have to be externally synchronized within, say, one minute and internally synchronized within minutes. A reference time server r which has suered an arbitrary failure could tell two non-reference time servers p and q at reference time 13:00 that the time is 1:30 and 14:0, respectively. If p and q would just set their clocks to their own approximation of r's clock (i.e. 1:30 and 14:0, respectively), they would be neither externally nor internally synchronized. In general, arbitrary failures of reference time servers can not only disturb the external synchronization of non-reference time servers, they can also perturb their internal synchronization. Since the cost of the hardware components needed to access external reference time signals is in general not negligible, one typically tries to minimize the number of reference time servers needed in a given system. Our rst goal is therefore to investigate the issue of how many reference time servers are necessary to provide external synchronization of non-reference time servers when up to F reference time servers can suer arbitrary failures. Note that the hardware clocks of non-reference time servers can be used to detect some failures of reference time servers. For example, if p reads the externally synchronized clock of reference time server r when p's hardware clock shows 1:00 and 13:00 and p's approximations are 1:00 and 1:30, respectively, then p knows that r is faulty (under the assumption that p itself is correct) and can reject r's clock value. When all failures of reference time servers would be detectable, one obviously needs only F + 1 reference time servers to mask F detectable reference time server failures. However, since hardware clocks can drift apart from real-time, not all failures of reference time servers are detectable. We prove that even when processes use their hardware clocks to detect failures of reference time servers, to provide external synchronization when up to F reference time servers can suer arbitrary failures, a total of F +1 reference time servers is necessary and sucient. A related question that we investigate is how closely can one bound the drift rate of the clocks of nonreference time servers when half or more of the reference time servers can suer arbitrary failures. When we require that non-reference time servers have to be externally synchronized as long as a majority of the reference time servers stay correct and the maximum drift rate of hardware clocks is, we can show the following result: the best maximum drift rate of nonreference time servers achievable when half or more of the reference time servers have suered arbitrary failures is. In most fault-tolerant real-time systems, we are aware of, internal synchronization is required for the correct operation of the system, while external synchronization is mainly used to give users convenient reference time information. For example, in the Advanced Automation System, the correctness of automated recovery from component failures depends on correct internal clock synchronization [] but the system can continue to function in the absence of external clock synchronization in a degraded way. Our second goal is to integrate external and internal clock synchronization: we require that an external/internal clock synchronization algorithm maintains externally and internally synchronized clocks for as long as at least a majority of reference time servers remain correct, and degrades to internal clock synchronization when half or more of the reference time servers become faulty. The two main motivations for integrating internal and external clock synchronization are as follows: 1. The correctness of some applications must be guaranteed independently of the correctness or availability of reference time signals. For example, when the correctness of an application only depends on internal clock synchronization, external/internal clock synchronization guarantees the application's correctness even when there are no radio time signals or these signals are produced/forged by an adversary.. Reduction of cost by minimizing the number of reference time servers. For example, a practical system could contain only one reference time server. This server would be a single point of failure which could disturb the correctness of the complete system. An integrated external/internal clock synchronization algorithm will guarantee that the clocks of correct non-reference time servers will stay externally synchronized for as long as the reference time server stays correct, and will be internally synchronized even when the reference time server fails. An external/internal time service is in one of the following modes: standard, degraded, or integration mode. We introduce these modes to specify the requirements and properties of the time service. However, only a global observer can correctly determine

3 the current mode of a time service because this requires to decide how many reference time servers are currently faulty. Time servers cannot always correctly do this decision (see Section 5.) and thus, cannot always correctly determine the current mode. We assume that the system starts in standard mode: (1) all clocks of correct non-reference time servers are externally and internally synchronized, and () a majority of the reference time servers are correct. In case half or more of the reference time servers become faulty, the service transits to degraded mode. When failed reference time servers are repaired and the correct reference time servers form a majority again, the service is in integration mode. During integration mode the external synchronization of correct non-reference time servers is reestablished in case it was lost during degraded mode. The service transits to standard mode, when the external synchronization is reestablished. In this paper we also address the question of how close an external clock synchronization algorithm can synchronize a non-reference time server to reference time. We show that the best maximum external deviation achievable is the sum of the following terms: the maximum error made by a correct reference time server in approximating reference time, the maximum error reading a remote clock, and an additional error term which takes care about the drift of the hardware clock of a correct nonreference time server between two successive adjustments of its externally synchronized clock. The optimal maximum external deviation is of particular interest because external clock synchronization algorithms typically bound the maximum internal deviation by twice the maximum external deviation. When connecting a reference time signal receiver via a standard interface (like a serial interface) to avoid additional hardware and operating system support, the external deviation can become quite signicant. For example, in our Dependable Systems Laboratory, a reference time server can make an error of more than 10 ms approximating reference time because the used GPS receiver guarantees no better approximation when accessed via its serial interface. Using a simple external clock synchronization algorithm would mean that the clocks of non-reference time servers can be up to 0 ms apart. Our third goal is therefore to show how one can bound the maximum internal deviation independently of the maximum external deviation. The proposed external/internal clock synchronization algorithm achieves a tight internal synchronization even when the reference time servers are only loosely synchronized to reference time. Other properties of the proposed algorithm are as follows: in standard mode the maximum external deviation and the maximum drift rate of the clocks is optimal, and in degraded mode the maximum drift rate of the clocks is optimal (about twice the maximum drift rate of hardware clocks). Related Work Integration of external and internal clock synchronization has recently been addressed by several authors [11, 1, 5, 9, 15]. The paper of Schmid [11, 1] addresses the integration of external and internal clock synchronization in the context of real-time systems. Kopetz et al. [9] addressed the problem of how to integrate external and internal clock synchronization in the context of synchronizing multicluster real-time systems. Their paper only considers one reference time server. [15] proposes an external synchronization scheme that degrades to internal synchronization in case reference time servers become unavailable. The scheme switches again to external synchronization as soon as the reference time servers become available again. This approach does not t completely with our requirements for external/internal clock synchronization: the switch from internal to external clock synchronization can violate our requirement that the drift rates of externally/internally synchronized clocks are always bounded by a small constant (see Section 4.3). 3 System Model We consider distributed systems consisting of computer nodes connected by a communication network. The communication service allows the processes that run on the nodes to communicate with each other by exchanging messages. Each node runs either a reference time server or a non-reference time server process (see Figure 1). The reference time servers have local access to reference time, i.e. they can approximate reference time without communicating with other nodes. The goal of reference time servers is to provide access to reference time for the non-reference time servers: each reference time server maintains a \reference clock" (Section 3.) which can be read by 3

4 non-reference time servers. Each time server has a local hardware clock (Section 3.3). Non-reference time servers have no local mean to approximate reference time and have to read remote reference clocks to do that. The goal of non-reference time servers is to provide an externally/internally synchronized \virtual clock" (Section 3.4) for local clients. We denote the set of time servers by P, the set of reference time servers by P R, and the set of nonreference time servers by P N. Let N = jpj, N R = jp R j, and N N = jp N j denote the number of time servers, reference time servers, and non-reference time servers, respectively. We assume that each reference and each non-reference time server has a rank which is unique amongst the reference and non-reference time servers, respectively. We denote the rank of a time server by function rank and can formally express these constrains by: 8r P R : rank(r) f0; ::; N R? 1g 8r 1 ; r P R : rank(r 1 ) = rank(r )! r 1 = r 8p P N : rank(p) f0; ::; N N? 1g 8p; q P N : rank(p) = rank(q)! p = q We sometimes use the term process or server to refer to a time server. tr i starts. More precisely, reference time satises the following requirements: 1. the distance between two successive ticks is exactly g r, where constant g r is the granularity of reference time: 8i : to(tr i+1 )? to(tr i ) = g r ;. real-time and reference time are in perfect synchrony: 8i : tr i = to(tr i ); 3. for any i and any point in real-time t such that tr i t < tr i+1 the value of reference time at real-time t is tr i. The granularity g r of reference time is negligible for the systems we consider. In what follows, we will therefore not distinguish between reference time and real-time and sometimes use term \real-time" instead of term\reference time". t 3 reference time GPS-Receiver GPS-Receiver t Reference Server... Reference Server t 1 Non-reference server node... node Non-reference server LAN t 0 t 0 t 1 t t 3 real-time Client Client node Client Client node Figure : Reference time represents real-time by a sequence of ticks (tr i : i f0; 1; ; ::g. Figure 1: Typical architecture of an external/internal time service. 3.1 Reference Time We assume that the granularity of real-time is in- nitesimal, and we denote with RT the set of all realtime values. Let us also assume that with each event e one can associate its point of occurrence in real-time to(e) [10]. Reference time is a granular representation of real-time. It is dened by a sequence of ticks tr i, i f0; 1; ; :::g and tr i RT (see Figure ). Tick tr i represents the point in real-time when reference time 3. Reference Clocks Reference time servers have a local access to reference time which allows them to approximate reference time at any point in time within some a priori given error. To abstract from implementation details of this access, we assume that each reference time server r maintains a reference clock X r which represents r's current approximation of reference time. The reference clock of a correct reference time server is within a constant interval of real-time (see Figure 3). We represent the reference clock of a reference time server r by a function X r from real-time to 4

5 clock-time: X r : RT! CT. Term X r (t) denotes the value of r's reference clock at real-time t. A reference clock X r is correct at time t if the deviation between real-time and X r is bounded by an a priori given constant : jx r (t)? tj : The drift rate of a correct reference clock X r is zero, because for any two points s and u in real-time such that u s the following property holds: (u? s)? X r (u)? X r (s) (u? s) +. Reference clocks can be implemented in various ways. For example, in our Dependable Systems Laboratory two workstations have access to a GPS-receiver via a serial interface and a thread in each reference time server p periodically reads the current reference time provided by the GPS-receiver to determine the dierence D between reference time and p's hardware clock. The current value of p's reference clock is de- ned by the sum of the current value of p's hardware clock plus the last computed dierence value D. In our system the maximum deviation between a reference clock and reference time is more than 10 milliseconds. Implementing reference clocks in the operating system kernel or in hardware could reduce the deviation to a few microseconds. T3 T T 1 T0 clock-time t 0 t 1 t+ t t 3 Xr t- real-time Figure 3: Process r's reference clock X r is within of real-time. 3.3 Hardware Clocks Each time server has access to a local hardware clock. A hardware clock typically consists of an oscillator and a counting register that is incremented by the ticks of the oscillator. A hardware clock has a certain granularity G: a tick of the oscillator increments the value of the hardware clock by G. Correct hardware clocks are therefore monotonically increasing. Because of impressions of the oscillator, temperature changes, and aging, a hardware clock can drift apart from real-time. The drift of a hardware clock is however within a narrow \envelope of real-time" (see Figure 4). We denote with CT the set of clock values, i.e. CT contains all values a hardware clock can display. We represent process p's hardware clock by a function H p from real-time to clock-time: H p : RT! CT. Term H p (t) therefore denotes the value displayed by the counter of p's hardware clock at real-time t. A hardware clock H p is correct at time u if for all times s and t such that s t u, H p measures the passage of time during real-time interval [s; t] with an error of at most (t? s) + G, where is the maximum hardware clock drift rate specied by the clock manufacturer: (1-)(t-s)-G H p (t)-h p (s) (1+)(t-s)+G. For most quartz clocks available in modern computers, the constant is of the order of 10?4 to 10?6. Since is such a small quantity, we ignore terms i for i. For example, we equate (1 + )?1 with (1? ) and (1? )?1 with (1 + ). In what follows, we will assume that the granularity of hardware clocks is negligible. T3 T T 1 T0 clock-time t 0 dt -1 = +ρ dt t 1 t dt -1 = ρ dt t 3 dt -1 = 0 dt H p real-time Figure 4: The drift rate of process p's hardware clock H p is within [-,+]. 3.4 Virtual Clocks Clock synchronization algorithms do in general not synchronize the local hardware clocks of nodes. Instead, they introduce the concept of virtual clocks 5

6 which are maintained by time servers (see Figure 5). Our goal will be to ensure that the virtual clocks of correct non-reference servers are externally/internally synchronized. Clients can read the virtual clock of their local time server to access reference time (standard mode) or internal time (degraded and integration mode). Since we assume that only non-reference time servers have local clients, only non-reference time servers maintain virtual clocks. In what follows we will use the term \correct virtual clock" to denote the virtual clock of a correct non-reference time server. The virtual clock of time server p P N is represented by a mapping C p from real-time to clock-time: C p : RT! CT. Term C p (t) therefore denotes the value of p's virtual clock at real-time t. In the protocols we propose, the value C p (t) is determined by adding to the local hardware clock H p (t) a periodically updated adjustment value. We denote the points in real-time when a process p updates its adjustments value (that is, adjusts its virtual clock) by an innite, strictly increasing sequence (t k p ). The value of p's adjustment value between two successive adjustments that occur at times t k p and tp k+1 is denoted by A k p : for any time t [tk p ; tk+1 p ) the value of p's adjustment value at time t is A k p. We say that process p is in round k at time t i t k p t < t k+1 p. We assume that for any time t t 0 p there exists a k such that t k p t < t k+1 p. Process p's virtual clock at t is dened by, C p (t) = A k p + H p (t). To adjust its clock, a process has to read remote virtual and reference clocks. In particular, in the protocols we propose a process p wants to read the remote virtual clocks with respect to the adjustment values of its current round k. We therefore use the well known concept of round clocks. Process p's round clock Cp k for round k is dened by: 8t : Cp k (t) = A k p + H p (t). Note that round clocks are monotonically increasing because the hardware clocks are monotonically increasing and the adjustment value A k p of round clock Cp k is constant. A virtual clock C p is not necessarily monotonic because decreasing p's adjustment value at some time t k+1 p from A k p to Ak+1 p < A k p decreases the value of C p at time t k+1 p. 3.5 Remote Clock Readings In the proposed clock synchronization protocols each non-reference time server estimates at the end of round k the values displayed by all round clocks of Adjustment Value Virtual Clock + Hardware Clock Figure 5: Example: process p's virtual clock C p is dened as the sum of the current value of p's hardware clock and a periodically determined \adjustment value". round k and the values of the reference clocks. This approximation is performed by a remote clock reading method. The reading method allows each time server p to estimate the value displayed by a remote round clock Cq k or a reference clock X r when p's own current round clock Cp k displays some given time T. Let term Cq k (T; p) denote p's approximation of q's round clock Cq k at the time when p's round clock Cp k shows T. We assume that the remote clock reading error provided by the reading method is bounded by an a priori given constant, to be called the maximum clock reading error. When the remode clock reading method is invoked by process p during round k and p requests to read q's round clock Cq k at a time when p's current round clock Cp k shows T, we require that for any time t such that Cp k (t) = T and p and q are correct at t: jcq k (t)? Cq k (T; p)j : The remote clock reading method can also estimate remote reference clocks. We denote with Xr k (T; p) the approximation of reference clock X r by a nonreference time server p when p's current round clock Cp k displays T. We assume that the reading error for remote reference clocks is bounded by the same constant. When process p invokes the method at time s during its current round k and p requests to read reference clock X r at a time p's current round clock Cp k shows T, we require that for any time t such that Cp k (t) = T and p and r are correct in interval [s; t]: jx r (t)? X k r (T; p)j : The execution of the remote clock reading method takes time. We assume that any remote clock reading performed by a correct process p terminates in at most RD clock-time units: when p invokes the method at real-time s to ap- 6

7 proximate a reference clock X r (or a remote round clock C k q ) for a time when p's round clock Ck p shows T, then the clock reading method returns its approximation of X r (or C k q ) at some time u, and p's hardware clock progressed by at most RD between times s and u: H p (u)?h p (s) RD. Thus, RD(1 + ) real-time units are sucient to read any remote round or reference clock. To be able to bound the clock reading error by a constant, the time T at which the remote clock is approximated has to be constrained: when a process p at real-time u approximates the value that X r (or C k q ) shows at real-time t, the reading error contains a part that is proportional to the distance between t and u because (1) p does not know the the actual drift rates of H p (and H q ), and () H p and X r can drift apart by up to jt? uj between t and u, while (3) H p and H q can drift apart by up to jt? uj between times t and u. In other terms, we have to constrain the distance between t and u to be able to bound the reading error by an a priori given constant. Therefore, we assume that Cp k has not yet shown time T when p invokes the method at time s: Cp k (s) < T, and Cp k shows T in at most RD clock-time units after p has invoked the method: Cp k (s) + RD T. Reading a local hardware, reference, or virtual clock can be done with a much smaller error than reading a remote clock. We therefore assume that a process can read a local clock with a negligible reading error. 3.6 Failure Model We assume that hardware clocks, reference clocks, and processes can experience arbitrary failures. A correct time server has a correct hardware clock, and a correct reference time server has a correct reference clock. For simplicity, we assume that a failed nonreference time server p does not recover: when p is faulty at real-time s, then p is faulty for all times t s. Thus, we do not explain the reintegration of recovered non-reference time servers. However, we address issues related to the repair of reference time servers. Our failure model allows faulty reference time servers to recover. In Section 8 we describe how the external synchronization of correct virtual clocks can be reestablished duing integration mode in case it was lost during degraded mode. In this paper we do not discuss clock reading failures which occur when a correct time server p estimates a remote virtual or reference clock of a correct time server q with an error greater than. Methods for making synchronization algorithms tolerant to remote clock reading failures are proposed in [3]. 4 Requirements In this section we will dene internal, external, and external/internal clock synchronization more precisely. 4.1 Internal Clock Synchronization The rst objective of an internal clock synchronization algorithm is to bound the deviation between any two correct virtual clocks by a constant to be called the maximum internal deviation. Requirement (bounded internal deviation): for any two non-reference time servers p and q that are correct at real-time t: jc p (t)? C q (t)j. The second goal is to bound the drift rate of virtual clocks by a constant v 1. To account for the granularity and the adjustments of virtual clocks, a constant D is introduced. Constant D is called the maximum discontinuity of virtual clocks. Requirement (bounded drift rate): for any non-reference time server p that is correct at time t and for all times s t: (1- v )(t-s)-dc p (t)-c p (s)(1+ v )(t-s)+d: An internal clock synchronization algorithm is correct when it satises both of the above requirements. The two requirements of internal clock synchronization allow that a virtual clock is non-monotonic, i.e. there can exist a correct process p and two points in real-time s and t > s such that C p (s) > C p (t). The clock synchronization protocols we propose will actually show such a non-monotonic behavior for certain runs. However, the protocols can be extended to provide monotonic clock behavior using continuous clock amortization without increasing the maximum internal deviation nor the maximum external deviation [13]. The idea is that any adjustment of a virtual clock can be spread over some time interval, that is, eectively the speed of a virtual clock is temporarily slowed down or increased. Continuous clock amortization does therefore not have to adjust a virtual clock backwards. 7

8 4. External Clock Synchronization A correct external clock synchronization algorithm has to bound the deviation between a correct virtual clock and real-time by an a priori given constant, to be called the maximum external deviation. Requirement (bounded external deviation): for any time t and any non-reference time server p that is correct at t, jc p (t)? tj : Like the drift rate of a reference clock, the drift rate of an externally synchronized clock is zero. 4.3 External/Internal Clock Synchronization External/internal clock synchronization requires that in standard mode all correct virtual clocks are externally and internally synchronized, in degraded and integration mode all correct virtual clocks are internally synchronized, and external synchronization is eventually reestablished during integration mode (unless half or more of the reference time servers become faulty again). To dene the requirements of external/internal clock synchronization more precisely, we rst give a precise denition of the dierent modes of an external/internal time service. The denition of degraded mode is based on the fact that a non-reference time server has to read a majority of reference clocks to be able to bound the external deviation of its virtual clock (see Section 5.). To guarantee that a correct process succeeds to read a majority of correct reference clocks, it is actually not sucient to require that at any point in time there exists a majority of correct reference time servers. To explain this, consider the following. When the remote clock reading method is invoked at some point in time u to read reference clock X r, the method can take up to RD(1+) time units to return its approximation of X r for some point in time t u+rd(1+). The approximation returned is only guaranteed to be within + of real-time t when r is correct for all times between u and t (see Section 3.5). Thus, we have to require that a majority of reference time servers are correct in interval [u; u+rd(1+)] to ensure that a correct process that invokes the remote clock reading method at u (to read all reference clocks) succeeds to read a majority of correct reference clocks with an error of at most. Knowing that, in the following we use expression \that a majority of reference time servers are correct at t" to say \that there exists a majority M of reference time servers such that each r M is correct in interval [t? RD(1 + ); t]". The service is in degraded mode at time t i there exists no majority of reference time servers that are correct at t. The service is in integration mode at t i a majority of reference time servers are correct at t, the service switched from degraded mode to integration mode at some time s < t: the service is in degraded mode at s and in integration mode during interval (s; t), and not all correct virtual clocks are externally synchronized at time t. The service is in standard mode at t i a majority of reference time servers are correct at t, the service switched from degraded mode or integration mode to standard mode at some time s t: the service is in degraded mode or integration mode in some non-empty interval [u; s), in standard mode during interval [s; t), and all correct virtual clocks were externally synchronized by? r max at time s. We assume that an external/internal time service starts its execution at some point s in real-time. For simplicity, we assume that the service is in standard mode at time s and that initially all correct virtual clocks are externally and internally synchronized. A correct external/internal clock synchronization protocol has to satisfy requirements (bounded internal deviation), (bounded drift rate) and the following two requirements: Requirement (bounded external/internal deviation): for any time t such that the service is in standard mode at time t and any non-reference time server p which is correct at t: jc p (t)? tj : We also require that the external clock synchronization is eventually reestablished during integration mode unless the service transits to degraded mode again: Requirement (integration requirement): for any time t such that the service is in integration mode at t, there exists a time u > t such that the system is in standard mode or degraded mode at u. 8

9 5 Redundancy Requirements for External Clock Synchronization In this section we derive a lower and an upper bound for the number of reference time servers needed to provide externally synchronized clocks in case that up to F reference time servers can suer arbitrary failures. Using the lower bound, we derive also a lower bound on the best maximum drift rate of virtual clocks achievable by an external/internal clock synchronization algorithm in degraded mode. In systems without access to reference time and without message authentication, at least 3F +1 time servers are needed to guarantee internal clock synchronization despite F arbitrary time server failures [6]. We show below that F +1 reference time servers are needed and sucient to guarantee external synchronization of non-reference time servers in case that up to F reference time servers can suer arbitrary failures. Since external clock synchronization implies internal clock synchronization, we have to explain why external clock synchronization needs less \redundancy". To do this, let us attempt to apply an external clock synchronization algorithm to synchronize the virtual clocks of non-reference time servers internally: we assign a set R of F +1 non-reference time servers the role of reference servers. The set S of remaining non-reference time servers synchronize their clocks to the servers in R using an external clock synchronization algorithm. Obviously, this scheme only provides internal synchronization of the clocks in S when the clocks in R are themselves internally synchronized. Achieving this internal synchronization of the servers in R has to be provided by another algorithm, i.e. an internal clock synchronization algorithm. However, the clocks in R cannot be internally synchronized because F of the F +1 servers in R can fail [6]. When we replace the servers in R by reference time servers, these servers receive external time signals which allow them to synchronize their clocks externally and therefore also internally. The F +1 lower bound result for external clock synchronization depends on the assumption that reference time servers can suer arbitrary failures. In particular, it depends on the fact that not all arbitrary failures of reference time servers are \detectable". By detectable we mean that whenever a non-reference time server p tries to read a reference clock X r, then p can decide if r is correct or faulty. For example, in synchronous systems crash failures are detectable and in case only crash failures can occur, all reference time server failures are detectable. Only F +1 reference time servers are needed to mask F detectable reference time server failures because a non-reference time server p can achieve external clock synchronization in the following way: 1. p reads at least every, say, r max time units all reference clocks,. p rejects all readings from reference time servers which are detectably faulty, and 3. p synchronizes its clock to one of the remaining, that is, correct reference clock readings. When at most F out of F +1 reference time servers suer detectable failures, p succeeds to read the clock of a correct reference time server r such that p will not reject r's reading and synchronize its clocks to its approximation of r's reference clock. Thus, p's clock is externally synchronized. Our lower bound proof will use the fact that hardware clocks can drift apart from real-time and that this drift permits faulty reference time servers to drift apart from real-time (within certain bounds) without being detectable by non-reference time servers. This proof allows us to show that in degraded mode the maximum drift rate of correct virtual clocks is at least twice the maximum hardware clock drift rate. The upper bound proof is based on a simple external clock synchronization protocol which achieves external synchronization as long as a majority of reference time servers stay correct. 5.1 Maximum Replication We propose an external clock synchronization algorithm ECS to show that F +1 reference time servers are sucient to guarantee external clock synchronization under the assumption that up to F reference time servers can suer arbitrary failures. Our second goal is to show an upper bound for the maximum external deviation achievable by an external clock synchronization algorithm. Let us assume that any correct virtual clock C p is adjusted at least every r max time units. Algorithm ECS achieves a maximum external deviation of + + r max, where accounts for the maximum external deviation of reference clocks, for the maximum error p makes in reading correct reference clocks, and r max for the maximum drift of p's hardware clock between two adjustments. The intuition behind ECS is as follows (see Figure 6). Each correct non-reference time server p reads at 9

10 incorrect readings X[0] within +Λ of real-time +Λ t +Λ incorrect readings... X[l]... X[F]... X[o]... X[N R -1] F+1 > F Figure 6: The F +1 smallest and F +1 greatest readings of reference clocks contain the reading of at least one correct reference clock (denoted by X[l] and X[o], respectively). The F +1th smallest clock reading (X[F ]) is therefore within + of real-time. least every, say, r max time units all reference clocks. Server p sorts these readings in increasing order and stores them in an array X[0::N R -1]. We assume that at most F reference time servers are faulty and that there are at least F + 1 correct reference time servers, i.e. N R F + 1. Therefore, amongst the F +1 smallest readings there exists at least one reading that belongs to a correct reference clock. In particular, there exists an index l such that 0 l F and entry X[l] belongs to a correct reference clock. Similarly, we can identify a reading of a correct reference clock amongst the F +1 greatest clock readings and therefore nd an index o such that N R? F? 1 o N R? 1 and entry X[o] belongs to a correct reference clock. Since we assume that N R F + 1, entry X[F ] is \between" X[l] and X[o], i.e. l F o and because array X is sorted, X[l] X[F ] X[o]. Because X[l] belongs to a correct reference clock, X[l] is at most + smaller than t (see Figure 6). Similarly, because X[o] belongs to a correct reference clock, X[o] is at most + greater than t. Since entry X[F ] is between X[l] and X[o], it is at most + apart from real-time. A pseudo-code description of algorithm ECS is given in Figure 7. This algorithm is executed by all non-reference time servers. We describe the execution of algorithm ECS by a correct non-reference time server p. Algorithm ECS has to externally synchronize p's virtual clock C p which is represented by function C (line ). The external function H() returns the current value of p's hardware clock. The value of clock C p at time t is the sum of p's hardware clock at t and the value of p's adjustment variable A at t. Process p adjusts its virtual clock in an endless loop (lines 4-8): 1. p reads all reference clocks (line 6) for some point (1) variable Time A = 0, X[0::N R -1], T; () function Time C() f return H() + A; g (3) function void synchronizer () f (4) forever f (5) T = C() + RD; (6) parallel f 8q P R : X[rank(q)] = X q (T ) g (7) sort(x); (8) A = A + (X[F ]? T ) g g Figure 7: External clock synchronization algorithm. t in real-time when its current round clock will show T (line 5),. p sorts the clock readings in increasing order and stores them in array X (line 7), and 3. p adjusts its virtual clock by the dierence between the value p's new round clock should show at t (which is X[F ]) and the value p's current round clock shows at t (which is T ). The approximation of a reference clock q is performed by the external function X p. We assume that all execution times except that of the remote clock reading method (which can take up to RD clocktime units) are negligible. The execution time of the loop body (lines 5-8) is therefore at most RD clocktime units because the approximation of the reference clocks is done in parallel. Moreover, p's clock C p shows a value of at most T when starting the execution of line (8). The adjustment value A is updated at least every r max = RD(1 + ) real-time units. The maximum external deviation of virtual clock C p is therefore at most = ++r max because p's hardware clock drifts apart from real-time by at most r max between two adjustments. Theorem 1 For any time t and any non-reference time server p that is correct at t and executes algorithm ECS: jc p (t)? tj. Proof To analyze algorithm ECS, we describe the execution of ECS by a correct non-reference time server p by ve innite sequences: sequence (s k p) consists of the points in real-time when p adjusts its clock, i.e. the points in realtime when p updates adjustment variable A in line (8), 10

11 sequence (A k p): entry A k p denotes the new content of variable A at time s k p, i.e. the content of variable A is changed at time s k p from A k?1 p to A k p, sequence (Tp k ): entry Tp k?1 variable T at time s k p, denotes the content of sequence (t k p) consists of the points in real-time for which p approximates the reference clocks, and sequence (Xp k ): entry Xp k denotes the content of array X at time s k p. For any round k > 0 and any time t such that s k p t < s k+1 p, p's virtual clock is dened by C p (t) = A k p + H p(t). Process p approximates the reference clocks at the end of round k (i.e. at time tp k+1 ) with respect to its adjustment value A k p of round k. The relation between Tp k and tk+1 p is therefore, A k p + H p (tp k+1 ) = Tp k. Reading a remote clock takes at most RD clocktime units. Since we neglect all other execution times, for any round k the distance s k+1 p? s k p is bounded by, 0 < s k+1 p? s k p RD(1 + ) = r max : By denition of T (line 5), the maximum distance between s k p and t k p is bounded by, 0 t k p? sk p r max: The error p makes reading a correct reference clock is bounded by : for any reference time server r such that r is correct at t k p, when entry i of the sorted array Xp k contains p's approximation of X r for time t k p, then jxp k [i]? X r (t k p)j. Since at most F reference time servers can suer failures, the array Xp k has at most F entries that belong to faulty reference servers. Since each of the two sub-arrays X[0 :: F ] and X[F :: N R? 1] has at least F + 1 entries (N R F + 1), there exist two indices l f0; :::; Fg and o ff; ::; N R? 1g such that the entries Xp k [l] and Xp k [o] contain the approximations of two correct reference time servers. Because a correct reference clock is at most apart from real-time and a correct time server succeeds to read the reference clock of a correct reference time server with an error of at most, the following properties are valid: jxp k [l]? t k pj +, jxp k [o]? t k pj +. These two properties imply that t k p?? Xp k [l], Xp k [o] t k p + +. Since the entries of array X at time t k p are sorted in increasing order and l F o: Xp k [l] Xp k [F ] Xp k [o]. Thus, by transitivity: t k p?? Xp k [F ] t k p + +. This is equivalently to: jxp k [F ]? t k pj +. By denition of A (line 8), the following property holds: A k p = A k?1 p + (Xp k [F ]? Tp k?1 ). Using the fact that Tp k?1 = A k?1 p + H p (t k p) (by denition of C p ), we get A k p = A k?1 p + (Xp k [F ]? A k?1 p? H p (t k p)). By simplication, A k p = Xp k [F ]? H p (t k p). By inserting that property in the denition of C p, C p (t) = (Xp k[f ]? H p(t k p )) + H p(t). Since the drift rate of p's hardware clock is bounded by, and t is bounded by s k p t < s k+1 p and jt? t k pj r max, we can derive: jc p (t)? tj = j(x k p [F ]? H p (t k p) + H p (t))? tj = j(x k p [F ]-t k p) + (H p (t)-h p (t k p))? (t-t k p)j + + jt? t k p j + + r max =. 5. Minimum Redundancy We will show in the lower bound theorem that at least F +1 reference time servers are needed to mask up to F arbitrary failures of reference time servers. Since we are interested in external synchronization of non-reference time servers, we assume that there exists at least one non-reference time server, i.e. P N 6= ;. The intuition behind this theorem is that a nonreference time server p cannot always correctly decide which reference time servers are correct and which are faulty. In particular, when a faulty reference time server r drifts apart from real-time by at most, p cannot necessarily detect that r is faulty because p's hardware clock can also drift apart from real-time by up to (see Figure 8). When half of the reference time servers would be faulty, in a scenario like the one depicted in Figure 8, p has to be able to decide which reference clocks are correct to bound the external deviation: p has eventually to decide if it adjusts its clock towards X r1 or X r. The proof of the lower bound theorem is done by contradiction. When less than F +1 reference time servers are available, for any synchronization algorithm that claims to achieve external synchronization 11

12 T3 T T 1 T0 clock-time t 0 dt dt -1 = t 1 dr + ρ t X r1 dt dt -1 = H p dt dt -1 = X r dr ρ dr t real-time 3 Figure 8: Process p cannot correctly decide if reference clock X r1 or X r (or both) are faulty, because p does not know the drift rate dr of its hardware clock: (1) when dr =?, X r is faulty, () when dr =, X r1 is faulty, (3) when jdrj <, X r1 and X r are faulty. it is possible to nd two for non-reference time servers indistinguishable runs in which the set of reference time servers is partitioned into two sets P and Q with each containing at most F reference time servers. In the rst run P contains all correct reference time servers and Q contains all faulty ones, while in the second run the situation is reversed. The runs can be chosen such that all reference clocks in each set show at any time the same value, but there is a small drift between two reference clocks that belong to the dierent sets P and Q. For example, let the drift rate dr of all hardware clocks in the system be the same, let the drift rate of the reference clocks in P be dr?, and let the drift rate of the reference clocks in Q be dr +. Let dr be dened by dr = + in the rst run, i.e. the reference time servers in P are correct and the ones in Q are faulty. In the second run let dr=-, that is, the reference time servers in P are faulty and the ones in Q are correct. The deviation between the reference clocks of P and Q will eventually be greater than any given bound, thus each non-reference time server p has eventually to decide to which of the two sets it should synchronize its clock. We show that this decision cannot always be done correctly: if the decision is to synchronize towards P and the algorithm actually experiences the second run, i.e. the decision is wrong. Similarly, if the decision is towards Q and the algorithm experiences the rst run, the decision is equally wrong. Thus, less than F +1 reference clocks are not enough for achieving external clock synchronization despite up to F faulty reference clocks. Theorem (Lower Bound Theorem) There is no correct external clock synchronization algorithm capable of masking F arbitrary failures of reference time servers if the total number of reference time servers is not greater than F. Proof To prove this theorem, we introduce a formal model which is a variant of our system and failure model. This formal model makes stronger assumptions than the ones used in the system/failure model, e.g. all remote clock reading errors are assumed to be zero. External clock synchronization in this model becomes therefore simpler to accomplish. We prove that the lower bound theorem is valid in this formal model: at least F + 1 reference time servers are needed to provide external clock synchronization in case up to F reference time servers can suer arbitrary failures. Since the formal model uses stronger assumptions, any run that satises the requirements of the formal model also satises the requirements of the system/failure model. In other terms, the set F R of valid runs of the formal model is a subset of the set IR of valid runs of the system/failure model. Therefore, any external clock synchronization algorithm that would achieve external clock synchronization for less than F + 1 reference time servers in any run of IR has also to achieve external synchronization for any run of F R. Since we show that this is not even possible for the runs in F R, there exists no algorithm that can achieve external clock synchronization in the assumed system/failure model. Formal Model The formal model consists of two components (see Figure 9): the environment which determines the behavior of the reference and hardware clocks, and the algorithm that determines the values of the virtual clocks. The behavior of the environment is described by a set of innite state sequences (e-runs), e.g. these e-runs determine for each point in reference time the values of the reference and hardware clocks. A clock synchronization algorithm is represented by a set of in- nite state sequences (c-runs): they describe for any valid e-run, all possible reactions of the algorithm. The intuition behind the lower bound proof is that we show how to select for each algorithm an e-run such that one of the reactions of the algorithm { one c-run 1