CS 347 Parallel and Distributed Data Processing Spring 2016 & Clocks, Clocks, and the Ordering of Events in a Distributed System. L. Lamport, Communications of the ACM, 1978 Notes 15: & Clocks CS 347 Notes 15 2 & Clocks Fundamental concepts in distributed (database) systems Example use cases outs Identifying messages, transactions Setting priorities Tracking data versions & Clocks What is time? How can clocks be implemented? CS 347 Notes 15 3 CS 347 Notes 15 4
Ordering of Events Event ordering is even more basic than time Ordering of Events Introduce a partial order relation Event a 9 am Happened before Physical time not needed to order events Event b 10 am Reminder Total ordering: for any two events a, b (a b) either a b or b a Partial ordering: some events a, b can be concurrent Event a Happened before Can affect Event b CS 347 Notes 15 5 CS 347 Notes 15 6 Ordering of Events Ordering of Events Model Process P Process Q Process R P1 Q1 R1 Q2 P2 Q3 R2 Q4 Q5 R3 P3 Q6 Model Events within a process are totally ordered Sending or receiving a message is an event No assumptions about message transmission times Examples P4 Q7 R4 P 1 P 2 P 1 Q 2 P 3, Q 3 are concurrent CS 347 Notes 15 7 CS 347 Notes 15 8
Ordering of Events Logical Clocks Definition The relation on the set of events of a system is the smallest relation satisfying conditions 1. If a, b are in the same process, and a comes before b, then a b 2. If a is a message send and b is the receipt of the same message by a different process, then a b 3. If a b and b c then a c (transitivity) Process P Process Q Process R P1 Q1 R1 Q2 P2 Q3 R2 Q4 Q5 R3 P3 Q6 R4 CR = 4 CR = 5 CR = 6 CR = 7 CR = 8 Assume a a If a b and b a, then a, b are concurrent P4 Q7 CR = 9 CS 347 Notes 15 9 CS 347 Notes 15 10 Logical Clocks C i = logical clock (counter) at process i C[b] = reading of C j when event b occurs at process j Clock condition If a b then C [ a ] < C [ b ] No relationship to physical time Logical Clocks Clock condition satisfied if the following conditions hold: 1. If a and b are events in process i and a comes before b then C i [ a ] < C i [ b ] 2. If a is the sending of a message by process i and b is the receipt of that message by process j then C i [ a ] < C j [ b ] CS 347 Notes 15 11 CS 347 Notes 15 12
Logical Clocks Logical Clocks Process P Process Q Process R CR = 4 Q1 P1 R1 Q2 6 > 4 (condition 1) P2 CR = 6 Q3 R2 Q4 Q5 R3 P3 Q6 CQ = 9 R4 P4 Q7 9 > 6 (condition 2) Implementation LC1 Each process i increments C i between any two successive events LC2 Let a be the event of process i sending a message m to j Message contains timestamp T m = C i [ a ] When message arrives at j 1. If T m > C j then C j T m 2. Event b of message receipt takes place CS 347 Notes 15 13 CS 347 Notes 15 14 Logical Clocks Notes C i [ a ] < C j [ b ] does not imply a b a, b being concurrent does not imply C i [ a ] = C j [ b ] If C i [ a ] = C j [ b ] then a, b are concurrent Breaking Ties Problem Some server wants to execute requests in order a is event that originated one request (at client) b is event that originates another request (at another client) CS 347 Notes 15 15 CS 347 Notes 15 16
Breaking Ties Solution If C [ a ] < C [ b ] then service a first It could be that a b, but it could be that they re concurrent Assume that a b to stay on the safe side Anomalous Behavior Server C = 100 C = 50 If C [ a ] = C [ b ] then pick one a, b are concurrent E.g., pick the one with lower [ nodeid, processid ] 1 Buyer Make the deposit 2 Seller Cash the check 3 Make phone call CS 347 Notes 15 17 CS 347 Notes 15 18 Anomalous Behavior Anomalous Behavior 1 Server C = 9:00 am C = 9:01 am C = 100 C = 50 Buyer Seller Make the deposit Cash the check 3 Solutions 1. Include phone call into the system 2. Use perfect physical clocks 3. Use real physical clocks May be slightly off Do not fully eliminate anomaly, but reduce its likelihood Also useful for failure detection 2 Make phone call CS 347 Notes 15 19 CS 347 Notes 15 20
Physical Clocks C i ( t ) = clock reading at process i at physical time t Physical Clocks Assume C i ( t ) is a continuous, differentiable function except when clock is reset C i ( t ) C i ( t ) C i +(x) C i (x) time = t x time = t CS 347 Notes 15 21 CS 347 Notes 15 22 Physical Clocks Physical Clock Properties How to enforce clock condition? If a b then C [ a ] < C [ b ] LC1 Each process i increments C i between any two successive events PC1 For each process i, if i does not receive a message at time t, then C i is differentiable at t and dc i(t) > 0 dt CS 347 Notes 15 23 CS 347 Notes 15 24
Physical Clock Properties Physical Clock Properties LC2 Let a be the event of process i sending a message m to j PC2 Let a be the event of process i sending a message m to j Message contains timestamp T m = C i [ a ] When message arrives at j 1. If T m > C j then C j T m Message contains timestamp T m = C i [ a ] Let μ m be minimum transmission delay for m Message m arrives at at physical time t > t + μ m 2. Event b of message receipt takes place When message arrives at j 1. If T m + μ m > C j (t ) then C j+ (t ) T m + μ m 2. Event b of message receipt takes place CS 347 Notes 15 25 CS 347 Notes 15 26 Physical Clock Properties Physical Clock Properties If T m + μ m > C j (t ) then C j + (t ) T m + μ m PC3 Clock drift There exists a constant k << 1 such that for all processes i m t μ m dc i(t) dt 1 < k t i j CS 347 Notes 15 27 CS 347 Notes 15 28
Physical Clock Properties Physical Clock Properties PC3 Clock drift dc i(t) 1 < k dt y+1 Ci k k y+1+k PC4 Clock synchronization For all i, j: C i ( t ) C j ( t ) < ε y+1 k y x x+1 For typical crystal controlled clocks, k < 10-6 CS 347 Notes 15 29 CS 347 Notes 15 30 Physical Clock Synchronization PC4 can be satisfied if a message is sent on every link at least every τ seconds Physical Clock Synchronization Let Δ m be the transmission time of message m Δ m = minimum delay + unpredictable delay Diameter d Δ m = μ m + Z m Assume constant ζ such that Z m < ζ for all m Assume that μ m = μ for all m t 2 Diameter d is the max of min distances (in this example 4, not 3) t 1 Shortest transit time μ Maximum transit time μ + ζ CS 347 Notes 15 31 CS 347 Notes 15 32
Physical Clock Synchronization Theorem PC4 holds for all t > t 0 + dτ with ε d(2kτ + ζ) assuming μ + ζ τ Physical Clock Synchronization Proof for d = 1 Worst case scenario slow At t 1, m leaves f for s with T m = C f ( t 1 ) At t 2, m arrives at s and sets C s( t 2 ) = C f ( t 1 ) + μ α = C f ( t 2 ) C s ( t 2 ) is as large as possible No communication for duration τ Clock f is as fast as possible, s is as slow as possible Just before message at t 2+ τ, clocks are as far as possible = ε s f fast CS 347 Notes 15 33 CS 347 Notes 15 34 Physical Clock Synchronization Physical Clock Synchronization Cs (t2) t2 μ + ζ Clocks τk Cf(t1) t1 f α ε f s α τk C f ( t 2 ) = C f ( t 1 ) + ( 1 + k )( μ + ζ ) C s ( t 2 ) = C f ( t 1 ) + μ α = ( 1 + k ) ( μ + ζ ) μ = k ( μ + ζ ) + ζ t2 s t2 + τ CS 347 Notes 15 35 CS 347 Notes 15 36
Physical Clock Synchronization ε = 2kτ + α α = k ( μ + ζ ) + ζ Anomalous Behavior No anomalous behavior as long as C s ( t 2 ) > C f ( t 1 ) Corresponds to fastest allowed external transmission ε = 2kτ + ζ + k ( μ + ζ ) μ min > ε 1 k Assuming that k 1 and ( μ + ζ ) τ we get ε 2kτ + ζ Exercise: Why? CS 347 Notes 15 37 CS 347 Notes 15 38 Uses for Physical Clocks 1. To order events 2. Detect timeouts (failures) Uses for Physical Clocks t 2 = ( ε + μ + ζ ) ( 1 + k ) + t 1 Example Expecting reply to request by time t 1 At time t 2 = ( ε + μ + ζ ) ( 1 + k ) + t 1 can declare timeout Account for own clock drift Max transmission μ + ζ t1 t1 at remote clock ε Self Other CS 347 Notes 15 39 CS 347 Notes 15 40
Clock Servers With PC2, a fast clock speeds up all others Use a single, reliable clock service instead Broadcast time synchronization messages Assume time server has perfect real time (i.e., k = 0) E.g., based on atomic clocks and GPS Exercise: What is the corresponding ε? Summary Ordering of events Logical clocks Physical clocks Properties Synchronization Clock servers CS 347 Notes 15 41 CS 347 Notes 15 42