Monitoring Distributed Controllers

Transcription

1 Monitoring Distributed Controllers When an Efficient LTL Algorithm on Sequences is Needed to Model-Check Traces A. Genon T. Massart C. Meuter Université Libre de Bruxelles Département d Informatique August 25, 2006

2 Need for validation Need for validation Monitoring Centralized v.s. Distributed Systems Related Works Distributed control systems concurrent processes running on physically distributed hardware hard to design in nature critical systems (e.g. plant control system,...)

3 Need for validation Need for validation Monitoring Centralized v.s. Distributed Systems Related Works Distributed control systems concurrent processes running on physically distributed hardware hard to design in nature critical systems (e.g. plant control system,...) Validation techniques Model-Checking: exhaustive verification but: not (yet) scalable to real-sized systems Testing and Monitoring: non-exhaustive verification scalable for real-sized systems widely used in industry

4 Monitoring Need for validation Monitoring Centralized v.s. Distributed Systems Related Works The system distributed and asynchronous instrumented to emit relevant events (e.g. variable assignments, message transfer) an execution trace is collected

5 Monitoring Need for validation Monitoring Centralized v.s. Distributed Systems Related Works The system distributed and asynchronous instrumented to emit relevant events (e.g. variable assignments, message transfer) an execution trace is collected The monitor seperate process which collects those events checks whether a certain property holds can be done online or offline

6 Centralized v.s. Distributed Systems Need for validation Monitoring Centralized v.s. Distributed Systems Related Works Centralized System only one process events are totally ordered φ The OK e1 e2 e3 e4 Monitor KO

7 Centralized v.s. Distributed Systems Need for validation Monitoring Centralized v.s. Distributed Systems Related Works Centralized System only one process events are totally ordered φ The OK e1 e2 e3 e4 Monitor KO Distributed System multiple processes events are not totally ordered but a partial order can be obtained using vector clocks [Lam78, Mat89] P3 e2 e1 e3 e5 e6 e4 φ The Monitor OK KO

8 Related Works Need for validation Monitoring Centralized v.s. Distributed Systems Related Works Global predicate detection Several classes of predicate have been studied: Stable predicates [CL85] (once true, remains true) Disjunctive predicates (of the form lp 1 lp 2 lp n) Conjunctive predicates [GW94, GW96] (of the form lp 1 lp 2 lp n) Observer independent predicates [CDF95] (such that φ φ) Linear, semi-linear [CG98] or regular predicates (RCTL logic) [GM01, SG03]

9 Related Works Need for validation Monitoring Centralized v.s. Distributed Systems Related Works Global predicate detection Several classes of predicate have been studied: Stable predicates [CL85] (once true, remains true) Disjunctive predicates (of the form lp 1 lp 2 lp n) Conjunctive predicates [GW94, GW96] (of the form lp 1 lp 2 lp n) Observer independent predicates [CDF95] (such that φ φ) Linear, semi-linear [CG98] or regular predicates (RCTL logic) [GM01, SG03] Trace model checking Temporal logics for Mazurkiewicz traces: TrPTL [Thi94] TCL [APP95] LTrL [TW02] LTL over traces [DG02] = = our approach

10 Traces and Monitors Traces and Monitors The problem Complexity result Traces partially ordered set of events events are labelled with assignments finite number of events

11 Traces and Monitors Traces and Monitors The problem Complexity result Traces partially ordered set of events events are labelled with assignments finite number of events Monitor non-deterministic finite automaton with guards on transitions built from an LTL formula using tableau construction [VW86] loop transitions are ommitted and the monitor never blocks each guard is a boolean formula on one variable some states are marked as bad x > 0 init tmp w = 0 bad

12 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state?

13 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad

14 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad

15 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad x:=0

16 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad x:=0;

17 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad x:=0; ; y:=3

18 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad x:=0; ; y:=3; x:=5

19 The problem Traces and Monitors The problem Complexity result Question Does there exist an interleaving (i.e. total order) of events compatible with the partial order, leading the monitor to a bad state? Example init x > 0 tmp w = 0 bad x:=0; ; y:=3; x:=5;

20 The problem Traces and Monitors The problem Complexity result A simple solution Compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad

21 The problem Traces and Monitors The problem Complexity result A simple solution Compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad ([0,0],init) x := 0 ([1,0],init) y:= 3 ([2,0],init) x:=5 ([3,0],tmp) ([0,1],init) x := 0 ([1,1],init) y:=3 ([2,1],init) x:=5 ([3,1],tmp) ([3,2],bad) ([2,2],init) x:=5 ([3,2],tmp)

22 The problem Traces and Monitors The problem Complexity result A simple solution Compose the trace with the monitor to explore all possibilities: x > 0 w = 0 Cut init tmp bad ([0,0],init) x := 0 ([1,0],init) y:= 3 ([2,0],init) x:=5 ([3,0],tmp) ([0,1],init) x := 0 ([1,1],init) y:=3 ([2,1],init) x:=5 ([3,1],tmp) ([1,1],init) Configuration = Cut + Monitor State ([2,2],init) x:=5 ([3,2],bad) ([3,2],tmp)

23 The problem Traces and Monitors The problem Complexity result A simple solution Compose the trace with the monitor to explore all possibilities: x > 0 w = 0 Cut init tmp bad ([0,0],init) x := 0 ([1,0],init) y:= 3 ([2,0],init) x:=5 ([3,0],tmp) ([0,1],init) x := 0 ([1,1],init) y:=3 ([2,1],init) x:=5 ([3,1],tmp) ([1,1],init) Configuration = Cut + Monitor State ([2,2],init) x:=5 ([3,2],bad) ([3,2],tmp) Problem: exponential number of interleavings/configurations! = Can we do better?

24 Complexity result Traces and Monitors The problem Complexity result Theorem The trace monitoring problem is NP-Complete

25 Complexity result Traces and Monitors The problem Complexity result Theorem The trace monitoring problem is NP-Complete NP-easyness Simple non-deterministic algorithm: guess an interleaving of the events in the trace check in polynomial time if it leads the monitor to a bad state

26 Complexity result Traces and Monitors The problem Complexity result Theorem The trace monitoring problem is NP-Complete NP-easyness Simple non-deterministic algorithm: guess an interleaving of the events in the trace check in polynomial time if it leads the monitor to a bad state NP-hardness Reduction from 3-SAT problem: use the monitor to encode the formula φ use the trace to encode all possible valuations a valuation satisfying φ iff the bad state is reachable

27 Sensitive events Sensitive events Observation The monitor is not always sensitive to all events:... x:=3... x < 6 m... z:=5... w < 5

28 Sensitive events Sensitive events Observation The monitor is not always sensitive to all events:... x:=3... x < 6 m... z:=5... w < 5

29 Sensitive events Sensitive events Observation The monitor is not always sensitive to all events:... x:=3... x < 6 m... z:=5... w < 5 Firing non-sensitive events has no effect on the monitor state

30 Sensitive events Sensitive events Observation The monitor is not always sensitive to all events:... x:=3... x < 6 m... z:=5... w < 5 Firing non-sensitive events has no effect on the monitor state Can we do something about it? Exploring all interleavings of non-sensitive events is not necessary = explore only one arbitrary interleaving But : those events might be interesting in another state of the monitor = keep them seperate during exploration

31 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order init x > 0 tmp w = 0 bad

32 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order init x > 0 tmp w = 0 bad

33 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order init x > 0 tmp w = 0 bad

34 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order 2 Both the origin cut and the final cut are kept init x > 0 tmp w = 0 bad [1,0] [2,2]

35 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order 2 Both the origin cut and the final cut are kept 3 Then, sensitive events are fired on all the cuts in between, in one step init x > 0 tmp w = 0 bad [1,0] [2,2]

36 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order 2 Both the origin cut and the final cut are kept 3 Then, sensitive events are fired on all the cuts in between, in one step init x > 0 tmp w = 0 bad [3,0] [3,2]

37 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order 2 Both the origin cut and the final cut are kept 3 Then, sensitive events are fired on all the cuts in between, in one step 4 Some previously non-sensitive events become sensitive = they must be explored! init x > 0 tmp w = 0 bad [3,0] [3,2]

38 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order 2 Both the origin cut and the final cut are kept 3 Then, sensitive events are fired on all the cuts in between, in one step 4 Some previously non-sensitive events become sensitive = they must be explored! init x > 0 tmp w = 0 bad [3,0] [3,2]

39 Sensitive events Idea 1 First all non-sensitive events are fired in an arbitrary order 2 Both the origin cut and the final cut are kept 3 Then, sensitive events are fired on all the cuts in between, in one step 4 Some previously non-sensitive events become sensitive = they must be explored! init x > 0 tmp w = 0 bad [3,2] [3,2]

40 (cont d) Sensitive events Symbolic Composition Symbolically compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad

41 (cont d) Sensitive events Symbolic Composition Symbolically compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad x := 0 x:=5 ([0,0],[0,1],init) ([1,0],[2,2],init) ([3,0],[3,2],init) ([1,0],[2,2],init)] ([3,1],[3,1],init) ([3,2],[3,2],bad)

42 (cont d) Sensitive events Symbolic Composition Symbolically compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad x := 0 x:=5 ([0,0],[0,1],init) ([1,0],[2,2],init) ([3,0],[3,2],init) ([1,0],[2,2],init)] ([3,1],[3,1],init) ([3,2],[3,2],bad) Symbolic Configuration = Origin Cut + Final Cut + Monitor State

43 (cont d) Sensitive events Symbolic Composition Symbolically compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad x := 0 x:=5 ([0,0],[0,1],init) ([1,0],[2,2],init) ([3,0],[3,2],init) ([1,0],[2,2],init)] ([3,1],[3,1],init) ([3,2],[3,2],bad) Symbolic Configuration = Origin Cut + Final Cut + Monitor State One symbolic configuration represents a set of configurations: ([1,0],[2,2],init) { ([1,0],init), ([1,1],init), ([1,2],init), ([2,2],init) }

44 (cont d) Sensitive events Symbolic Composition Symbolically compose the trace with the monitor to explore all possibilities: init x > 0 tmp w = 0 bad x := 0 x:=5 ([0,0],[0,1],init) ([1,0],[2,2],init) ([3,0],[3,2],init) ([1,0],[2,2],init)] ([3,1],[3,1],init) ([3,2],[3,2],bad) Symbolic Configuration = Origin Cut + Final Cut + Monitor State One symbolic configuration represents a set of configurations: ([1,0],[2,2],init) { ([1,0],init), ([1,1],init), ([1,2],init), ([2,2],init) } Symbolic composition is both sound and complete!

45 In practice In practice Does it work? Algorithm input : T = (E, λ, ), M = (M, m 0, B, m) output: reachable s(,, m 0 ) B begin T, W {(,, m 0 )} while W do let (t, w, m) W W W \ {(t, w, m)} T T (t, w, m) repeat x w w w {e enabled(w) e sensitive(m)} until (w = x) if (w = E) (m B) then return false return true end forall (t, w, m ) : (t, w, m) e s (t, w, m ) (t, w, m ) T do W W {(t, w, m )}

46 In practice In practice Does it work? Algorithm input : T = (E, λ, ), M = (M, m 0, B, m) output: reachable s(,, m 0 ) B begin { } T, W {(,, m 0 )} while let (t, W w, m) do W let (t, w, m) W WW W \{(t, {(t, w, w, m)} m)} T T T (t, w, w, m) m) repeat x w w w {e enabled(w) e sensitive(m)} until (w = x) if (w = E) (m B) then return false return true end 1 pick a symbolic configuration forall (t, w, m ) : (t, w, m) e s (t, w, m ) (t, w, m ) T do W W {(t, w, m )}

47 In practice In practice Does it work? Algorithm input : T = (E, λ, ), M = (M, m 0, B, m) output: reachable s(,, m 0 ) B begin T, W {(,, m 0 )} while W do let (t, w, m) W W W \ {(t, w, m)} T T (t, w, m) repeat repeat xx w w w {e enabled(w) e sensitive(m)} until x) until (w = x) if (w = E) (m B) then return false return true end w w {e enabled(w) e sensitive(m)} 1 pick a symbolic configuration 2 extend it with non-sensitive events forall (t, w, m ) : (t, w, m) e s (t, w, m ) (t, w, m ) T do W W {(t, w, m )}

48 In practice In practice Does it work? Algorithm input : T = (E, λ, ), M = (M, m 0, B, m) output: reachable s(,, m 0 ) B begin T, W {(,, m 0 )} while W do let (t, w, m) W W W \ {(t, w, m)} T T (t, w, m) repeat x w w w {e enabled(w) e sensitive(m)} until (w = x) if if (w (w = E) (m (m B) B) then then return false return true end return false 1 pick a symbolic configuration 2 extend it with non-sensitive events 3 check if a bad state is reached forall (t, w, m ) : (t, w, m) e s (t, w, m ) (t, w, m ) T do W W {(t, w, m )}

49 In practice In practice Does it work? Algorithm input : T = (E, λ, ), M = (M, m 0, B, m) output: reachable s(,, m 0 ) B begin T, W {(,, m 0 )} while W do let (t, w, m) W W W \ {(t, w, m)} T T (t, w, m) repeat x w w w {e enabled(w) e sensitive(m)} until (w = x) if (w = E) (m B) then return false forall (t, w,, m ) :)(t, : w, (t, m) w, e m) s (t e, w, m ) (t, w, m s (t, w, m ) )(t T, w do, m ) T do {(t, w, m )} W W {(t, w, m )} return true end 1 pick a symbolic configuration 2 extend it with non-sensitive events 3 check if a bad state is reached 4 fire symbolically sensitive events

50 Does it work? In practice Does it work? Early results Experiment Explicit Symbolic Spin Model Processes Events Property Error Time Conf. Error Time Conf. Time Peterson Mutex NO 1.39s NO 0.35s s Mutex NO 16.88s NO 3.45s s Mutex NO 34.75s s Peterson Mutex YES 1.11s YES 0.01s s Faulty Mutex YES 15.95s YES 0.05s s Mutex YES 0.53s s ABProtocol Received NO 2.17s NO 0.42s s Received NO 31.08s NO 4.25s s Received NO 43.09s s ABProtocol Received YES 2.06s YES 0.01s s Faulty Received YES 29.70s YES 0.06s s Received YES 0.53s s Philosopher Fork NO 1.03s 6190 NO 0.05s s Fork NO 87.02s NO 0.21s s Fork NO 1.52s Philosopher Fork YES 0.09s 1187 YES 0.01s s Faulty Fork YES 78.72s YES 0.01s s Fork YES 0.01s 55 - Table 1. s ( - means that the execution ran out of memory)

51 What has been achieved? we proposed a symbolic method to monitor distributed systems has been implemented and works well in practice a tool, TraX (Trace explorer) is available:

52 What has been achieved? we proposed a symbolic method to monitor distributed systems has been implemented and works well in practice a tool, TraX (Trace explorer) is available: Future Work integrate the method into our distributed controler development environment dsl [DGMM05] apply the method to similar models, such as MSC extend the method to do model-checking, using McMillan s unfoldings