Testing from a Finite State Machine: An introduction 1

Transcription

1 Testing from a Finite State Machine: An introduction 1 The use of Finite State Machines (FSM) to model systems has lead to much interest in deriving tests from them. Having derived a test sequence from an FSM, this is applied to the implementation under test (IUT) and the test output compared with that expected. At the end of this section you should be able to: Define a finite state machine and explain how it operates Define and explain the relevance of terms related to finite state machines Find sequences to reach or distinguish states Find a distinguishing sequence, UIOs, or a characterizing set for an FSM Generate a transition tour from an FSM Apply Chow s method Reading The material on FSM based testing is not covered in the core text. This document thus provides an overview of the topic. Section 1 briefly describes finite state machines. Section 2 considers the problem of reaching a state of an FSM. Approaches to verifying a state are then described in Section 3 and test generation is discussed in Section Finite state machines A finite state machine (also called a Mealy Machine) M is defined by a tuple (S,s 1,!,",X,Y) where S is a finite set of states, s 1 is the initial state (and thus s 1 #S),! is the state transfer function, " is the output function, X is the finite input alphabet, and Y is the finite output alphabet. If M receives input x#x when in state s i #S, it moves to state s j =!(s i,x) and produces output y="(s i,x). This defines a transition (s i,s j,x/y). A graphical representation of an FSM given in Aho et al. [1988] is shown in Figure 1. Here the initial state is s 1. This FSM will be denoted M 0 throughout this document. A general FSM with n states will be denoted M=(S,s 1,!,",X,Y). The functions! and " can be extended to take input sequences, giving functions! * (type S$X * %S) and " * (type S$X * %Y * ) respectively. Here given a set A, A * denotes the set of strings of elements taken from A (this includes the empty sequence). 1 These notes are closely based on notes by Prof. Robert Hierons, whom I gratefully acknowledge. b/1 s 2 s 3 b/0 c/1 b/0 s 5 b/0 a/2 s 1 s 4 c/1 Figure 1: The FSM M 0 The formal definition of an FSM can seem daunting. Don t be daunted. Many of the elements of the six-element tuple which define an FSM and really simple. In fact, four are very simple ones. X is the finite input alphabet, and Y is the finite output alphabet. These are just the set of things which can be input to the machine (X) and the set of things which it can produce as output (Y). In defining them we are simply declaringthe input type and the output type. S is the set of states, we usually label these S1,,Sn for n states, so s is really nothing more than a set of labels. OK, so that is three elements ofthe six element tuple. The element S1, is simply the start state of the machine. Without formally defining it, we would not know here to start execution from, because the drawing ofthe state machine is just a graph (as you can see, above). So that leaves two elements, the delta function (!) and the lambda function ("). These are there to show how the machine behaves. The delta function tells us how the states are connected together. In the diagram, it is the delta function which defines the arrows of the graph. The delta function tells is to which state we should move, when we find ourselves in a state s, and we receive an input i. Therefore, it is a function, which takes as parameters, the state we are currently in the s value) and the input we receive (the i value) and returns as its result, the label of the state to which we should move. The lambda function defines the output that is produced. It needs to take as parameters the input received by the machine and the state that the machine is in. Based upon these two parameters, it determines the output to be produced. Consider the behaviour of M 0. If M 0 receives input abaa, when in the initial state s 1, we get the following: 1 2

2 The initial input a triggers the transition (s 1,s 1,) and thus M 0 does not change state and outputs 0 (the final state of the transition is s 1 ). The second input b now triggers the transition (s 1,s 2,b/0) and thus M 0 moves to state s 2 and outputs 0. The third input a is now received when M 0 is in state s 2 and triggers the transition (s 2,s 5,). Thus M 0 moves to state s 5 and outputs 0. The final input a is received when M 0 is in state s 5 and triggers the transition (s 5,s 4,a/2). Thus M 0 moves to state s 4 and outputs 2. Thus the input of abaa in state s 1 leads to output 0002 and to M 0 moving to state s 4. From this we can see that in M 0,! * (s 1,abaa)=s 4 and " * (s 1,abaa)=0002. An FSM is said to be deterministic if there are no pairs of transitions that have the same initial state and input. This is implicit in the above definition, since! and " are functions (in each case there is only one allowed value given a state and input). We shall only consider deterministic FSMs. An FSM M is said to be completely specified if for each input x#x and each state s i there is a transition in M from s i that has input x. If an FSM M is not completely specified, a completely specified FSM M can be produced from M by applying one of the following approaches: 1. Adding an error state. For each (s i,x), with s i #S and x#x, such that there is no transition from s i with input x, we add a transition with input x from s i to the error state and make the output of this transition be some error message. 2. Adding null actions. For each (s i,x), with s i # S and x#x, such that there is no transition from s i with input x, we add a transition with input x from s i to s i and make the output of this transition be null. 3. The first approach corresponds to the case where there being no specified behaviour for input x and state s i means that the system should not receive x when in state s i and thus that this should lead to an error message or an exception being raised. The second case corresponds to the assumption that there being no specified behaviour for input x and state s i means that in state s i the input of x has no effect. Consider now the FSM M 0 in Figure 1 and assume that we apply the second approach. The FSM M 0 is converted to a completely specified FSM by adding the transitions (s 2,s 2,c/null), (s 3,s 3,a/null), (s 4,s 4,c/null), (s 5,s 5,b/null) and (s 5,s 5,c/null). Only completely specified FSMs shall be considered. An FSM M is initially connected if every state of M is reachable from the initial state of M. If there is a state s that is not reachable from the initial state, s may be removed since it does not contribute to the specified behaviour of M. We may thus assume that any FSM considered is initially connected. FSM M is strongly connected if, given any ordered pair of states (s i,s j ), there is a sequence of transitions that moves M from s i to s j. A reset operation is an input that takes M to its initial state irrespective of the current state. Many real systems have a reset operation; sometimes this simply represents switching the system off and then on again. A number of test techniques assume the existence of a reliable reset operation: a reset operation that is known to have been implemented correctly. Clearly, if M has a reset operation and is initially connected then it is strongly connected. Since most systems have some reset it is normal to assume that any FSM considered is strongly connected. It is easy to confirm that M 0 is initially connected. For example, in order to reach the state s 5 from the initial state it is sufficient to use the input sequence ca. Thus, M 0 with a reset operation added is strongly connected. Two states s i and s j of an FSM M are said to be distinguished by input sequence x if the input of x in states s i and s j leads to different output sequences. Essentially this means that, if you knew that the implementation was currently either in state s i or s j then in order to determine the current state it would be sufficient to input x and observe the output. More formally, x distinguishes s i and s j if " * (s i,x)&" * (s j,x). If some input sequence distinguishes s i and s j they are said to be distinguishable. For example, in M 0 the states s 1 and s 5 are distinguishes by input a. Thus, if we knew that the IUT was either in state s 1 or s 5 it would be sufficient to input a: it the response was 2 then we were is s 5 and if the response was 0 we were in s 1. Two states of an FSM M are said to be equivalent if they are not distinguishable. This is exactly when the same set of input/output sequences is produced from each state. Two FSMs are equivalent if their initial states are equivalent. An FSM M is said to be minimal if there is no equivalent FSM with fewer states. When M is initially connected, M is minimal if it has no pair of equivalent states. It is relatively easy to check that no two of states M 0 are equivalent. However, the FSM given in Figure 2 has two states and these are equivalent: for every input sequence x, the output sequences produced by the input of x in these two states are identical. b/1 b/1 Figure 2: An FSM with two equivalent states Given an FSM it is possible to produce an equivalent minimal FSM by merging equivalent states. For example, the FSM in Figure 2 is equivalent to a one state FSM in which there is an arc from the state to itself with label and there is an arc from the state to itself with label b/1. A directed graph (digraph) is defined by a set of vertices and a set of directed edges between the vertices. Labels can be added to the edges. Thus, an FSM can be 3 4

3 represented by a digraph in which the vertices correspond to states and the edges, labelled with the input/output, correspond to the transitions. 2. Reaching States Suppose we wish to test the behaviour of the IUT corresponding to a transition from state s of M. Then we need to find some input sequence that takes M from its initial state to state s. This may be achieved by simply searching through the paths from the initial state. If done by hand, the easiest approach is to find a path to state s and then find the corresponding input sequence. Consider, for example, the problem of reaching state s 5 of M 0. Here we start at s 1 and we might note that we can follow the arc from s 1 to s 2 and then the arc from s 2 to s 5. The first of these edges has input b and the second has input a. Thus, s 5 is reached by the input sequence ba from s 1 (the corresponding output sequence is 00) 3. State verification Testing from an FSM can often be seen as a process of checking the transitions of the IUT against those of the FSM. In order to check a transition t=(s i,s j,x/y) we need to input a sequence to move to state s i, input the value x (checking that y is output) and then check that the state is now s j. Testing from an FSM specification is typically black-box testing: the actual state of the implementation is not visible to the tester. This leads to two additional problems: Controllability: when testing a transition from state s it is necessary to apply an input sequence that takes the implementation to the corresponding state Observability: when testing a transition from state s to state s it is necessary to check that the state of the IUT is correct after the transition. The problem of controllability has already been discussed. In order to overcome the observability problem it is sufficient to be able to check the state of the IUT by providing further input and observing the output produced. Thus in order to test a transition it is necessary to have some way of checking its final state. There are three main approaches to checking a state. These are based upon: 1. Distinguishing sequence (DS) 2. Unique Input/Output sequences (UIO) 3. Characterizing set A distinguishing sequence is an input sequence that produces a different output for each state. Thus, when a distinguishing sequence D is input, the output produced in response to this identifies the state the system was in before D was input. Given a distinguishing sequence D, in order to check transition t we follow it by D. It is easy to check that the FSM M 0 (with null transitions added) has a number of distinguishing sequences, including ba. Note that here we have to consider the additional transitions with null outputs. In order to check the final state of a transition using a distinguishing sequence D it is sufficient to follow the transition by D. For example, to check the final state of the transition (s 2,s 5,) of M 0 it is sufficient to input b (to get to state s 2 ), then input a (to execute the transition) and then input ba (to check the final state of the transition). Thus the input sequence baba checks the transition (s 2,s 5,) of M 0. An input/output sequence x/y is a unique input/output sequence for a state s i if the following hold: y is the output produced when x is input in state s i (y=" * (s i,x)); and for all s j &s i we have that " * (s j,x)&y. Thus x distinguishes s i from all other states of M. Although x can verify s i it need not be able to verify any other state. Thus, when testing a transition t we follow it by the UIO for the final state of t. Clearly a DS is a UIO for each state. However, some FSMs have a UIO for every state but no DS. M 0 has the following UIOs:, for s 1, b/1 for s 2, c/1,a/2 for s 3, b/0,c/1 for s 4, and a/2 for s 5. Note: since ba is a DS for M 0, it also forms the input of a UIO for each state of M 0. In order to find a UIO for a state s it is sufficient to search through the set of input sequences until one is found. In practice we can form a table in which each row represents a state and each column represents an input sequence. The contents of a cell, for state s and input sequence x, is the response of M when it receives input x when in state s. An input sequence x is the input of a UIO for state s if the contents of the cell for (s,x) is different from the contents of all other cells in that column: the output sequence corresponding to (s,x) is unique. Note: if all the cells in a column are different, the input sequence is a distinguishing sequence. Consider the problem of finding a UIO for state s 1 of M 0. We may extend the table until a UIO is found, forming the following table. State Response to a Response to b Response to c Response to aa s ,0 s null 0,2 s 3 null 0 1 null,null s null 0,2 s 5 2 null null 2,0 Note that from the above, the input aa gives the input of UIOs for states s 1, s 3, and s 5 since their cell contents in this column do not appear elsewhere in this column. Further, a forms the input of UIOs for states s 3 and s 5 and b forms the input of a UIO for states s 2 and s 5. To check the transition (s 2,s 5,) of M 0 using the UIO b/1 it is sufficient to use the following input sequence: bab. Some FSMs do not have either a DS or a UIO for each state. Every minimal FSM does, however, have a characterising set. A characterising set W is a set of input 5 6

4 sequences with the property that, if every sequence is executed from some state s i, the set of output sequences verifies s i. Thus W is a characterizing set if for every pair of states (s i,s j ) there is some input sequence w#w that distinguishes s i and s j. When checking a transition t=(s i,s j,x/y) using characterising set W we need to, separately, follow t by each element from W. The set {a,b,c} is a characterising set for M 0. Naturally, since ba is a DS for M 0, {ba} is an alternative characterising set for M 0. In order to generate a characterising set for an FSM it is sufficient to produce a table as above and find a set of columns with the property that the corresponding values of the cells identify a state. From the above table it is possible to confirm that {a,b,c} is a characterising set for M 0 : for each row there is a unique sequence of values in these columns. For example, the row for s 1 has values 0,0,1 and this appears in no other row (no other state of M 0 responds to a with 0, responds to b with 0, and responds to c with 1). Thus, to check the transition (s 2,s 5,) of M 0 using this characterising set it is sufficient to use the following separate input sequences: baa, bab, bac. Naturally, where there may be state transfer faults a transition tour is usually seen to be insufficient. In fact, where there may be state transfer faults a transition tour is not even guaranteed to detect every output fault: state transfer faults may mask such output faults. Figure 3 gives an example of this. Here the specification is M 3 and the implementation machine M 4 contains an output fault (in the transition from s 3 ) that, due to the state transfer fault in the transition from s 2, is not detected by the transition tour aaa. To see that M 4 is faulty, consider its response to input aaaa and the response of the specification M 3 to aaaa. 4. Test generation This section will provide an overview of some FSM based test techniques. The techniques that will be described are: 1. The transition tour 2. Chow's method (also called the W-method) 3. The UIO method (not covered) When testing from an FSM M it is normal to assume that the implementation under test (IUT) can be modelled by an unknown FSM M I that has the same input and output alphabets as M. Thus testing involves comparing the behaviour of two FSMs (one of which is unknown: it acts as a black box). Normally we also assume that M I has at most m states for some predetermined m. A fault in a transition can be categorized as one or more of the following: 1. an output fault: a transition has the wrong output; or 2. a state transfer fault: the transition takes M to the wrong state. The different test techniques vary in the types of faults they look for and the properties of the specification they assume. We will now describe these techniques The transition tour method If we assume that only output faults exist it is sufficient to produce some test sequence that executes every transition of M. Such a sequence is called a transition tour. For example, M 0 has the following transition tour (note the need to use resets): s 1 -a->s 1 -b->s 2 -a-s 5 -a->s 4 -b->s 3 -c->s 5 -a->s 4 -a->s 5 -a->s 4 -b->s 3 -b-s 5 -reset->s 1 -c->s 4 -reset->s 1 -b-s 2 -b-s 3 -reset->s 1 By following this path through M 0 it is possible to confirm that it goes through every edge and thus includes every transition. This path leads to the following test sequence: a,b,a,a,b,c,a,a,a,b,b,reset,c,reset,b,b,reset 7 8

5 s 1 s 2 s 1 a/1 a/1 M 3 M 4 a/1 s 3 s 3 s 2 VW'VXW'VXXW' 'VX m-n+1 W Here, given a set A the term A k denotes AA A (k times). Thus, for example, A 4 denotes AAAA and contains all sequences that are composed of 4 elements from A. For example, since X={a, b, c}, X 2 ={aa, ab, ac, ba, bb, bc, ca, cb, cc}. If, for example, m=n+3 we get the following test set: Figure 3: Specification M 3 and faulty implementation M Chow's method We shall now describe Chow's method, which produces a checking experiment: a set of input sequences that, between them, determine correctness as long as the IUT behaves like some (unknown) FSM that has no more than m states and with the same input alphabet as M. The sequences produced are separated by a reset and thus the method relies upon the existence of a reliable reset operation. Chow's method uses a characterising set W and a state cover V, where V is a state cover if it is a set of input sequences with the property that: each state of M is reached from s 1 by exactly one sequence from V. Naturally, there will normally be more than one choice for this state cover but a minimal state cover may be found using a breadth-first search (see Section 2). M 0 has state cover {(, b, c, bb, ca} (where ( denotes the empty sequence). Here ( reaches s 1, b reaches s 2, c reaches s 4, bb reaches s 3, and ca reaches s 5. We might also use the characterising set W={a, b, c}. Where it is assumed that the implementation M I has no more state than M (and thus m=n) Chow's method produces the following test: VW'VXW Here two sets P and Q being sequenced to form PQ means we take all the sequences that may be formed by: following a sequence from P by a sequence from Q. Thus PQ={pq p#p,q#q}. Test sequences are separated by (reliable) resets. In the example this test is: {(, b, c, bb, ca}{a, b, c}'{(, b, c, bb, ca}{a, b, c}{a, b, c} The first of these sets ({(, b, c, bb, ca}{a, b, c}) is: VW'VXW'VXXW' VXXXW ' VXXXXW It is not as hard as it seems This material may seem quite hard, but it is very easy if you practice. I strongly recommend that you attempt all of the exercises for this week s lectures and check the model answers to see that you understand this material. The definitions may take a while to master, but the ideas are really very intuitive. You can use the diagrammatic form of the FSM to guide you. A lot of the ideas I present are much more immediate in terms of the diagrams, than in terms of the definitions. This is particularly the case for those students who are not mathematically minded. I would point out that testing from finite state machines, using the techniques we are learning here, is a very widespread activity in industry, particularly in the telecommunications industry and in any situation where embedded systems are used. 5. References 1. Aho A. V., Dahbura A. T., Lee D., and Uyar M. U., 1988, An Optimization Technique for Protocol Conformance Test Generation Based on UIO Sequences and Rural Chinese Postman Tours, In the Proceedings of Protocol Specification, Testing, and Verification VIII, pages 75-86, Elsevier (North-Holland). 2. Sidhu D. P. and Leung T.-K., 1989, Formal Methods for protocol testing: A detailed Study, IEEE Transactions on Software Engineering, 15 4, pp {a, b, c, ba, bb, bc, ca, cb, cc, bba, bbb, bbc, caa, cab, cac} The second set is rather large (it has 45 elements) and so its elements will not be listed here. Note: in this example W=X but in general this will not be the case. Where m>n Chow's method produces the following test set: 9 10