LBT for Procedural and Reac1ve Systems Part 2: Reac1ve Systems Basic Theory

Transcription

1 LBT for Procedural and Reac1ve Systems Part 2: Reac1ve Systems Basic Theory Karl Meinke, School of Computer Science and Communica:on KTH Stockholm

2 0. Overview of the Lecture 1. Learning Based Tes:ng for Reac:ve Systems 2. DFA learning with Angluin s L* algorithm 3. Complexity of learning DFA based on: D. Angluin, Learning regular sets from queries and counterexamples, Informa:on and Computa:on, 75:87-106, K. Meinke and M. Sindhu, Incremental Learning- Based tes2ng for Reac2ve Systems, in: Proc. TAP 2011 K. Meinke and F. Niu, Learning- Based Tes2ng for Reac2ve Systems using Term Rewri2ng Technology Proc. ICTSS 2011 M. Czerny, Learning- based so\ware tes:ng: Evalua:on of Angluin s L* algorithm and adapta:ons in prac:ce, Bachelor Thesis, KIT, 2014.

3 2. Learning-Based Testing (LBT) Meinke & Sindhu 2011, Proc. TAP 2011 SUT- Req pass/fail Model checker Input SUT Output Oracle SUT- Model Learner Feedback aka. Model based testing without a model

4 3. Framework for Study: Reac2ve Systems Generally control- oriented tes:ng 1. Requirements language = proposi:onal linear temporal logic (PLTL) 2. Model = FSM, Moore machine 3. Model checker = BDD/SAT- based checkers 4. Learning = regular inference algorithms

5 DFA (Moore) Representation DFA A= ( Q, Σ, q 0, F Q, δ : Q X Σ Q ) 0 1 q 0 q 1 0 System under Learning SUL 1 q L(A) = regular language accepted by A

6 DFA Learning with Observa:on Tables P A Σ * is a finite prefix- closed set of prefixes S A Σ * is a finite suffix- closed set of suffixes T A : P A (P A. Σ) X S A {1, 0,?} is the observa2on table Write T A (p) for a row T A (p,s 1 ),, T A (p,s n )

7 0 1 q 0 q 1 0 OBSERVATION TABLE 1 q ε 0 1 ε ? prefixes suffixes

8 Basic Principle of DFA learning Accessor strings prefixes p that reach each dis:nct state Dis2nguishing strings suffixes s that separate dis:nct states INFERENCE PRINCIPLE If T A (p, s) T A (p, s) then p and p cannot reach the same state in A; s is a dis2nguishing string for p, p

9 Closed & Consistent Tables T A is closed iff, for each p P A.Σ there exists p P A st. T A (p) = T A (p ) T A is consistent iff, for each p, p P A if T A (p) = T A (p ) then for all a Σ, T A (p.a) = T A (p.a )

10 Algebraic Proper:es Being closed is an algebraic closure condi2on Being consistent is an algebraic congruence condi2on The automaton construc:on is a quo2ent algebra construc2on Learning DFA as string rewri2ng systems K. Meinke, CGE: a Sequen2al Learning Algorithm for Mealy Automata, in Proc. ICGI 2010 K. Meinke and F. Niu, Learning- Based Tes2ng for Reac2ve Systems using Term Rewri2ng Technology in Proc. ICTSS 2011

11 Equivalence Oracles Termina:on requires an equivalence oracle If A and SUL are behaviourally equivalent, i.e. L(A) = L(SUL) then equivoracle(a, SUL) = true Otherwise equivoracle(a, SUL) = v Σ * where A(v) SUL(v) LBT uses stochas2c equivalence checking

12 Complexity Observa:ons Stochas:c equivalence checking is more powerful than random test cases Why? Theorem: ac2ve learning of DFA is much more efficient than passive learning : polynomial :me [Angluin 1987] vs. NP- hard [Gold 1978]. Using stochas:c equivalence checking we can PAC learn DFA in polynomial :me (c.f. Kearns and Vazirani 1994).

13 DFA function LStar(DFA: SUL) P A Σ * P A. Σ Σ * S A Σ * T A : P A (P A. Σ ) X S A {accept, reject,?} // table begin A = getinitialhypothesis() while(equivoracle(a, SUL)!= true)) do A =getnexthypothesis(equivoracle(a, SUL)) end return A

14 DFA function getnexthypothesis(counterexample Σ * ) begin P A = P A PrefixClosure{counterExample} P A. Σ = P A Σ - P A S A = S A SuffixClosure{counterExample} // fill in any new table entries here while (P A, S A, T A ) is not closed or consistent do if!consistent(p A,S A,T A ) makeconsistent() if!closed(p A,S A,T A ) makeclosed() end

15 DFA function getinitialhypothesis() begin P A = // emptyset P A. Σ = S A = return getnexthypothesis(ε) end

16 DFA function makeconsistent() begin end find p, p P A, a Σ, s S A st. T A (p) = T A (p ) and T A (p.a,s)!= T A (p.a,s) add a.s to S A // suffix extension extend T A to P A (P A.Σ) X S A using active membership queries

17 DFA function makeclosed() begin find p P A, a Σ st. T A (p.a)!= T A (p ) for all p P A P A = P A {p.a} // prefix extension P A.Σ = P A.Σ {p} Σ end extend T A to P A (P A.Σ) X S A using membership queries

18 DFA function DFASynthesis() begin Q = {u : u P A, v<u, T A (u) T A (v) } q 0 = ε F = {u : u Q, T A (u, ε)= 1} foreach u Q do foreach a Σ do δ(u, a) = v Q st. T A (u.a)=t A (v) return A=(Q, Σ, q 0, F, δ) end

19 0 1 q 0 q q Let s L* learn this DFA

20 ε ε Closed: no Consistent: yes ε ε Closed: yes Consistent: yes 0 1 H 0 q 0 q 1 0,1

21 equivoracle(h 0, SUL) = 110 = counterexample ε ε counterexample Not consistent because T A (ε) = T A (11) = 1 but T A (1) T A (111) since 0 1 Also T A (0) T A (110) since 0 1 Closed: yes Consistent: no

22 equivoracle(h 0, SUL) = 110 = counterexample ε ε ε 0 1 ε Closed: yes Consistent: no Closed: yes Consistent: yes H 1 = FINISHED!

23 Theore:cal Complexity Membership queries = O(m Σ Q 2 ) where m is the maximum length of any counterexample (Angluin 87). Assume oracle returns shortest counterexample then Q m, so Membership queries = O( Σ Q 3 )

24 Essentially quadratic Essentially linear

25

26 5. Conclusions L* is pedagogically easy to learn some of its principles are universal looks promising on paper emphasis on complete learning LBT needs incremental learning Open Ques:ons How complex are real SUTs? Can try to benchmark other learning algorithms in same framework