Efficient Sampling of SAT Solutions for Testing

Transcription

1 Efficient Sampling of SAT Solutions for Testing Rafael Dutra, Kevin Laeufer, Jonathan Bachrach, Koushik Sen EECS Department UC Berkeley International Conference in Software Engineering May 31,

2 Problem definition Input: Boolean constraint (SAT formula) (x1 x4) (x1 x3 x8) (x1 x8 x6) (x2 x5) ( x7 x3 x9) ( x7 x8 x9) (x7 x8 x10) (x7 x10 x6) Goal: Quickly generate lots of solutions that satisfy the constraint σ 1 σ 2 σ 3 σ 4 x1 x2 x3 x4 x5 x6 x7 x8 x9 x σ

3 Motivation Thoroughly exercising some target functionality Constrained-Random Verification Symbolic execution void main(int4 x, int4 y) { if ((x & y) == 4) { if (x < y) { interesting(x, y); } } } (x & y) == 4 x < y 3

4 Example void main(int4 x, int4 y) { if ((x & y) == 4) { if (x < y) { interesting(x, y); } } } 4

5 Example void main(int4 x, int4 y) { if ((x & y) == 4) { if (x < y) { interesting(x, y); } } } Path Constraint ((x & y) == 4) (x < y) 5

6 Example void main(int4 x, int4 y) { if ((x & y) == 4) { if (x < y) { interesting(x, y); } } } int4 x; x3 x2 x1 x0 int4 y; y3 y2 y1 y0 Path Constraint ((x & y) == 4) (x < y) Bit-blast 6

7 Example void main(int4 x, int4 y) { if ((x & y) == 4) { if (x < y) { interesting(x, y); } } } int4 x; x3 x2 x1 x0 int4 y; y3 y2 y1 y0 Path Constraint ((x & y) == 4) (x < y) Bit-blast Boolean formula φ(x0,x1,x2,x3,y0,y1,y2,y3) ( x0 y0) ( x1 y1) x2 y2 ( x3 y3) ( (x3 y3) ( (x3=y3 x2 y2) ( (x3=y3 x2=y2 x1 y1) ( (x3=y3 x2=y2 x1=y1 x0 y0) ) 7

8 QuickSampler 8

9 QuickSampler Our goals: Samples should satisfy the formula >50% of the times Generate samples >100x faster than other techniques Sampling should be close to uniform Our approach: Compute patterns of bit flips which preserve satisfiability Combine those bit flip patterns to generate lots of samples 9

10 QuickSampler Algorithm Random assignment Start by generating a random assignment σ σ 10

11 QuickSampler Algorithm Random assignment Base solution Start by generating a random assignment σ Use to find the closest solution σ to σ σ σ 11

12 QuickSampler Algorithm Random assignment Base solution Closest solutions Start by generating a random assignment σ Use to find the closest solution σ to σ From σ, use to find the closest solutions that flip some bits (such as and σ 1 ) σ σ σ 1 12

13 QuickSampler Algorithm Start by generating a random assignment σ Use to find the closest solution σ to σ From σ, use to find the closest solutions that flip some bits (such as and σ 1 ) Combine those mutation to generate new samples (such as 1 ) Random assignment Base solution Closest solutions Generated samples σ σ σ

14 14

15 Random assignment σ

16 Random assignment σ Solution

17 Random assignment σ Solution

18 Random assignment σ Solution σ

19 Random assignment σ Solution σ

20 Random assignment σ Solution σ σ 1 20

21 Random assignment σ Solution σ σ 1 21

22 Random assignment σ Solution σ σ 1 22

23 Random assignment σ Solution σ UNSAT σ

24 Random assignment σ Solution σ UNSAT σ

25 Random assignment σ Solution σ UNSAT σ 3 25

26 Random assignment σ Solution UNSAT σ 3 26

27 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 = σ = σ σ 1 = σ σ 3 27

28 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ = δ 0 δ 1 28

29 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ = σ δ 01 29

30 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ = δ 0 δ 3 30

31 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ = σ δ 03 31

32 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ δ = δ 1 δ 3 32

33 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ δ = δ 1 δ 3 33

34 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ δ = δ 0 δ 1 δ 3 34

35 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ δ = δ 0 δ 1 δ 3 35

36 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ

37 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ At most n calls to generate atomic mutations ( ) n 6 =O(n 6 ) samples by combining mutations: NO 37

38 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ At most 50 calls to generate atomic mutations samples by combining mutations: NO 38

39 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ

40 Random assignment σ Solution UNSAT σ 3 δ δ δ 3 δ δ

41 Implementation Implemented in C++ using Z3 as the constraint solver Optimizations: Eager generation of samples Independent support Unsatisfiable variables 41

42 Experiments We evaluated QuickSampler on 163 industrial benchmarks. Largest 6 Benchmarks # Variables # Clauses tutorial3.sk_4_ diagstencilclean.sk_41_ karatsuba.sk_7_ enqueueseqsk.sk_10_ sk_1_ sk_3_

43 Experiments We compared QuickSampler against two state-of-the-art samplers: UniGen2 [1] Uses universal hashing to partition the solution space and produce provably uniform samples SearchTreeSampler [2] Generates pseudo-solutions: partial assignments that can be completed to full solutions [1] Supratik Chakraborty, Daniel J Fremont, Kuldeep S Meel, Sanjit A Seshia, and Moshe Y Vardi On Parallel Scalable Uniform SAT Witness Generation. In TACAS [2] Stefano Ermon, Carla P Gomes, and Bart Selman Uniform solution sampling using a constraint solver as an oracle. In UAI

44 Experiments: Correctness # of Atomic Mutations combined Average Samples generated % of Valid Samples % % % % % % % Total % 44

45 QuickSampler generates valid solutions ±0.8 times faster than SearchTreeSampler ±1.0 times faster than UniGen2 QuickSampler generates unique valid solutions ±0.7 times faster than SearchTreeSampler ±1.1 times faster than UniGen2 45

46 Experiments: Speed Number of valid solutions per time Higher is better 46

47 Experiments: Unique Solutions Number of unique valid solutions per time Higher is better 47

48 Experiments: Unique Solutions Higher is better 48

49 Experiments: Uniformity 49

50 Conclusion Generating lots of solutions efficiently given a SAT formula QuickSampler can generate millions of samples with tens of solver calls The samples satisfy the formula with high probability and are reasonably uniform δ 0 Solution δ δ 0 δ 1 = δ σ δ 01 = σ 1 50