Constrained Counting and Sampling Bridging Theory and Practice

Transcription

1 Constrained Counting and Sampling Bridging Theory and Practice Supratik Chakraborty, IIT Bombay [ Joint work with Kuldeep S. Meel (NUS, Singapore), Moshe Y. Vardi (Rice University), Rakesh Mistry (currently at Oracle Corp.) ]

2 Constrained Counting and Sampling in IITB How many in IITB like green tea or black coffee? Universe: Residents of IITB Constraint: Must like green tea or black coffee Naïve method: Ask everybody, tabulate and count Not efficiently scalable Can we do better with Sample uniformly randomly from those in IITB who like green tea or black coffee? provable guarantees? Universe: Residents of IITB Constraint: Must like green tea or black coffee Naïve method: Find out all those who like green tea or black coffee, and sample uniformly from them Not efficiently scalable 1

3 An Abstract View All IITB Residents 2

4 Abstract Problem Formulation Given X 1, X n : variables with finite domains D 1, D n Constraint (logical formula) over X 1, X n Weight function W: D 1 D n Q 0 Sol() : set of assignments of X 1, X n satisfying Determine W() = y Sol() W(y) If W(y) = 1 for all y, then W() = Sol() Constrained Discrete Counting Randomly sample from Sol() such that Pr[y is sampled] W(y) If W(y) = 1 for all y, then uniformly sample from Sol() Constrained Discrete Sampling For this talk, D i s are {0,1} Boolean variables 3

5 Application 1: Probabilistic Inference An alarm rings if it s in a working state when an earthquake happens or a burglary happens The alarm can malfunction and ring without earthquake or burglary happening Given that the alarm rang, what is the likelihood that an earthquake happened? Given conditional dependencies (and conditional probabilities) calculate Pr[event evidence] What is Pr [Earthquake Alarm]? Efficiently reduced to constrained weighted counting 4

6 Application 2: Network Reliability s t Graph G = (V, E) represents a (power-grid) network Nodes (V) are towns, villages, power stations Edges (E) are power lines Assume each edge e fails with prob g(e) [0,1] Assume failure of edges statistically independent What is the probability that s and t become disconnected? Efficiently reduced to constrained weighted counting 5

7 Application 3: Quantitative Information Flow A password-checker PC takes a secret password (SP) and a user input (UI) and returns Yes iff SP = UI [Bang et al 2016] Suppose passwords are 4 characters ( 0 through 9 ) long PC1 (char[] SP, char[] UI) { for (int i=0; i<sp.length(); i++) { if(sp[i]!= UI[i]) return No ; } return Yes ; } PC2 (char[] H, char[] L) { match = true; for (int i=0; i<sp.length(); i++) { if (SP[i]!= UI[i]) match=false; else match = match; } if match return Yes ; else return No ; } Which of PC1 and PC2 is more likely to leak information about the secret key through side-channel observations? 6

8 Unweighted Counting Suffices in Principle Probabilistic Inference Network Reliability Quantified Information Flow KML 1989, Karger 2000 DMPV 2017 Weighted Constrained Counting Weighted Constr Counting Unweighted Constr Counting IJCAI 2015 Reduction polynomial in #bits representing weights 7

9 Application 4: Constr Random Verification Functional Verification Formal verification Challenges: formal requirements, scalability ~10-15% of verification effort Dynamic verification: dominant approach No. of test vectors needed exceedingly large Can we sample uniformly from good test vectors 8

10 A Practical Approach Constrained Random Verification a b Set of Constraints on Inputs 64 bit 64 bit c = f(a,b) Logical Formula 64 bit c Sample satisfying assignments uniformly at random Efficiently reduced to constrained sampling 9

11 Application 5: Automated Problem Generation Large class sizes, MOOC offerings require automated generation of related but randomly different problems Randomness makes it hard for students to guess what the solution would be Allows instructors to focus on broad parameters of problems, rather than on individual problem instances Filling question templates with values satisfying constraints Efficiently reduced to constrained sampling 10

12 How Hard is it to Count? Trivial if we could enumerate R F : Almost always impractical Computational complexity of counting (discrete integration): Exact unweighted/weighted counting: #P-complete Approximate unweighted/weighted counting: Randomized: Poly-time prob Turing Machine with NP oracle Pr R F 1 RandEstima te(f,, ) R F (1 ) 1, for 0, 0 1 Probably Approximately Correct (PAC) algorithm 11

13 How Hard is it to Sample? Computational complexity of sampling: Uniform sampling: Poly-time prob. Turing Machine with NP oracle [Bellare,Goldreich,Petrank 2000] Pr[ y UniformGen erator(f)] c, where indep Almost uniform sampling: Poly-time prob. Turing Machine with NP oracle [Jerrum,Valiant,Vazirani 1986, also from Bellare,Goldreich,Petrank 2000] c c 0 0 if and y R F of y if y R F c c 0 if y R F Pr[ y AUGenerato r(f, )] c (1 ), where 1 c 0 and indep of y if y R F Pr[Algorithm outputs some y] ½, if F is satisfiable 12

14 Markov Chain Monte Carlo Techniques Rich body of theoretical work with applications to sampling and counting Convergence to desired distribution guaranteed only after large number of steps In practice, steps truncated early heuristically Nullifies/weakens theoretical guarantees 13

15 Approximate Counting and Sampling: Close Cousins Seminal paper by Jerrum, Valiant, Vazirani 1986 Almost-Uniform Generator Polynomial reduction PAC Counter Yet, no practical algorithms that scale to large problem instances were derived from this work No scalable PAC counter or almost-uniform generator existed until a few years back The inter-reductions are practically computation intensive 14

16 Guarantees Empirical Performance Prior Work BGP BDD/ other exact tech. Theoretical guarantees Our Work Existing approx. techniques MCMC SAT- Based Performance 15

17 A High-Level View of Core Idea in our Work 16

18 Counting Dots Solution to constraints 17

19 Partitioning into equal small cells 18

20 Partitioning into equal small cells Pick a random cell Estimate = # of solutions (dots) in cell * # of cells 19

21 How to Partition? How to partition into roughly equal small cells of solutions without knowing the distribution of solutions? 2-Universal Hashing using parity XOR constraints [Carter-Wegman 1977] 20

22 Partitioning 1. How large is the small cell? 2. How do we compute solutions inside a cell? 3. How many cells? 21

23 Question 1: Size of cell Too large Hard to enumerate Too small Ratio of variance to mean is very high 22

24 Question 2: Solving a cell m XORs 23

25 Question 3: How many cells? (CP 2013) 24

26 #sols < pivot NO 25

27 #sols < pivot NO 26

28 YES #sols < pivot 27

29 28

30 Runtime Performance of ApproxMC 29

31 Time (seconds) Can Solve a Large Class of Problems ApproxMC Cachet Benchmarks Large class of problems that lie beyond the exact algorithms but can be computed by ApproxMC 30

32 Count Mean Error: Only 4% (allowed: 75%) 3.6E E E E E E E E E E+03 Cachet*1.75 Cachet/1.75 ApproxMC 3.2E E Benchmarks Mean error: 4% much smaller than the theoretical guarantee of 75% 31

33 Almost Same Idea Works for Sampling Too! 32

34 Theoretical Guarantees 33

35 Runtime Performance of UniGen 34

36 case47 case_3_b14_3 case105 case8 case203 case145 case61 case9 case15 case140 case_2_b14_1 case_3_b14_1 squaring14 squaring7 case_2_ptb_1 case_1_ptb_1 case_2_b14_2 case_3_b14_2 1-2 Orders of Magnitude Faster Time(s) UniGen XORSample' 0.1 Benchmarks 35

37 Frequency Results: Uniformity #Solutions Benchmark: case110.cnf; #var: 287; #clauses: 1263 Total Runs: 4x10 6 ; Total Solutions :

38 Frequency Results: Uniformity US UniGen #Solutions Benchmark: case110.cnf; #var: 287; #clauses: 1263 Total Runs: 4x10 6 ; Total Solutions :

39 Contribution of Hashing-based Approaches ApproxMC: The first scalable approximate model counter UniGen: The first scalable uniform generator Both scale to constraints with Boolean vars Earlier work (with similar guarantees) scaled only to 100s of vars Outperforms state-of-the-art generators/counters 38

40 Thanks! Questions? 39