High-Contrast Algorithm Behavior Observation, Hypothesis, and Experimental Design

Transcription

1 Summary High-Contrast Algorithm Behavior Observation, Hypothesis, and Eperimental Design Matthias F. Stallmann and Franc Brglez Two competing algorithms: int-dual - all-integer dual simple old idea, only limited success cple - industrial strength optimizer Restricted domain (logic minimization). Etreme differences across instances: time out for one algorithm vs. seconds for the other. Want to profile the instances, or at least eplain the differences. EpCS, 7/6/4, San Diego EpCS, 7/6/4, San Diego Outline a typical problem instance Observation in a limited domain Initial eperiments: design Initial eperiments: results A hypothesis More eperimental evidence Theoretical eplanation Summary and future work algorithms treated as black boes int-dual cple minimize subject to + 4 >= >= >= 4 >= 4 >= 3 4 >= 3 4 constraint matri pictorial representation EpCS, 7/6/4, San Diego 3 EpCS, 7/6/4, San Diego 4

2 A pattern? A pattern: Less pronounced # succ. completions, one hour original + 3 random row/col permutations dedicated fast processor alg N/33 med. mean stdev alg N/33 9 med. 5.9 mean stdev Intdual Intdual cple cple Benchmark: e64.b (logic synthesis) -- a canonical picture Benchmark: rot.b EpCS, 7/6/4, San Diego 5 EpCS, 7/6/4, San Diego 6 A pattern: Not much of one The pattern alg N/33 med. mean stdev Intdual cple Benchmark: ma4 Benchmark: e64.b int-dual Benchmark: ma4 cple EpCS, 7/6/4, San Diego 7 EpCS, 7/6/4, San Diego 8

3 Process for generating pictures this is as good as it gets... crossing minimization may still look random (moves nonzeros close to the diagonal) This is ma4 after crossing minimization EpCS, 7/6/4, San Diego 9 EpCS, 7/6/4, San Diego Conversion to block structure does not improve performance Basic idea crossing minimization int-dual performance: not necessarily improved More underlying structure => better for int-dual Less underlying structure => better for cple e64 as posted: 7.5 after minimization: 55. recall that median = 5.9 underlying block structure random permutations... } better aggregate performance for int-dual Why is this interesting? e.g., optimum for e64.b was unknown prior to int-dual What do we mean by structure? More precise (and testable) hypothesis? EpCS, 7/6/4, San Diego EpCS, 7/6/4, San Diego

4 Outline From observation to controlled eperiment Observation in a limited domain Initial eperiments: design Initial eperiments: results A hypothesis More eperimental evidence Theoretical eplanation Summary and future work Behavior in the wild : algorithms on industrial instances Behavior in captivity : algorithms on carefully designed instances EpCS, 7/6/4, San Diego 3 EpCS, 7/6/4, San Diego 4 Instances with pure blocks Construction of blocks with added rows two pure blocks add new row(s) with random nonzeros below both blocks Create K copies of a small instance, arrange on diagonal, permute rows and columns randomly 3 times to obtain a class of equivalent instances. row factor = # of rows added each step repeat apply same process to new blocks Create 3 instances with different random choices of the nonzeros; randomly permute each one EpCS, 7/6/4, San Diego 5 EpCS, 7/6/4, San Diego 6

5 Blocks used in results reported here st5 - instance based on Steiner triples (5 vars, 35 const) maincont - small logic synthesis instances (6 vars, 5 const) f5m - slightly larger logic synthesis instance (75 vars, 87 const) Outline Observation in a limited domain Initial eperiments: design Initial eperiments: results A hypothesis More eperimental evidence Theoretical eplanation Summary and future work EpCS, 7/6/4, San Diego 7 EpCS, 7/6/4, San Diego 8 Details and reporting results for st5, pure blocks Class of 3 random related instances blocks Intel(R) Xeon(TM) CPU 3.GHz, GB memory, 48 KB cache Report runtime only (seconds, time out = 6) int-dual cple > hours Here we report median only, but note number of successful completions, if not 3 EpCS, 7/6/4, San Diego 9 EpCS, 7/6/4, San Diego

6 results for maincont, pure blocks results for f5m, pure blocks blocks int-dual <. <. 4 < blocks int-dual cple > > > 6 cple <. <. <...5 > 6 9/3 completions > 6 between 4 and 8 blocks int-dual is roughly between O(n^) and O(n^3) cple is wildly eponential EpCS, 7/6/4, San Diego EpCS, 7/6/4, San Diego added rows? results for maincont, added rows 8 blocks with increasing row factor row fact net page... N/3 id >6 5 >6 >6 cple blank = neither dominates, X = neither finishes in hour I = int-dual dominates, C = cple dominates N/ mean = 59.9 stdev = Wild distributions EpCS, 7/6/4, San Diego 3 EpCS, 7/6/4, San Diego 4

7 this is not going well all is not lost: let s stick with what we know not much hope of a statistically meaningful comparison between int-dual and cple. too many variables: will anything hold up not clear what to measure id domain already very limited EpCS, 7/6/4, San Diego >6 >6 >6 int-dual gets worse as more random nonzeros are added (has been observed for columns, too) 5 Outline.... int-dual performs well on pure blocks cple performs badly on pure blocks EpCS, 7/6/4, San Diego 6 A viable hypothesis Observation in a limited domain Initial eperiments: design Initial eperiments: results A hypothesis More eperimental evidence Theoretical eplanation Summary and future work int-dual runtime is predictable on pure-blocks instances and grows polynomially with increasing number of blocks cple runtime on pure-blocks instances becomes erratic as soon as an instance-dependent threshold is reached int-dual runtime increases and/or becomes erratic as the number of nonzeros added to a pure-blocks instance is increased erratic = heavy tail distribution (stdev >> mean >> median) or time outs and overflows EpCS, 7/6/4, San Diego 7 EpCS, 7/6/4, San Diego 8

8 why this is good not dependent on comparison between the algorithms not dependent on statistics etreme contrasts are what we observed initially -- so we need to deal with them appears to be true in a variety of circumstances even though instances are artificial, may eplain why int-dual has limited use Outline Observation in a limited domain Initial eperiments: design Initial eperiments: results A hypothesis More eperimental evidence Theoretical eplanation Summary and future work EpCS, 7/6/4, San Diego 9 EpCS, 7/6/4, San Diego 3 row fact. id cple f5m, added rows 3 blocks, increasing row factor N/ >6 >6 >6 >6 >6 > N/3 distributions that are not erratic Outline Observation in a limited domain Initial eperiments: design Initial eperiments: results A hypothesis More eperimental evidence Theoretical eplanation Summary and future work EpCS, 7/6/4, San Diego 3 EpCS, 7/6/4, San Diego 3

9 All-Integer Dual Simple a pivot step verte cover of a triangle + >= e= 3 min e e e3 tableau 3 3 e e e e3 e3 choose variable to do the trick choose constraint to satisfy e 3 swap them now =, e= EpCS, 7/6/4, San Diego 33 EpCS, 7/6/4, San Diego 34 eample (continued) oops: solution is not integral 3/ / / / / / / / / 3/ / 3/ / / / another pivot... and another, but this one involves division by... and we get fractions. The LP solution is =, =, 3 = EpCS, 7/6/4, San Diego 35 EpCS, 7/6/4, San Diego 36

10 Maintaining an integer tableau int-dual: another look e e3 3 e e3 ct in pivot row means no change in pivot column e ct e 3 add a new constraint that must be met by all integer solutions = 3 =, cost = in pivot column means no change in pivot row EpCS, 7/6/4, San Diego 37 EpCS, 7/6/4, San Diego 38 int-dual on pure blocks why does int-dual work well on pure blocks? element of second block not changed by pivot in first block this phenomenon is not affected by random permutation because each block is solved independently progress is made in whatever block contains the pivot One out of three parts (of the hypothesis) eplained EpCS, 7/6/4, San Diego 39 EpCS, 7/6/4, San Diego 4

11 part : cple does badly on pure blocks cple is a branch and bound algorithm branching tree for one block y tree for two blocks Note: this is a gross oversimplification Summary Some observations about algorithm behavior on industrial benchmarks Hypothesis specializes observations to a limited, verifiable domain Etensive eperiments consistent with hypothesis Partial theoretical eplanation for hypothesis Aside: int-dual dates to 96 s; eperiments in early 97 s deemed it good for small instances of set cover (but impractical otherwise) EpCS, 7/6/4, San Diego 4 EpCS, 7/6/4, San Diego 4 Future work Thanks... better understanding of both algorithms to eplain what s going on pure branch and bound instead of cple eperiments to eplore variations on variables (how to add rows, columns too, add entries) observations in other domains (particular where there has been success with int-dual) other approaches to claiming one algorithm better than another where distributions are crazy can we learn from outliers, i.e., easy and hard instances of a class Xiao Yu Li, now at Amazon, started us down this path Eric Sills, NCSU High Performance Computing Center, helped with hardware and software availability Source code for software and (eventually) instance classes and results at EpCS, 7/6/4, San Diego 43 EpCS, 7/6/4, San Diego 44