What is the Cost of Determinism? Cedomir Segulja, Tarek S. Abdelrahman University of Toronto

Transcription

1 What is the Cost of Determinism? Cedomir Segulja, Tarek S. Abdelrahman University of Toronto

2 Source: [Youtube] [Intel]

3 Non-Determinism Same program + same input same output This is bad for Testing Too many interleaving to test Debugging Hard to debug when behavior is not repeatable Selling CAD tools users expect each run to produce the same circuit

4 Determinism 1 2 Is good, but costly What is the fundamental cost of determinism? What is this cost across various execution environments? Determinism in the field Deterministic Schedulers Maximum Slowdown DMP [Devietti et al. 2009] 1.7x Kendo [Olszewski et al. 2009] 1.6x Grace [Berger et al. 2009] 3.6x CoreDet [Bergan et al. 2010] 10x Calvin [Hower et al. 2011] 1.7x RCDC [Devietti et al. 2011] 1.7x Dthreads [Liu et al. 2011] Conversion [Merrifield and Eriksson 2013] Parrot [Cui et al. 2013] 3.8x RFDet [Lu et al. 2014] 2.6x Source: [Bergan et al. 2011] and the respective papers *Only to show that determinism comes at a cost, and not to be used for a direct comparison (different features, benchmarks, # threads, etc.) 4x 5x

5 What is Determinism? Property that requires observing the same output whenever program runs with the same input SyncOrder determinism [Lu and Scott 11] Require the same program result and same order of synchronization More flexible than internal determinism Still greatly eases testing [Cui et al. 13] We assume data-race-freedom Determinism during debugging is needed But the cost of determinism matters the most in production External SyncOrder Internal All data races are bugs [Boehm 2008, S. Adve 2010, Marino et al. 2010, Lucia et al. 2010, ] Data races in general do not help performance [Boehm 12]

6 What is the impact of enforcing a fixed synchronization order on program execution time?

7 Schedule-Record-Replay Framework 1 2 application schedule application thread 1 thread 2 scheduler serial replayer hybrid round-robin dynamic-a dynamic-s perturber architectures idle small perturbations recorder NUMA DVFS background processes

8 barnes cholesky fft fmm lu_cb lu_ncb ocean_cp ocean_ncp radiosity radix raytrace_splash2x volrend water_nsquared water_spatial blackscholes bodytrack dedup facesim ferret fluidanimate raytrace_parsec streamcluster swaptions vips Normalized Execution Time Replayer Force threads to wait only when absolutely necessary under the schedule And do so with as little overhead as possible Non-deterministic execution vs. Non-deterministic execution with the replayer s overhead

9 Schedules When does a thread pass its turn? At the end serial After each synchronization operation round-robin After each instruction/store dynamic-a/dynamic-s After N instructions hybrid N = 100,000 No reduced serial mode Deterministic Schedulers Grace [Berger et al. 2009] Dthreads [Liu et al. 2011] Conversion [Merrifield and Eriksson 2013] Parrot [Cui et al. 2013] Kendo [Olszewski et al. 2009] RCDC [Devietti et al. 2011] RFDet [Lu et al. 2014] DMP [Devietti et al. 2009] CoreDet [Bergan et al. 2010] Calvin [Hower et al. 2011] Schedule serial round-robin round-robin round-robin dynamic dynamic dynamic hybrid hybrid hybrid

10 Platform 8-core Xeon E SPLASH-2 and PARSEC benchmarks, 8 threads Deterministic slowdown deterministic execution time non deterministic execution time Data races in general do not help performance [Boehm 12] 15 benchmarks had races, performance degradation in only 3 barnes (11%), radiosity (5%), raytrace_parsec (8%)

11 parsec splash Benchmarks serial round-robin dynamic-s dynamic-a hybrid barnes cholesky fft fmm lu_cb lu_ncb ocean_cp ocean_ncp radiosity radix raytrace volrend water_nsquared water_spatial blackscholes bodytrack dedup facesim ferret fluidanimate raytrace streamcluster swaptions vips average slowdown maximum slowdown

12 For this set of benchmarks and our platform, and implementation overhead set aside, the fundamental cost of determinism is small.

13 What is the performance cost of insisting on the same schedule across different environments?

14 Schedule-Record-Perturb-Replay Framework 1 2 application schedule application thread 1 thread 2 scheduler serial replayer hybrid round-robin dynamic-a dynamic-s perturber architectures idle small perturbations recorder NUMA DVFS background processes

15 Perturber Small perturbations (context switches, thread migrations, page faults) Simulate first order effects by inserting small delays (μs and ms) Background processes Spawn additional threads and control their work to sleep ratio Dynamic voltage and frequency scaling (DVFS) Use Linux s cpufreq system to explore different DVFS policies Non-uniform memory access (NUMA) Spread threads over two NUMA nodes Asymmetric architectures Use DVFS to create asymmetry [Shelepov et al. 2009]

16 Metric Deterministic slowdown deterministic execution time non deterministic execution time Same conditions during both runs, for example deterministic execution time with background processes non deterministic execution time with background processes

17 parsec splash Benchmarks Quiet Small perturbations Backgroud proc. DVFS Asym. Arch. NUMA balanced unbalanced balanced unbalanced auto manual 4/4 1/7 barnes cholesky fft fmm lu_cb lu_ncb ocean_cp ocean_ncp radiosity radix raytrace volrend water_nsquared water_spatial blackscholes bodytrack dedup facesim ferret fluidanimate raytrace streamcluster swaptions vips avg. slowdown max. slowdown

18 Insisting on the same schedule in the presence of skewed conditions can slow down execution by a factor of almost 2x.

19 Conclusions Employed the schedule-record-replay framework to divorce implementation overhead from the fundamental cost of enforcing deterministic execution Fundamental cost of determinism is small (4% on avg., 33 % max.) There is room for lowering overheads in current deterministic systems Measured this fundamental cost across a range of execution environments The cost of raises to almost 2x when threads face skewed conditions Do we need a more relaxed definition of determinism? Quantified various sources of non-determinism Deterministic logical clocks are not deterministic (not only due to the performance counters imperfections [Weaver et al. 2013])

20 Thank you!