Probabilistic and Approximate Computing. Dr. Laura Monroe

Transcription

1 Probabilistic and Approximate Computing Dr. Laura Monroe Ultrascale Systems Research Center Los Alamos National Laboratory Rebooting Computing Summit 4 Washington, DC December 10, 2015

2 Probabilistic Computing A non-deterministic approach to computation Can be probabilistic hardware or software (or both) Taking a wide view, and including probability calculations Both a challenge and an opportunity A challenge because this is coming, in the late-cmos and post-cmos time frames A challenge because this is a fundamental change in thinking about computing An opportunity, since we may be able to represent and solve problems not easily computable in other ways

3 Probabilistic Computing Motivations State of the Art Probabilistic vs. Approximate Research Challenges

4 Why Probabilistic? Emerging hardware directions Power limitations Resilience Performance gains? New approaches to compute

5 Motivation: Futures Physical limits relating to feature size As feature size is reduced to a certain point, deterministic computing will become impossible Because soft error susceptibility increases Vendor roadmaps (Near-threshold voltage) Also sub-threshold? Some savings, but we do hit a wall there

6 Motivation: Power Savings Current probabilistic chip designs are able to reduce power Rice PCMOS prototype has shown 7x performance improvement on image processing using 30x less power Lyric s error correction chip has shown 10x reduced power consumption over conventional chips DARPA UPSIDE is showing multiple orders of magnitude improvements available to computations that can run effectively on probabilistic hardware Attention to both HW and SW is needed This is a true co-design problem

7 Motivation: Reliability FUTURES: We expect more faults in the future (feature size, near-threshold voltage) We are seeing them more and more today POWER: Some of these features (such as nearthreshold voltage) should lead to power savings RESILIENCE: Some probabilistic approaches can be seen as an approach to calculate correctly in the face of faults

8 Motivation: Resilience Field observations show many errors caused by bitflips to data values We have studied faults in DRAM, SRAM and GPGPUs on supercomputers at LANL, NERSC, and ORNL. Vilas Sridharan, Nathan DeBardeleben, Sean Blanchard, Kurt B. Ferreira, Jon Stearley, John Shalf, and Sudhanva Gurumurthi Memory Errors in Modern Systems: The Good, The Bad, and The Ugly. SIGARCH Comput. Archit. News 43, 1 (March 2015), We have found a relatively large number of single bitflips LANL ~0.3 correctable errors / min NERSC ~1 correctable error / min ORNL ~1.4 correctable errors / min NCSA (DSN2014) ~4.2 correctable errors / min ECC prevents these errors from being seen at the application level BUT -- Error protection can be costly Raw hardware cost (server grade components) Performance reduction, memory capacity reduction Power Current ECC corrects single-bit errors, but we see multiple-bit errors too

9 Motivation: New Technical Areas In existence now: Early prototypical uses include such things as error-correcting code memory and image processing. Integer math, needs exact answers but can be made to work probabilistically Possible uses later: Problems that do not require exact answers, in particular, those problems with real-time constraints. This may include perceptual computation or social networking. Power-sensitive compute fields, such as mobile or satellite-based computing. Fields already making use of probabilistic computation, such as machine learning. Certain special-purpose tasks special-purpose probabilistic co-processors can be designed to suit a given problem, which could then be executed in hardware with performance benefit, in the same way GPUs are used as graphics coprocessors. Real-world problems that are probabilistic in nature, such as social interactions. Crossover with many future compute areas: Biological computing, neuro-mimetic computing, bio-systems, clusters-on-chips, social computing

10 Motivation: New Compute Paradigms Probabilistic computing is a novel computational paradigm It isn t new -- dates back to the early 50s -- but hasn t been explored as much as deterministic Applies to brain-inspired Neurons fire in a probabilistic manner Applies to social compute or compute based on largescale populations This tends to be inherently statistical and demands a good enough answer, not the correct answer Applies to biological and/or analog There is inherent measurement error in any analog scheme It is underexploited! What forms of compute might we do once released from the need for exactness and determinism? How do we as humans compute? It isn t especially deterministic.

11 Why Now? Reliability and power Both are becoming pressing problems Both are exascale problems, but post-exascale, both get worse Emerging technologies require probabilism New problems may be approached We have much existing groundwork Both late-cmos and post-cmos time frames They considered probabilistic computation in the 50s, for the same reasons. Von Neumann, Probabilistic logics and the synthesis of reliable organisms from unreliable components, "Automata studies," edited by C. E. Shannon and J. McCarthy, Princeton University Press, 1956, pp Mostly reliability, also for new problems They moved from less reliable vacuum tubes to more reliable transistors We are moving in the other direction

12 Different Way to Think About Compute You aren t always going to get the right answer When do you care and why How do you know when the answer is good enough You won t always be able to exactly reproduce your results The computer is not perfect Messy reality vs. very clean compute model This is disruptive

13 We Aren t Starting from Scratch Automata Studies Shannon, C.E., McCarthy, J., Automata studies, Annals of Mathematics Studies No. 34, Princeton University Press, Princeton Probabilistic Turing Machines With complexity classes Probabilistic Polynomial PP Runs on a probabilistic Turing machine in polynomial time Randomized Polynomial RP Runs on a probabilistic Turing machine in polynomial time If answer is NO, returns NO; if answer is YES, returns YES with prob > ½ Bounded-error Probabilistic Polynomial BPP Runs on a probabilistic Turing machine in polynomial time Gives wrong answer with probability p, 0<p<½ P = BPP? An open question Are there problems that can be solved using a probabilistic Turing machine in polynomial time, that cannot be solved on a Turing machine in poly time?

14 Hardware Intentional Probabilistic CMOS (PCMOS) (Rice) Prototype used for image processing. This application, being perceptual, does not demand perfect accuracy Krishna Palem and Avinash Lingamneni Ten Years of Building Broken Chips: The Physics and Engineering of Inexact Computing. ACM Trans. Embed. Comput. Syst. 12, 2s, Article 87 (May 2013). Biased Voltage Scaling (BIVOS) Protects more significant bits, directs error to less significant bits. In effect, provides variable bit length J. George, B. Marr, B. Akgul, and K. Palem, Probabilistic arithmetic and energy efficient embedded signal processing, in Proc. of the IEEE/ACM Intl. Conf. on Compilers, Architecture, and Synthesis for Embedded Systems, 2006, pp Lyric chip Was used for ECC, aiming at a general purpose probability calculating chip Unintentional Error-prone hardware? Can the errors be quantified? Non-determinism introduced by small feature size

15 Software Takes input including a source of random numbers Or makes random choices during execution Examples Monte Carlo methods Sorting and searching a la Google Graph algorithms Big Data problems True random number generation via quantum noise in flash memory Wang Y., Yu, W., Wu, S., Malysa, G., Suh, G., Kan, E., Flash Memory for Ubiquitous Hardware Security Functions: True Random Number Generation and Device Fingerprints, IEEE Symposium on Security and Privacy, 2012.

16 Probabilistic and Approximate Computing Probabilistic - Accuracy Leverages the intrinsic probabilistic behavior of the underlying circuit fabric Or the stochastic behavior of a binary switch under the influence of thermal noise and other disturbances And probabilistic algorithms on deterministic or non-deterministic hardware One version: Uses random binary bit streams implemented in series and in time and computation is performed by applying gates to the stream and measuring its statistics Approximate - Precision Employs deterministic hardware designs that produce imprecise results and achieves energy efficiency by leveraging the statistical properties of the data or algorithms

17 Accuracy vs. Precision Accurate and precise Precise but not accurate Accurate but not precise Neither accurate nor precise

18 Approximate Calculation of π 1/ (1+ w) =1 w + w 2 w / (1+ x 2 ) =1 x 2 + x 4 x y 1/ (1+ x 2 ) = arctan y = y y y5 5 y Substitute y = 1: π/4 = 1-1/3 + 1/5-1/ Get sequence for π: 4, , , , , , , , , , etc. Always the same approximate answer after the same number of summands.

19 Probabilistic Calculation of π The ratio of the dark circle to the light square is π/4

20 Probabilistic Calculation of π The ratio of the dark circle to the light square is π/4 Randomly insert points

21 Probabilistic Calculation of π The ratio of the dark circle to the light square is π/4 Randomly insert points

22 Probabilistic Calculation of π The ratio of the dark circle to the light square is π/4 Randomly insert points As the number of points increases, the ratio of points inside the circle to total points also approaches π/4

23 Probabilistic Calculation of π The ratio of the dark circle to the light square is π/4 Randomly insert points As the number of points increases, the ratio of points inside the circle to total points also approaches π/4 100,000 runs, different every time: , , , , , ,

24 Probabilistic vs. Approximate Calculation Probabilistic Approximate Different answer every time Always the same answer after the same number of iterations Non-deterministic Accurate Can give power savings, better resilience Deterministic Precise Can give power savings, better resilience

25 Some Domain Examples Image processing HW, SW Sort and search - SW Error-correcting codes HW, SW Integer arithmetic SW, hopefully HW soon Bayesian inference machine probability calculations with nano hardware support Direct mapping of the calculations to analog hardware These are things that have been or are being done. Q: What other domains might benefit from this?

26 Research Challenges Across the Stack Paradigm change major shift in thought But not really so far off of the scientific method More of a change in how we think of computers Mapping problems to probabilistic methods Similar to the move from serial to parallel models Applications and Tools e.g., debugging! Programming languages Programming models Software stack Standards Architecture Power, resilience, performance Device level Understanding of fault models Mathematical mapping to physical devices

27 Big questions Is it in principle better than conventional for certain problems? (P=BPP) What is the meaning of a right answer? How best to map problems to probabilistic computation techniques? Heuristics or patterns for probabilistic computation? How best to write apps? How to understand the device itself? How to model a computer build on a given device? How can probabilistic computing best target problems in national security and open science?

28 Takeaways Probabilistic hardware is coming. Applies to many new paradigms, is cross-cutting. Lots of benefits. Lots of new opportunities. But a different way of thinking about compute, so Lots of interesting questions to explore.