A brain-inspired neuromorphic architecture for robust neural computation

Size: px

Start display at page:

Download "A brain-inspired neuromorphic architecture for robust neural computation"

Carmel Beasley
6 years ago
Views:

Indiveri Institute of Neuroinformatics University of

1 A brain-inspired neuromorphic architecture for robust neural computation Fabio Stefanini and Giacomo Indiveri Institute of Neuroinformatics University of Zurich and ETH Zurich BIC ISCA40 June 24th, 2013

2 State-of-art Neuromorphic and neurocomputing systems SW simulated neural networks (CPUs, GPUs). Fully digital cognitive computing chips (IBM). Fabio Stefanini (INI) Real-time ARM-based neural network simulator (SpiNNaker). Real-time neuromorphic multi-chip emulator (NeuroGrid). Neuromorphic classifiers Wafer-scale analog neural accelerators (BrainScaleS). FPGA, Analog/digital CMOS, 3D VLSI, ReRAM, memritors, asynchronous,... 1 / 16

The neuromorphic approach 100µ [Nuno da Costa, INI, 2008] To exploit the physics of silicon

To develop multi-chip spike-based computing systems using asynchronous digital VLSI circuits

To employ these technologies to understand neural computation and to build behaving systems

3 The neuromorphic approach 100µ [Nuno da Costa, INI, 2008] To exploit the physics of silicon to reproduce the bio-physics of neural systems, using subthreshold analog VLSI circuits. To develop multi-chip spike-based computing systems using asynchronous digital VLSI circuits to encode and transmit signals between computational nodes. To employ these technologies to understand neural computation and to build behaving systems able to carry out behavioral tasks in complex environments, in real time. Fabio Stefanini (INI) Neuromorphic classifiers 2 / 16

4 The neuromorphic approach 100µ [Nuno da Costa, INI, 2008] To exploit the physics of silicon to reproduce the bio-physics of neural systems, using subthreshold analog VLSI circuits. To develop multi-chip spike-based computing systems using asynchronous digital VLSI circuits to encode and transmit signals between computational nodes. To employ these technologies to understand neural computation and to build behaving systems able to carry out behavioral tasks in complex environments, in real time. Fabio Stefanini (INI) Neuromorphic classifiers 2 / 16

5 Neuromorphic circuits Silicon synapses Vdd Vdd Vdd V τ I τ C syn I syn (t) = I 0 e κ U T (V syn (t) V dd ) V w0 M S1 V std M S2 M Vdd D5 V thr M M D4 D1 V w M D2 C std V nmda M N1 M D6 M N2 I thr = I 0 e κ(v thr V dd ) U T C syn d dt V syn = (I in I τ ) I w V in V G V in M S3 M D3 M G1 M G2 I syn V mem τ d dt I syn + I syn = I thr I w I τ The diff-pair integrator (DPI) circuit [Bartolozzi and Indiveri, Neural Computation, 2007] Fabio Stefanini (INI) Neuromorphic classifiers 3 / 16

6 Neuromorphic basic architectures Looking for traces of cognition. It emulates the physics of the neural systems (distributed, inhomogeneous, imprecise, and noisy) Vision sensor Multi-Neuron Chips Memory and computation are co-localized and distributed (highly parallel). Time represents itself (asynchronous). github.com/inincs/pyncs (Python in Neuroscience Frontiers in Neuroinformatics) Fabio Stefanini (INI) Neuromorphic classifiers 4 / 16

7 Neuromorphic basic architectures Looking for traces of cognition. Vision sensor Multi-Neuron Chips How do you do computation using this computing substrate? github.com/inincs/pyncs (Python in Neuroscience Frontiers in Neuroinformatics) Fabio Stefanini (INI) Neuromorphic classifiers 4 / 16

model optimize communication calibrate use theoretical framework (e.g.

8 From neuromorphic components to neuromorphic systems Looking for traces of cognition. Top-down design model optimize communication calibrate use theoretical framework (e.g., NEF) Bottom-up emulate biology (system-level) enable learning multi-disciplinary approach use neuromorphic engineering Eliasmith and Anderson (2004), Eliasmith et al. (2012), Rigotti et al. (2013), Chicca et al. (submitted) Fabio Stefanini (INI) Neuromorphic classifiers 5 / 16

9 A VLSI binary classifier for complex patterns Combining theory, neuroscience and engineering: a win-win approach Why RCN? simple implementation powerful transformation neuroscientific valence D. J. Amit and S. Fusi, 1994, V. N. Vapnik, 1998, Y. Cho and L.K. Saul, 2010 J. Brader et al., 2007, M. Barak et al., 2013 M. Rigotti et al., 2013 J. Tapson et al., 2013 Fabio Stefanini (INI) Neuromorphic classifiers 6 / 16

10 Spike-driven learning-rule in analog VLSI Pre-synaptic component Post-synaptic component ~pre V whi V UP I&F Circuit V spk Diff-pair Integrator V Ca V UP AER input spike V wth V DN V Wi Diff-pair Intergator I syn V mem V mem Current Comparator V wlow V ilk V mth Comparator V cmp V DN pre V mem Fabio Stefanini (INI) Neuromorphic classifiers 7 / 16

11 Stochastic weight update LTP/LTD probabilities and stop-learning No LTD LTD consolidation V H θ V L w V mem V H θ V L w V mem pre pre Time(s) Time(s) Fabio Stefanini (INI) Neuromorphic classifiers 8 / 16

12 Training procedure Learning to classify random patterns probability Post Frequency Fabio Stefanini (INI) Neuromorphic classifiers 9 / 16

13 Training procedure Learning to classify random patterns probability Post Frequency Fabio Stefanini (INI) Neuromorphic classifiers 9 / 16

Training procedure Learning to classify random patterns 0.5 0.4 probability 0.3 0.2 0.

14 Training procedure Learning to classify random patterns probability Post Frequency Fabio Stefanini (INI) Neuromorphic classifiers 9 / 16

15 Synaptic plasticity on VLSI Theory Fabio Stefanini (INI) Neuromorphic classifiers 10 / 16

16 Synaptic plasticity on VLSI Theory VLSI hardware Fabio Stefanini (INI) Neuromorphic classifiers 10 / 16

17 Synaptic plasticity on VLSI Theory VLSI hardware Fabio Stefanini (INI) Neuromorphic classifiers 10 / 16

18 Aggregating classifiers Exploiting stochastic learning and variability (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 11 / 16

19 Aggregating classifiers Exploiting stochastic learning and variability (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 11 / 16

20 Aggregating classifiers Exploiting stochastic learning and variability (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 11 / 16

21 Aggregating classifiers Exploiting stochastic learning and variability (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 11 / 16

22 Effects of mismatch on classification Using the software simulation to validate the system s robustness Fabio Stefanini (INI) Neuromorphic classifiers 12 / 16

23 Results and comparisons MNIST deep/cnn (Hinton et al. 2012) 98.4% random + bistable synapses 85% random + bistable synapses + (mod. protocol) 96% Fabio Stefanini (INI) Neuromorphic classifiers 13 / 16

24 Results and comparisons MNIST AUDIO deep/cnn (Hinton et al. 2012) 98.4% random + bistable synapses 85% random + bistable synapses + (mod. protocol) 96% deep/cnn (Hinton et al. 2012) (TIMIT) 77% VLSI cochlea + bistable synapses (spoken digits) 60% Fabio Stefanini (INI) Neuromorphic classifiers 13 / 16

25 Neuromorphic Cognitive Systems Putting together all the pieces of the puzzle Additional pieces Event-based sensors (silicon retinas, silicon cochleas, etc.) [Liu and Delbruck, Current Opinion in Neurobiology, 20, 2010] Stochastic, spike-based learning [Mitra et al., IEEE TBCAS, 2009] Probabilistic computing and Bayesian inference [Corneil et al, ISCAS 2012; Pecevski et al., PLOS, 2011] Memristive circuits for biophysically realistic synapses [Nanotechnology 2013, (submitted)] Outlook Developing circuits and devices able to compute using these principles Building (small-scale?) autonomous, real-time behaving systems Fabio Stefanini (INI) Neuromorphic classifiers 14 / 16

26 Conclusions Toward neuromorphic cognitive behaving systems 1 Neuro-computing has a long tradition and history of successes (and failures). 2 We are witnessing a renaissance of neuro-computing approaches, with some very promising ones. 3 Experimental and computational neuroscience are providing useful insights for the bottom-up design and construction of neuromorphic cognitive systems. 4 By combining engineering, neuroscience, theory we obtained powerful classification. Fabio Stefanini (INI) Neuromorphic classifiers 15 / 16

27 Acknowledgments Rodney INI Giacomo INI Sadique INI Emre UCSD Mattia NYU, NYC (NY) Stefano Columbia University, NYC (NY) Funding sources neurop (257219) ERC SCANDLE (ICT ) SoundRec (119973) SNF Fabio Stefanini (INI) Neuromorphic classifiers 16 / 16

$is slow: only a fraction of the synapses memorize the input patterns - Circuits are BIG - Classifiers are weak (noisy, imprecise) Fabio Stefanini (INI) Neuromorphic classifiers 17 /$

28 Distributed stochastic learning Pros We are not fighting against technology: + mismatch and variability are our friends + systems are fault-tolerant by construction Cons - Learning is slow: only a fraction of the synapses memorize the input patterns - Circuits are BIG - Classifiers are weak (noisy, imprecise) Fabio Stefanini (INI) Neuromorphic classifiers 17 / 16

29 Fusi s spike-based learning rule [Brader et al., 2007] Fabio Stefanini (INI) Neuromorphic classifiers 18 / 16

Neuromorphic circuits Silicon neurons Vdd Vdd Vdd Vdd M G1 M A5 M A1 M R1 /ACK Vdd I in V ahp M G2 V thrahp M G3 M G4 I a M A6 M R2 /REQ V thr M A2 M R3 M L1 M L2 C mem I g I r M A3 M R4 C R V lk M

30 Neuromorphic circuits Silicon neurons Vdd Vdd Vdd Vdd M G1 M A5 M A1 M R1 /ACK Vdd I in V ahp M G2 V thrahp M G3 M G4 I a M A6 M R2 /REQ V thr M A2 M R3 M L1 M L2 C mem I g I r M A3 M R4 C R V lk M L3 M G6 C P V lkahp M G5 M R6 M A4 I mem V ref M R5 τ d dt I mem + I mem I gi in I τ + f (I mem ) A low-power adaptive exponential I&F neuron [Indiveri et al., ISCAS 2010] Fabio Stefanini (INI) Neuromorphic classifiers 19 / 16

31 Neuromorphic circuits Biologically plausible response properties Fabio Stefanini (INI) Neuromorphic classifiers 20 / 16

32 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

33 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

34 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

35 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

36 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

37 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

38 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

39 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

40 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

41 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

42 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. 2 Bistability: it is sufficient to have just two synaptic states (n = 2). Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

43 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. 2 Bistability: it is sufficient to have just two synaptic states (n = 2). 3 Redundancy: implement many imprecise synapses (rather than fewer precise ones). Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

44 Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. 2 Bistability: it is sufficient to have just two synaptic states (n = 2). 3 Redundancy: implement many imprecise synapses (rather than fewer precise ones). 4 Spike-based: use Poisson distributed spike trains. Represent variables with mean firing rate. Use variability in precise timing of spikes as source of stochasticity. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

45 AER for large mixed selectivity C. Bartolozzi and G. Indiveri, 2007 M. Barak et al., 2013 J. Tapson et al., 2013 Fabio Stefanini (INI) Neuromorphic classifiers 22 / 16

46 Aggregating classifiers (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 23 / 16 ove

Synaptic plasticity in neuromorphic hardware. Stefano Fusi Columbia University

Synaptic plasticity in neuromorphic hardware Stefano Fusi Columbia University The memory problem Several efficient memory models assume that the synaptic dynamic variables are unbounded, or can be modified