A brain-inspired neuromorphic architecture for robust neural computation

A brain-inspired neuromorphic architecture for robust neural computation Fabio Stefanini and Giacomo Indiveri Institute of Neuroinformatics University of Zurich and ETH Zurich BIC Workshop @ ISCA40 June 24th, 2013

State-of-art Neuromorphic and neurocomputing systems SW simulated neural networks (CPUs, GPUs). Fully digital cognitive computing chips (IBM). Fabio Stefanini (INI) Real-time ARM-based neural network simulator (SpiNNaker). Real-time neuromorphic multi-chip emulator (NeuroGrid). Neuromorphic classifiers Wafer-scale analog neural accelerators (BrainScaleS). FPGA, Analog/digital CMOS, 3D VLSI, ReRAM, memritors, asynchronous,... 1 / 16

The neuromorphic approach 100µ [Nuno da Costa, INI, 2008] To exploit the physics of silicon to reproduce the bio-physics of neural systems, using subthreshold analog VLSI circuits. To develop multi-chip spike-based computing systems using asynchronous digital VLSI circuits to encode and transmit signals between computational nodes. To employ these technologies to understand neural computation and to build behaving systems able to carry out behavioral tasks in complex environments, in real time. Fabio Stefanini (INI) Neuromorphic classifiers 2 / 16

Neuromorphic circuits Silicon synapses Vdd Vdd Vdd V τ I τ C syn I syn (t) = I 0 e κ U T (V syn (t) V dd ) V w0 M S1 V std M S2 M Vdd D5 V thr M M D4 D1 V w M D2 C std V nmda M N1 M D6 M N2 I thr = I 0 e κ(v thr V dd ) U T C syn d dt V syn = (I in I τ ) I w V in V G V in M S3 M D3 M G1 M G2 I syn V mem τ d dt I syn + I syn = I thr I w I τ The diff-pair integrator (DPI) circuit [Bartolozzi and Indiveri, Neural Computation, 2007] Fabio Stefanini (INI) Neuromorphic classifiers 3 / 16

Neuromorphic basic architectures Looking for traces of cognition. It emulates the physics of the neural systems (distributed, inhomogeneous, imprecise, and noisy) Vision sensor Multi-Neuron Chips Memory and computation are co-localized and distributed (highly parallel). Time represents itself (asynchronous). github.com/inincs/pyncs (Python in Neuroscience II @ Frontiers in Neuroinformatics) Fabio Stefanini (INI) Neuromorphic classifiers 4 / 16

Neuromorphic basic architectures Looking for traces of cognition. Vision sensor Multi-Neuron Chips How do you do computation using this computing substrate? github.com/inincs/pyncs (Python in Neuroscience II @ Frontiers in Neuroinformatics) Fabio Stefanini (INI) Neuromorphic classifiers 4 / 16

From neuromorphic components to neuromorphic systems Looking for traces of cognition. Top-down design model optimize communication calibrate use theoretical framework (e.g., NEF) Bottom-up emulate biology (system-level) enable learning multi-disciplinary approach use neuromorphic engineering Eliasmith and Anderson (2004), Eliasmith et al. (2012), Rigotti et al. (2013), Chicca et al. (submitted) Fabio Stefanini (INI) Neuromorphic classifiers 5 / 16

A VLSI binary classifier for complex patterns Combining theory, neuroscience and engineering: a win-win approach Why RCN? simple implementation powerful transformation neuroscientific valence D. J. Amit and S. Fusi, 1994, V. N. Vapnik, 1998, Y. Cho and L.K. Saul, 2010 J. Brader et al., 2007, M. Barak et al., 2013 M. Rigotti et al., 2013 J. Tapson et al., 2013 Fabio Stefanini (INI) Neuromorphic classifiers 6 / 16

Spike-driven learning-rule in analog VLSI Pre-synaptic component Post-synaptic component ~pre V whi V UP I&F Circuit V spk Diff-pair Integrator V Ca V UP AER input spike V wth V DN V Wi Diff-pair Intergator I syn V mem V mem Current Comparator V wlow V ilk V mth Comparator V cmp V DN pre V mem Fabio Stefanini (INI) Neuromorphic classifiers 7 / 16

Stochastic weight update LTP/LTD probabilities and stop-learning No LTD LTD consolidation V H θ V L w V mem V H θ V L w V mem pre pre 0 0.1 0.2 0.3 0.4 Time(s) 0 0.1 0.2 0.3 0.4 Time(s) Fabio Stefanini (INI) Neuromorphic classifiers 8 / 16

Training procedure Learning to classify random patterns 0.5 0.4 probability 0.3 0.2 0.1 0 0 50 100 150 200 Post Frequency Fabio Stefanini (INI) Neuromorphic classifiers 9 / 16

Synaptic plasticity on VLSI Theory Fabio Stefanini (INI) Neuromorphic classifiers 10 / 16

Synaptic plasticity on VLSI Theory VLSI hardware Fabio Stefanini (INI) Neuromorphic classifiers 10 / 16

Aggregating classifiers Exploiting stochastic learning and variability (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 11 / 16

Effects of mismatch on classification Using the software simulation to validate the system s robustness Fabio Stefanini (INI) Neuromorphic classifiers 12 / 16

Results and comparisons MNIST deep/cnn (Hinton et al. 2012) 98.4% random + bistable synapses 85% random + bistable synapses + (mod. protocol) 96% Fabio Stefanini (INI) Neuromorphic classifiers 13 / 16

Results and comparisons MNIST AUDIO deep/cnn (Hinton et al. 2012) 98.4% random + bistable synapses 85% random + bistable synapses + (mod. protocol) 96% deep/cnn (Hinton et al. 2012) (TIMIT) 77% VLSI cochlea + bistable synapses (spoken digits) 60% Fabio Stefanini (INI) Neuromorphic classifiers 13 / 16

Neuromorphic Cognitive Systems Putting together all the pieces of the puzzle Additional pieces Event-based sensors (silicon retinas, silicon cochleas, etc.) [Liu and Delbruck, Current Opinion in Neurobiology, 20, 2010] Stochastic, spike-based learning [Mitra et al., IEEE TBCAS, 2009] Probabilistic computing and Bayesian inference [Corneil et al, ISCAS 2012; Pecevski et al., PLOS, 2011] Memristive circuits for biophysically realistic synapses [Nanotechnology 2013, (submitted)] Outlook Developing circuits and devices able to compute using these principles Building (small-scale?) autonomous, real-time behaving systems Fabio Stefanini (INI) Neuromorphic classifiers 14 / 16

Conclusions Toward neuromorphic cognitive behaving systems 1 Neuro-computing has a long tradition and history of successes (and failures). 2 We are witnessing a renaissance of neuro-computing approaches, with some very promising ones. 3 Experimental and computational neuroscience are providing useful insights for the bottom-up design and construction of neuromorphic cognitive systems. 4 By combining engineering, neuroscience, theory we obtained powerful classification. Fabio Stefanini (INI) Neuromorphic classifiers 15 / 16

Acknowledgments Rodney Douglas @ INI Giacomo Indiveri @ INI Sadique Sheik @ INI Emre Neftci @ UCSD Mattia Rigotti @ NYU, NYC (NY) Stefano Fusi @ Columbia University, NYC (NY) Funding sources neurop (257219) ERC SCANDLE (ICT-231168) SoundRec (119973) SNF Fabio Stefanini (INI) Neuromorphic classifiers 16 / 16

Distributed stochastic learning Pros We are not fighting against technology: + mismatch and variability are our friends + systems are fault-tolerant by construction Cons - Learning is slow: only a fraction of the synapses memorize the input patterns - Circuits are BIG - Classifiers are weak (noisy, imprecise) Fabio Stefanini (INI) Neuromorphic classifiers 17 / 16

Fusi s spike-based learning rule [Brader et al., 2007] Fabio Stefanini (INI) Neuromorphic classifiers 18 / 16

Neuromorphic circuits Silicon neurons Vdd Vdd Vdd Vdd M G1 M A5 M A1 M R1 /ACK Vdd I in V ahp M G2 V thrahp M G3 M G4 I a M A6 M R2 /REQ V thr M A2 M R3 M L1 M L2 C mem I g I r M A3 M R4 C R V lk M L3 M G6 C P V lkahp M G5 M R6 M A4 I mem V ref M R5 τ d dt I mem + I mem I gi in I τ + f (I mem ) A low-power adaptive exponential I&F neuron [Indiveri et al., ISCAS 2010] Fabio Stefanini (INI) Neuromorphic classifiers 19 / 16

Neuromorphic circuits Biologically plausible response properties Fabio Stefanini (INI) Neuromorphic classifiers 20 / 16

Stochastic spike-based learning Recipe for efficient STDP in VLSI (more good news) 1 Stochasticity & inhomogeneity: change the state of only a randomly selected subset of synapses. 2 Bistability: it is sufficient to have just two synaptic states (n = 2). 3 Redundancy: implement many imprecise synapses (rather than fewer precise ones). 4 Spike-based: use Poisson distributed spike trains. Represent variables with mean firing rate. Use variability in precise timing of spikes as source of stochasticity. Fabio Stefanini (INI) Neuromorphic classifiers 21 / 16

AER for large mixed selectivity C. Bartolozzi and G. Indiveri, 2007 M. Barak et al., 2013 J. Tapson et al., 2013 Fabio Stefanini (INI) Neuromorphic classifiers 22 / 16

Aggregating classifiers (R. E. Shapire, Y. Freund, 2012), (Breiman, 1996), (W. Senn and S. Fusi, 2005) Fabio Stefanini (INI) Neuromorphic classifiers 23 / 16 ove