Outline Neural dynamics with log-domain integrator circuits Giacomo Indiveri Neuromorphic Cognitive Systems group Institute of Neuroinformatics niversity of Zurich and ETH Zurich Dynamics of Multi-function Brain Networks MRI Winter School January 8, 2014 1 Neuromorphic Engineering 2 Neural dynamics in analog VLSI 3 Real and Silicon Synapses 4 Log-domain translinear circuits G. Indiveri (NCS @ INI) MRI Winter School Tutorial 1 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 2 / 39 Where it began Biophysics of membrane channels Neuromorphic Engineering Engineering meets biology... G. Indiveri (NCS @ INI) MRI Winter School Tutorial 4 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 5 / 39
Neuromorphic Engineering Physics of Computation Neuromorphic Engineering A silicon neuron circuit 1V Vm [Ca] Vm [Ca] Vm [Ca] I 50 ms G. Indiveri (NCS @ INI) MRI Winter School Tutorial 6 / 39 In 1991 Misha Mahowald and Rodney Douglas proposed a G. Indiveri (NCS @ INI) MRI Winter School Tutorial 7 / 39 conductance-based silicon neuron and showed that it had properties remarkably similar to those of real cortical neurons. Custom VLSI implementations of neural networks Early attempts The idea of making custom analog VLSI implementations of neural networks dates back to the late 80 s - early 90s: [Holler et al. 1989, Satyanarayana et al. 1992, Hammerstrom 1993, Vittoz 1996] General purpose computing Full-custom analog implementation Neural network accelerator PC-boards Current research Technological progress Power-dissipation/computational power Competing with Intel steamroller Communication - bandwidth limited Difficult to program Application-specific focus Embedded system integration Silicon neuron designs Many VLSI models of spiking neurons have been developed in the past, and many are still being actively investigated. Most designs can be traced back to one of two types of silicon neurons, proposed by the Carver Mead, Misha Mahowald and Rodney Douglas, in the late 80s - early 90 s: an integrate-and-fire model, and a conductance-based model. Several analog approaches focus on the details of the neuron s action potential mechanism. Many digital approaches focus on optimizing the use of resources for an efficient algorithmic implementation. Few approaches take into account neural dynamics for both neuron and synapse implementations. G. Indiveri (NCS @ INI) MRI Winter School Tutorial 8 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 9 / 39
Why biologically plausible neural dynamics Circuit design options Different from von Neumann architectures Massively parallel arrays of simple processing elements. Memory and computation are co-localized. Time represents itself. Input/data driven. Biologically plausible time constants For interacting with the environment in real-time. To process natural sensory signals efficiently. Neuromorphic systems Well matched to the signals they process Inherently synchronized with the real-world events. Above threshold (strong inversion) Mixed analog/digital Rate-based Real-time Conductance-based Large-scale, event-based networks Below threshold (weak inversion) Fully analog Spiking Accelerated-time Integrate-and-Fire Small-scale, hard-wired G. Indiveri (NCS @ INI) MRI Winter School Tutorial 11 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 12 / 39 MOSFETs in subthreshold Diffusion and saturation where Ids I 0 denotes the nfet current-scaling parameter κ n denotes the nfet subthreshold slope factor the thermal voltage n-fet subthreshold transfer function ( ) I ds = I 0 e κ n / e / e / the gate voltage, the source voltage, and the drain voltage. I r I f I r I f Q E s Q d V I ds = I 0 e κ n / ( e / e / ) is equivalent to: I ds = I 0 e κ Vg Vs I ds = I f I r I 0 e κ Vg If s > 4 the I r term becomes negligible, and the transistor is said to operate in the saturation regime: I ds = I 0 e κ n / / The current is defined to be positive if it flows from the drain to the source G. Indiveri (NCS @ INI) MRI Winter School Tutorial 13 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 14 / 39
Exponential voltage dependence Subthreshold n-fet Ids (A) 10-2 10-3 10-4 10-5 10-6 10-7 subthreshold above threshold 10-8 10-9 10-10 10-11 10-12 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Vgs (V) n-fets and p-fets In Complementary Metal-Oxide Semiconductor (CMOS) technology, there are two types of MOSFETs: n-fets and p-fets In traditional CMOS circuits, all n-fets have the common bulk potential (V b ) connected to Ground (Gnd), and all p-fets have a common bulk potential (typically) connected to the power supply rail (d ). The corresponding (complementary) equation for the p-fet is ( ) I ds = I 0 e κ p(v b )/ e (V b )/ e (V b )/ V b G. Indiveri (NCS @ INI) MRI Winter School Tutorial 15 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 16 / 39 One, two, and three transistor circuits The differential-pair Ideal current source Current-mirror κv 1 Vs I 1 = I 0 e T I 1 I 2 M 1 M 2 κv 2 Vs I 2 = I 0 e T κv b I b = I 1 + I 2 = I 0 e T V 1 M V 2 M3 2 V bn M 1 Inverting amplifier Differential pair e Vs = I b 1 κv I 1 0 e T + e κv 2 0.8 1 x 10 8 I 2 I 1 V out I 1 I 2 V 1 M M3 V 2 2 V bn M 1 I 1 = I b I 2 = I b κv 1 e T κv 1 e T κv 2 + e T κv 1 e T κv 1 κv 2 e T + e T I 1, I 2 (A) 0.6 0.4 0.2 0 0.3 0.2 0.1 0 0.1 0.2 0.3 V 1 V 2 (V) G. Indiveri (NCS @ INI) MRI Winter School Tutorial 17 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 18 / 39
What is a synapse? Synapses in the nervous system In 1897 Charles Sherrington introduced the term synapse to describe the specialized structure at the zone of contact between neurons as the point in which one neuron communicates with another. E ex (Na +,...) Glutammate V mem GABA E inh (K +, Cl,...) G l C mem Electrical Chemical Excitatory Inhibitory Depressing Facilitating AMPA NMDA... 2005 winner of the Science and Engineering Visualization Challenge. by G. Johnson, Medical Media, Boulder, CO. G. Indiveri (NCS @ INI) MRI Winter School Tutorial 20 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 21 / 39 Synaptic transmission EPSC and EPSP In chemical synapses the presynaptic and postsynaptic membranes are spearated by extracellular space. The arrival of a presynaptic action potential triggers the release of neurotransmitter in the extracellular space. The neurotransmitters react with the postsynaptic receptors and depolarize the cell. Chemical synaptic transmission is characterized by specific temporal dynamics. Superimposed excitatory post-synaptic currents (EPSCs) recorded in a neuron at different membrane potentials (from Sacchi et al., 1998). Excitatory post-synaptic potential (EPSP) in response to multiple pre-synaptic spikes (from Nicholls et al. 1992). G. Indiveri (NCS @ INI) MRI Winter School Tutorial 22 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 23 / 39
Neural network models VLSI synapses in pulse-based neural networks W i I Wi V i = I Wi t C mem V mem C mem In classical neural network theory signals are (tipically) continuous values that represent the neuron s mean firing rate, neurons implement a saturating non-linearity transfer function (S) on the input s weighted sum, the synapse implements a multiplication between the neuron s input signal (X i ) and its corresponding synaptic weight (w i ). In pulse-based neural networks the weighted contribution of a synapse can be implemented using a single transistor. In this case p-fets implement excitatory synapse, and n-fets implement inhibitory synapses. The synaptic weight can be set by changing the W i bias voltage or the t duration. G. Indiveri (NCS @ INI) MRI Winter School Tutorial 24 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 25 / 39 Linear pulse integrators Linear charge-and-discharge integrator A linear integrator is a linear low-pass filter. Its impulse response should be a decaying exponential. With VLSI and subthreshold MOSFETS its fairly easy to implement exponential voltage to current conversion, and linear voltage increase or decrease over time. V c I d C I d = C d dt V c (t) I d (t) I d (t)=i 0 e κ (d (t)) V τ M τ M w I w M pre yn C syn M syn I syn I w = I 0 e κvw, τ c C syn κ κ(d Vτ) = I 0 e T, τ d C syn κ I c = C d dt (d yn ) κ(d Vsyn) I syn = I 0 e T I syn (t)= I i ) syn e+(t t τ c + I + i ) syn e (t t τ d (charge phase) (discharge phase) I syn (t)=i 0 e τ c f t(τ c +τ d ) ( τ c τ t n ) ( ) τc 1 d, with f =, f < t τ c + τ d t G. Indiveri (NCS @ INI) MRI Winter School Tutorial 26 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 27 / 39
Translinear circuits The translinear principle The term translinear was coined by Barry Gilbert in 1975. In translinear circuits transistors have a transconductance which is linearly proportional to the output current. This applies to: Circuits using monolithic BJTs Circuits using subthreshold MOS FETs in saturation For subthreshold MOSFETs in saturation: I DS = I 0 e (κv G V S )/ g m. = d dv G I DS = κ I DS Many current-mode circuits comprise transistors arranged in one or more closed loops of junctions. I 2 I1 I 3 IN λ 3 λ 2 V 2 λ1 V1 λn I 4 λ 4 V 3 V N V 4 V n V 5 V 6 λn λ5 λ 6 In I 5 I 6 In a closed loop containing an even number of forward-biased junctions arranged so that there are an equal number of clockwise-facing and counter clock-wise facing polarities, the product of the current densities in the clockwise direction is equal to the product o the current densities in the counter clock-wise direction. G. Indiveri (NCS @ INI) MRI Winter School Tutorial 29 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 30 / 39 Subthreshold low pass filter circuit Log-domain Low Pass Filter I M1 ds I M2 ds = I M3 ds I M4 ds I th = I 1 I M3 ds = I b + C lp d dt V c I C κ(d V C ) = I 0 e T M1 M2 V b I b M3 I b V c C lp M4 d dt = κ d dt V c I b =(I b + C lp κ τ + = ) I th I 1 V C τ d dt I C = C dv C dt I 1 = + I C + = I th V M1 gs + V M2 gs V M3 gs V M4 gs = 0 κvg Vs I ds = I 0 e T with τ C lp κi b with τ C κ G. Indiveri (NCS @ INI) MRI Winter School Tutorial 31 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 32 / 39
Log-domain Differential-Pair Integrator The DPI Silicon Synapse I 1 + I 2 = I th V G V S I 1 I 2 V C I 2 = + I C κv C = I 0 e T 0 V τ M D5 V thr M D1 M D4 C syn M D6 Iτ I C I C = C d dt V C I C = C κ d dt td M S1 M S2 M S3 C std V nmda M D2 I w V G M D3 M G1 M N1 M G2 M N2 V mem I syn I th I 1 = I 2 I th ( I C )=( + I C ) ( τ 1+ I ) th d dt + = I th I th if τ d dt + = I th τ d dt I syn+ I syn = I thri w The diff-pair integrator (DPI) circuit [Bartolozzi and Indiveri, Neural Computation, 2007] G. Indiveri (NCS @ INI) MRI Winter School Tutorial 33 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 34 / 39 DPI measured response DPI response to spike-trains EPSC (na) 300 250 200 150 =420mV =440mV =460mV EPSC (na) 450 400 350 300 250 200 =300mV =320mV =340mV 100 150 50 0 0 0.05 0.1 0.15 Time (s) 100 50 0 0 0.5 1 1.5 2 2.5 3 3.5 Time (s) G. Indiveri (NCS @ INI) MRI Winter School Tutorial 35 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 36 / 39
Conclusions Thank you for your attention Ids V τ C syn M D5 0 V thr M D6 M M D4 D1 M S1 V nmda td M S2 C std M D2 M N1 M N2 I w Additional information The INI (http://www.ini.uzh.ch/) The INE (http://www.ine-web.org/) The NCS group (http://ncs.ethz.ch/) The CapoCaccia Workshop (http://capocaccia.ethz.ch/) Acknowledgments M S3 Analog subthreshold circuits implement faithful models of neural processes are very compact and ultra low-power M D3 V G M G1 M G2 V mem I syn The Institute of Neuroinformatics, niversity of Zurich and ETH Zurich Tobi Delbrück Shih-Chii Liu Matthew Cook Michael Pfeiffer Rodney Douglas Kevan Martin can produce biologically realistic time constants G. Indiveri (NCS @ INI) MRI Winter School Tutorial 37 / 39 G. Indiveri (NCS @ INI) MRI Winter School Tutorial 38 / 39 The CapoCaccia Cognitive Neuromorphic Engineering Workshop http://capocaccia.ethz.ch/ Capo Caccia, Sardinia, Italy. April 28 - May 10, 2014 Interdisciplinary, international, inter E-S project Morning lectures, afternoon hands-on work-groups Active and lively discussions (no powerpoint) Concrete results, establishment of long-term collaborations G. Indiveri (NCS @ INI) MRI Winter School Tutorial 39 / 39