Neuromorphic architectures: challenges and opportunites in the years to come

Neuromorphic architectures: challenges and opportunites in the years to come Andreas G. Andreou andreou@jhu.edu Electrical and Computer Engineering Center for Language and Speech Processing Johns Hopkins University http://www.ece.jhu.edu/faculty/andreou/aga/index.htm NRI Workshop 2009 8/18/2009 1

Part I: Neuromorphic architectures What did we learn the last 25 years? NRI Workshop 2009 8/18/2009 2

1986: Let the physics do the work! max x ci a+ c i= 1, N i System / Architecture Circuits Devices / Technology i i i i = 0 = 0 1 V () t = I() t dt C i Q I V i = 0 t I D SI n0 exp κ V n GS October 1986 (1st Draft) V t NRI Workshop 2009 8/18/2009 3

circuits: analog, digital and beyond CVDT Continuous-Value Discrete-Time Continuous-Value Continuous-Time CVCT CCD Switched Capacitor Binary digital Multivalue digital Linear and non-linear analog Asynchronous digital Neuron spikes EPSP Anisochronous Pulse Time Modulation DVDT Discrete-Value Discrete-Time Discrete-Value Continuous-Time DVCT P.M. Furth and A.G. Andreou, Comparing the bit-energy of continuous and discrete signal representations, Proceedings of the Fourth Workshop on Physics and Computation (PhysComp96), T.Toffoli, M. Biafore and J. Leao eds., New England Complex Systems Institute, pp. 127-133, Boston, MA, November 1996. NRI Workshop 2009 8/18/2009 4

the energy costs of computing 10 16 8-9 bits DVDT practical limit at 10nm CMOS C = fbw log2 1+ S N CVCT DVDT Landauer (theoretical limit) Power [ J ] BitEnergy Capacity [ bit] NRI Workshop 2009 8/18/2009 5

silicon retina NIPS 91 Chemical synapses Electrical synapses NRI Workshop 2009 8/18/2009 6

the mathematical abstraction of biology 1. Photons to electrons: transduction and amplification I ( x, y ) = β I ( x, y ) Φ( x, y ) in m n ph m n m n 2. Local gain control: source coding I ( x, y ) = I out m n u Iin( xm, yn) I ( x, y ) + ψ I ( x, y ) in m n M, N in i j 3. Spatial filtering: optimal smoothing i x y i x y i x y 2 2 h( m, n) + λ h( m, n) = in( m, n) = 2 out m n h m n i ( x, y ) i ( x, y ) Andreas NRI Workshop G. Andreou 2009 DARPA Monterey Workshop 8/18/2009 Tuesday, August 18, 2009 7

the statistics of natural scenes pdf (log( I)) W. Richards, Lightness scale from image intensity distributions, Applied Optics, vol. 21, no. 14, 2569-2582, 1982 1 PO ( ) 0.6 1 N 0.2 O m Rfl*SrfOr*Txtr 0.001 0.01 0.1 Image Intensity I = (log( I)) n (log( I )) + n c n m = 12.5 n = 1 I = 3 S NRI Workshop 2009 8/18/2009 8 1 log( I) P(log( I)) d( I ) = P( O) d( O) 1 PO ( ) N log( I) O = G(log( I )) = N P( x) d( x) 1 Naka-Rushton Equation

matching signals to circuits! challenge: matching the wide dynamic range of signals to limited dynamic range of analog computing hardware circuit design problem I I I I I I I p out C = = p p U U + H I = 0.6 na,1.8na U p = 1.2 I = 1.5 na, 3nA H non-linear analog processing to do source coding NRI Workshop 2009 8/18/2009 9

dealing with the dynamic range problem Silicon Retina CCD Camera (210 x 230 pixels) X (6 OPS per pixel for second order smoothing) X (6 OPS per pixel for Laplacian) X (6 OPS per pixel for gain control) X (10 5 OPS per second --100 khz temporal response--) = 5 X 10 4 X 2 x 10 2 X 10 5 10 12 OPS with 50mW total power dissipation at 5 Volts power supply Subthreshold CMOS 560,000 transistors, era 1995 NRI Workshop 2009 8/18/2009 10

embedded analog computing in digital memories exploiting problem statistics! minimal complexity CMOS circuits Andreas NRI Workshop G. Andreou 2009 DARPA Monterey Workshop Tuesday, 8/18/2009August 18, 2009 11

precision on demand architecture Andreas NRI Workshop G. Andreou 2009 DARPA Monterey Workshop Tuesday, 8/18/2009August 18, 2009 12

dealing with device mismatch again ISCAS 94 Floating Gate Adaptation Andreas NRI Workshop G. Andreou 2009 DARPA Monterey Workshop Tuesday, 8/18/2009August 18, 2009 13

Part II: What is the real problem? Physical world 3 dimensional world s problems N dimensional NRI Workshop 2009 8/18/2009 14

natural and synthetic computing structures The Brain IBM Blue Gene/L exist in three dimensional physical space but can deal with problems in hyper dimensional spaces NRI Workshop 2009 8/18/2009 15

visual representation of the world through cortical maps Multiple stimulus modalities such as orientation, spatial frequency, ocular dominance are mapped into 2D+δ patches on the surface of the cortex (V1) Conflicting constraints Maximize coverage every location in the physical space is mapped to all possible combinations of stimulus modalities- Minimize wiring length and metabolic costs neurons with similar stimulus response should exist in closed proximity on the cortical surface (smoothness of mapping). The ice-cube model for stimulus representation in V1 (Hubel and Wiesel 1977) suggests stimulus modalities in orthogonal dimensions orientation ocular dominance NRI Workshop 2009 8/18/2009 16

natural and synthetic computing structures: another view The Brain IBM Blue Gene/L 15W 125 KW 5 racks NRI Workshop 2009 8/18/2009 17

the energy costs of communication 3D CMOS a. 10nm CMOS inverter b. 100nm CMOS inverter c. Intra die 1cm metal line d. Electrical chip to chip link e. Optical chip to chip link f. FireWire link g. Wireless chip to chip link h. Ultra Wide Band radio M.A Marwick and A.G. Andreou, Retinomorphic system design in three dimensional SOI-CMOS, Proceedings of the 2006 IEEE International Symposium on Circuits and Systems. NRI Workshop 2009 8/18/2009 18

3D silicon cortex (a dynamical system approach) Supply Voltage 1.5V Technology MITLL 0.18μm 3DL1 Array size 64 64 Spatial Processing 8 orient Filter time 3-4ns External Communications ADER protocol (2 phase async.) Internal Communications 4 phase asynchronous Dynamic Range 6 bit Minimum Frame Rate 300Hz Maximum Frame Rate 20kHz Andreas NRI Workshop G. Andreou 2009 DARPA Monterey Workshop Tuesday, 8/18/2009August 18, 2009 19

Towards Field Programmable Spiking Array in 3D CMOS Results gold din green din ack red dout purple dout ack Block Diagram 5 asynchronous handshake buffers in each path (4 deep FIFO + 1 MUX) Utilizes all three tiers Handshake between tiers data_i ctrl_i d_b_o d_a_o Tier C Tier B Tier A Andreas NRI Workshop G. Andreou 2009 DARPA Monterey Workshop Tuesday, 8/18/2009August 18, 2009 20

Part III: Final remarks Group B: Andreou, Dally, Roukes, Cornwell, Nair, Simon NRI Workshop 2009 8/18/2009 21

Commodity The Tyrannies Compatibility NRI Workshop 2009 8/18/2009 22

Nano to the rescue Improving memory density Makes higher Bytes per FLOP economically feasible Improves the capacity at each level NRI Workshop 2009 8/18/2009 23

On hardware, algorithms and architectures Computational and energy efficiency can only be achieved through co-development of algorithms to application specific, reconfigurable architectures. NRI Workshop 2009 8/18/2009 24

summary Early 80s: Carver Mead s neuromorphic manifesto towards new ways of computing inspired by biology device physics based approach exploit statistics of the problem parallel distributed processing processors in memory adaptation learning analog circuits Late 90s: Some neuromorphic apostles took the wrong turn and got lost on the way. Today: Good news! The problems of the world have not yet been solved, and the apostles are 15 years older and hopefully wiser! NRI Workshop 2009 8/18/2009 25