Synaptic Devices and Neuron Circuits for Neuron-Inspired NanoElectronics Byung-Gook Park Inter-university Semiconductor Research Center & Department of Electrical and Computer Engineering Seoul National University
Human vs. AlphaGo
AlphaGo - Software Monte Carlo Tree Search (MCTS) <Silver, Nature (2016)> Neural Networks B. G. Park
AlphaGo - Hardware Supercomputer Performance 1920 CPU 280 GPU B. G. Park 4
Comparison Human Brain Digital Computer neuron + synapse massively parallel ~ ms speed low power recognition/reasoning self-organized B. G. Park 5 CPU + memory serial ~ ns speed high power computation manufacturing
Interconnection Bottleneck Human Brain (Face Recognition) Computer System Bottleneck!! Bottleneck!! B. G. Park 6
Human Brain and Its Building Blocks Processor: neuron (~10 11 ) Memory: Synapse (~10 15 ) B. G. Park 7
Neuron (1) Structure of a neuron B. G. Park 8
Neuron (2) Structure of a neuron 100 V (mv) 50 0 5 t(ms) Potential change due to K + ion channel B. G. Park 9
Synapse (1) Electrochemical signal transmission in a synapse Neurotransmitter is emitted in the unit of a vesicle conveyed to the post-synaptic cell through the cleft B. G. Park 10
Synapse (2) Short term memory and long term memory B. G. Park 11
Spike-Timing-Dependent Plasticity Spike-timing-dependent plasticity learning mechanism B. G. Park 12
Building Blocks: Biological vs. Electronic B. G. Park 13
Artificial Neural Network (ANN) Concept of neural network <perceptron> (1958) B. G. Park ** Weights are calculated by the back-propagation (BP) method (1986). 14
Deep Neural Network (DNN) Multiple hidden layers ** Vanishing gradient problem (VGP) unsupervised pre-training (RBM) + modified activation function (ReLU) B. G. Park 15
1 st Breakthrough (2006) Pre-training with restricted Boltzmann machine (RBM) Regard each pair of layers as RBM P ( x) E W ij T 1 ( x) b x 2 exp[ E ( x) / ] exp[ E ( x') / ] x' x ( ln P ) Wij W T Wx ij B. G. Park 16
2 nd Breakthrough (2010) Rectified Linear Unit (ReLU) sigmoid hyperbolic tangent ** Vanishing ReLU solves gradient the vanishing problem gradient due to saturating problem!! activation (+ Concept function of signal intensity included) B. G. Park 17
Comparison: ReLU vs. Sigmoid Speed of Learning: 8:1 Compression <Original> <ReLU> <Sigmoid> 512x512 Image Epoch = 100 200 400 800 60 MSE = 0.00177 0.00104 0.00097 0.00095 0.00116 0.00094 0.00093 B. G. Park 18 ** MSE (mean square error) Epoch = 800 100 200 400 60 MSE = 0.00403 0.00142 0.00395 0.00392 0.00380 0.00441 0.00250 0.00194
Recurrent Neural Network (RNN) Directed cycles in connections Sequential data modeling B. G. Park ** Complicated dynamics and difficulty in training 19
Long Short-Term Memory (LSTM) RNN with LSTM LSTM block ** Hidden layer units with LSTM can store and access information over a long period of time. B. G. Park 20
Spiking Neural Network (SNN) (1) 3 rd generation neural network model - input/output: spikes - signal intensity: firing rates PSP(t) Sequence of spikes Unit j V(t) Sequence of spikes F j t (f) j S Weighted sum of decayed PSPs Output spikes Sequence of spikes B. G. Park 21
Spiking Neural Network (SNN) (2) Learning mechanism - error back-propagation with time coding - spike-timing-dependent plasticity (STDP) S pre S post S pre S post B. G. Park 22
Spiking Neural Network (SNN) (3) Structure Supervised learning before learning random fixed weights B. G. Park learning with modified rule 23 after learning
STDP and Error Back-propagation STDP W ij i j BP W ij x x ( : spike rate) (x : neuron output) i j B. G. Park 24 <Bengio, arxiv.org, (2016)>
Floating-body Synaptic Transistor (1) Structure TEM image O D A A G1 FB G2 Gate1 Sidewall (MTO) Gate2 S O ~ 3.5 nm Fin O/N/O ~ 3.5/5/8 nm ONO B. G. Park 25 BOX Cross-section in A A direction 50 nm
Floating-body Synaptic Transistor (2) Short-term memorization Long-term memorization Charge trap layer Charge trap layer Impact-generated holes are temporarily stored in the body. Without further inputs, these holes gradually disappear through recombination process. B. G. Park 26 Hot holes are programmed to the floating gate. Even without further inputs, these charges do not disappear without special erasing actions.
Floating-body Synaptic Transistor (3) 1.0 0.5 Input Pulse Height [V] Transient response of FST to spikes 10 μs interval 100 μs interval 2 0 0.0 Source Current [ A]1.5 0 20 40 60 80 100 Time [ s] 0 200 400 600 800 1000 Time [ s] B. G. Park 27
Source current [na] Floating-body Synaptic Transistor (4) Short-term to long-term memory transition 10 μs interval 100 μs interval 150 100 50 short-term memory N = 1~ 10 one-time increase per step long-term memory due to hole trapping N = 1~ 10 No Transition 0 forgetting 10-9 10-7 10-5 10-3 10-1 Time [s] B. G. Park 28 10-7 10-5 10-3 10-1 Time [s]
Source Current [ A] Floating-body Synaptic Transistor (5) STDP characteristic 0.15 0.10 0.05 S pre t S post 0.00-0.05-0.10-0.15-5 -4-3 -2-1 0 1 2 3 4 5 t [ s] B. G. Park 29
Neuron Circuits (1) Integrate-and-fire neuron circuit with capacitor integration <synaptic integration part> B. G. Park 30 <spike generation part>
Neuron Circuits (2) Implementation with discrete devices <PCB layout> <Output of neuron> B. G. Park 31
Current change I [na] Neuron Circuits (3) Measured STDP characteristics < Transient current > <STDP characteristic> 50 40 30 20 10 0-10 -20 STDP chracteristic (measuremt) -30-6 -4-2 0 2 4 6 Time difference t [ s] B. G. Park 32
Neuron Circuits (4) Integrate-and-fire neuron circuit with a floating-body MOSFET B. G. Park 33
Neuron Circuits (5) Role of floating-body MOSFET in the circuit <transient current> <spike generation> - Temporal integration of input spike signal by the floating body B. G. Park 34
Neuron Circuits (6) Shapes of spikes and STDP characteristic <spike shapes> <STDP characteristic> - Output and feedback spike shapes are different!! B. G. Park 35
Integration of Neurons and Synapses Stacking of neuron and synapse arrays <primary sensory cortex> <neuromorphic system> Neuron Array Synapse Array Neuron Array Synapse Array Neuron Array Synapse Array Neuron Array B. G. Park 36
Conclusions q The recent advancement of ANNs has been achieved by imitating the biological neural networks (BNNs) more closely. Spiking neural networks with STDP weight adjustment is the closest to the BNN. q Combining the capacitor-less DRAM and SONOS flash memory, we have developed floating-body synaptic transistors (FSTs), which show short- and long-term memory and STDP. q Integrate-and-fire neuron circuits with capacitor and floating-body integration are proposed and implemented. B. G. Park 37
Thank you for your attention! B. G. Park 38