Recurrent Neural Networks (Part - 2) Sumit Chopra Facebook

Transcription

1 Recurrent Neural Networks (Part - 2) Sumit Chopra Facebook

2 Recap Standard RNNs Training: Backpropagation Through Time (BPTT) Application to sequence modeling Language modeling Applications: Automatic speech recognition, Machine translation Main problems in training

3 Major Shortcomings Handling of complex non-linear interactions Difficulties using BPTT to capture long-term dependencies Exploding gradients Vanishing gradients

4 Handling Non-Linear Interactions

5 Handling Non-Linear Interactions have depth not only in temporal dimension but also in space (at each time step) empirically shown to provide significant improvement in tasks like ASR, Unsupervised training using videos

6 Handling Non-Linear Interactions Gated RNNs shown to work on character based language modeling Sutskever et.al., 2011:Generating Test with Recurrent Networks

7 Training: Exploding Gradients Gradient Clipping during BPTT

8 Training: Exploding Gradients Gradient Clipping during BPTT

9 Training: Vanishing Gradients Multiple schools of thought better initialization of the recurrent matrix and using momentum during training Sutskever et.al.,: On The Importance of Initialization and Momentum in Deep Learning modifying the architecture

10 Structurally Constrained RNNs y t R h t U x t A Mikolov et.al., 2015:Learning Longer Memory in Recurrent Neural Networks

11 Structurally Constrained RNNs R y t h t U R h t U P y t V s t A A B x t x t s t = (1 )Bx t + s t 1, h t = (Ps t + Ax t + Rh t 1 ), y t = f (Uh t + Vs t )

12 Structurally Constrained RNNs Language Modeling on Penntree Bank Corpus Model #hidden #context Validation Perplexity Test Perplexity Ngram Ngram + cache SRN SRN SRN LSTM LSTM LSTM SCRN SCRN SCRN SCRN

13 Structurally Constrained RNNs Language Modeling on Text8 Corpus Model #hidden context = 0 context = 10 context = 20 context = 40 context = 80 SCRN SCRN SCRN able 3: Structurally constrained recurrent nets: perplexity for various sizes of the contextual layer,

14 Long Short-Term Memory (LSTM) recently gained a lot of popularity have explicit memory cells to store short-term activations the presence of additional gates partly alleviates the vanishing gradient problem multi-layer versions shown to work quite well on tasks which have medium term dependencies Hochreiter et.al., 1997: Long Short-Term Memory

15 Long Short-Term Memory (LSTM) y t R h t U x t A Hochreiter et.al., 1997: Long Short-Term Memory

16 Long Short-Term Memory (LSTM) h t = c t o t o t x t R y t h t U 1.0 Cell c t = c t h t g t i t x t A i t x t g t h t 1 h t 1 Hochreiter et.al., 1997: Long Short-Term Memory x t

17 Long Short-Term Memory (LSTM) h t = c t o t o t x t h t 1 h t 1 f x t t Cell c t = f t c t 1 + g t i t i t x t g t h t 1 h t 1 Hochreiter et.al., 1997: Long Short-Term Memory x t

18 Long Short-Term Memory (LSTM) h t = c t o t o t x t h t 1 h t 1 x f t t Cell c t = f t c t 1 + g t i t i t x t Peep-Hole Connections g t h t 1 h t 1 x t Hochreiter et.al., 1997: Long Short-Term Memory

19 LSTM Training Backpropagation Through Time: BPTT

20 Deep LSTMs

21 Bi-Directional LSTMs

22 Applications of LSTMs

23 Automatic Speech Recognition Use bi-directional LSTMs to represent the audio sequence plug a classifier on top of the representation to directly predict phone classes Graves et. al., 2014: Speech Recognition with Deep Recurrent Neural Networks

24 Automatic Speech Recognition Table 1. TIMIT Phoneme Recognition Results. Epochs is the number of passes through the training set before convergence. PER is the phoneme error rate on the core test set. NETWORK WEIGHTS EPOCHS PER CTC-3L-500H-TANH 3.7M % CTC-1L-250H 0.8M % CTC-1L-622H 3.8M % CTC-2L-250H 2.3M % CTC-3L-421H-UNI 3.8M % CTC-3L-250H 3.8M % CTC-5L-250H 6.8M % TRANS-3L-250H 4.3M % PRETRANS-3L-250H 4.3M % Graves et. al., 2014: Speech Recognition with Deep Recurrent Neural Networks

25 Sequence to Sequence Learning A B C > W X Y Z Machine Translation Short Text Response Generation Sentence Summarization Sutskever at. al., 2014: Sequence to Sequence Learning with Neural Network

26 Sequence to Sequence Learning Method test BLEU score (ntst14) Bahdanau et al. [2] Baseline System [29] Single forward LSTM, beam size Single reversed LSTM, beam size Ensemble of 5 reversed LSTMs, beam size Ensemble of 2 reversed LSTMs, beam size Ensemble of 5 reversed LSTMs, beam size Ensemble of 5 reversed LSTMs, beam size State-of-the-art WMT 14 result: 37.0 Sutskever at. al., 2014: Sequence to Sequence Learning with Neural Network

27 Unsupervised Training on Video Auto-encoder Model Learned Representation ˆv 3 ˆv 2 ˆv 1 W 1 W 1 copy W 2 W 2 v 1 v 2 v 3 v 3 v 2 Srivastav et. al., 2014: Unsupervised Learning of Video Representation using LSTMs

28 Unsupervised Training on Video Future Frame Predictor Model Learned Representation ˆv 4 ˆv 5 ˆv 6 W 1 W 1 copy W 2 W 2 v 1 v 2 v 3 v 4 v 5 Srivastav et. al., 2014: Unsupervised Learning of Video Representation using LSTMs

29 Unsupervised Training on Video Composite Model Input Reconstruction ˆv 3 ˆv 2 ˆv 1 W 2 W 2 Learned Representation copy v 3 v 2 W 1 W 1 copy ˆv 4 ˆv 5 ˆv 6 v 1 v 2 v 3 W 3 W 3 Sequence of Input Frames Future Prediction v 4 v 5 Srivastav et. al., 2014: Unsupervised Learning of Video Representation using LSTMs

30 Gated Recurrent Units z h ~ r h IN OUT Update gate: z j t = (W z x t + U z h t 1 ) j. Reset gate: r j t = (W r x t + U r h t 1 ) j. ustration of the GRU. Candidate activation: hj t = tanh (W x t + U (r t h t 1 )) j, h j t =(1 z j t )h j t 1 + zj t h j t,

31 Implementation Torch code available (soon!) Standard RNN LSTMs SCRNN and other models.. GPU compatible

32 Open Problems Encoding long-term memory into RNNs Speed-up the RNN training Control problems Language understanding