Machine Learning for Signal Processing Neural Networks Continue. Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016

Size: px

Start display at page:

Download "Machine Learning for Signal Processing Neural Networks Continue. Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec 2016"

Frederica Cannon
5 years ago
Views:

1 Machine Learning for Signal Processing Neural Networks Continue Instructor: Bhiksha Raj Slides by Najim Dehak 1 Dec

2 So what are neural networks?? Voice signal N.Net Transcription Image N.Net Text caption Game State N.Net Next move What are these boxes? 18797/

3 So what are neural networks?? It began with this.. Humans are very good at the tasks we just saw Can we model the human brain/ human intelligence? An old question dating 18797/11755 back to Plato and Aristotle.. 3

4 MLP - Recap MLPs are Boolean machines They represent Boolean functions over linear boundaries They can represent arbitrary boundaries Perceptrons are correlation filters They detect patterns in the input MLPs are Boolean formulae over patterns detected by perceptron Higher-level perceptrons may also be viewed as feature detectors MLPs are universal approximators Can model any function to arbitrary precision Extra: MLP in classification The network will fire if the combination of the detected basic features matches an acceptable pattern for a desired class of signal E.g. Appropriate combinations of (Nose, Eyes, Eyebrows, Cheek, Chin) Face 4

5 MLP - Recap MLPs are Boolean machines They represent arbitrary Boolean functions over arbitrary linear boundaries Perceptrons are pattern detectors MLPs are Boolean formulae over these patterns MLPs are universal approximators Can model any function to arbitrary precision MLPs are very hard to train Training data are generally many orders of magnitude too few Even with optimal architectures, we could get rubbish Depth helps greatly! Can learn functions that regular classifiers cannot 5

6 What is a deep network?

7 Deep Structures In any directed network of computational elements with input source nodes and output sink nodes, depth is the length of the longest path from a source to a sink Left: Depth = 2. Right: Depth = 3

8 Deep Structures Layered deep structure Deep Depth > 2

9 MLP as a continuous-valued regression x 1 1 T 1 T 1 T f(x) T 1 T 2 x + x T 2 MLPs can actually compose arbitrary functions to arbitrary precision Not just classification/boolean functions 1D example Left: A net with a pair of units can create a pulse of any width at any location Right: A network of N such pairs approximates the function with N scaled pulses 9

10 MLP features DIGIT OR NOT? The lowest layers of a network detect significant features in the signal The signal could be reconstructed using these features Will retain all the significant components of the signal 10

11 Making it explicit: an autoencoder X Y W T W X A neural network can be trained to predict the input itself This is an autoencoder An encoder learns to detect all the most significant patterns in the signals A decoder recomposes the signal from the patterns 11

12 Deep Autoencoder DECODER ENCODER

13 What does the AE learn X X W T W Y Y = WX X = W T Y E = X W T WX 2 Find W to minimize Avg[E] In the absence of an intermediate non-linearity This is just PCA 13

14 The AE DECODER With non-linearity Non linear PCA ENCODER Deeper networks can capture more complicated manifolds 14

15 The Decoder: DECODER The decoder represents a source-specific generative dictionary Exciting it will produce typical signals from the source! 15

16 The AE DECODER Cut the AE ENCODER 16

17 The Decoder: Sax dictionary DECODER The decoder represents a source-specific generative dictionary Exciting it will produce typical signals from the source! 17

18 The Decoder: Clarinet dictionary DECODER The decoder represents a source-specific generative dictionary Exciting it will produce typical signals from the source! 18

19 NN for speech enhancement 19

20 Story so far MLPs are universal classifiers They can model any decision boundary Neural networks are universal approximators They can model any regression The decoder of MLP autoencoders represent a non-linear constructive dictionary! 20

location of the pattern Moving it by one component results in an entirely

21 The need for shift invariance = In many problems the location of a pattern is not important Only the presence of the pattern Conventional MLPs are sensitive to the location of the pattern Moving it by one component results in an entirely different input that the MLP wont recognize Requirement: Network must be shift invariant

22 Convolutional Neural Networks History Hubel and Wiesel: 1959 (biological model), Fukushima: 1980 (computational model), Altas: 1988, Lecunn: 1989 (Backprop in convnets) Yann LeCun Kunihiko Fukushima

23 Convolutional Neural Networks A special kind of multi-layer neural networks. Implicitly extract relevant features. A feed-forward network that can extract topological properties from an image. CNNs are also trained with a version of back-propagation algorithm.

24 Connectivity & weight sharing All different weights All different weights Shared weights Convolution layer has much smaller number of parameters by local connection and weight sharing

25 Fully Connected Layer Example: 200x200 image 40K hidden units ~2B parameters!!! - Spatial correlation is local - Waste of resources + we have not enough training samples anyway.. Ranzato 25

26 Locally Connected Layer Example: 200x200 image 40K hidden units Filter size: 10x10 4M parameters Note: This parameterization is good when input image is registered (e.g., face recognition). Ranzato 26

27 Locally Connected Layer STATIONARITY? Statistics is similar at different locations Example: 200x200 image 40K hidden units Filter size: 10x10 4M parameters Ranzato 27

28 Convolutional Layer Share the same parameters across different locations (assuming input is stationary): Convolutions with learned kernels Ranzato 28

29 Convolution

30 Convolutional Layer Ranzato

31 Convolutional Layer Ranzato

32 Convolutional Layer Ranzato

33 Convolutional Layer Ranzato

34 Convolutional Layer Ranzato

35 Convolutional Layer Ranzato

36 Convolutional Layer Ranzato

37 Convolutional Layer Ranzato

38 Convolutional Layer Ranzato

39 Convolutional Layer Ranzato

40 Convolutional Layer Ranzato

41 Convolutional Layer Ranzato

42 Convolutional Layer Ranzato

43 Convolutional Layer Ranzato

44 Convolutional Layer Ranzato

45 Convolutional Layer Ranzato

46 Convolutional Layer Learn multiple filters. E.g.: 200x200 image 100 Filters Filter size: 10x10 10K parameters Ranzato 46

47 Convolutional Layers before: input layer hidden layer output layer now:

48 Convolution Layer 32x32x3 image 32 height 3 32 depth width

49 Convolution Layer 32x32x3 image 32 5x5x3 filter Convolve the filter with the image i.e. slide over the image spatially, computing dot products 3 32

50 Convolution Layer 32x32x3 image Filters always extend the full depth of the input volume 5x5x3 filter 32 Convolve the filter with the image i.e. slide over the image spatially, computing dot products 3 32

51 Convolution Layer 32 32x32x3 image 5x5x3 filter number: the result of taking a dot product between the filter and a small 5x5x3 chunk of the image (i.e. 5*5*3 = 75-dimensional dot product + bias)

52 Convolution Layer 32 32x32x3 image 5x5x3 filter activation map 28 convolve (slide) over all spatial locations

53 Convolution Layer consider a second, green filter 32 32x32x3 image 5x5x3 filter activation maps 28 convolve (slide) over all spatial locations

54 Convolution Layer For example, if we had 6 5x5 filters, we ll get 6 separate activation maps: activation maps Convolution Layer We stack these up to get a new image of size 28x28x6!

55 CNN Preview: ConvNet is a sequence of Convolution Layers, interspersed with activation functions CONV, ReLU e.g. 6 5x5x3 filters 6 28

56 CNN Preview: ConvNet is a sequence of Convolutional Layers, interspersed with activation functions CONV, ReLU e.g. 6 5x5x3 filters 28 6 CONV, ReLU e.g. 10 5x5x6 filters CONV, ReLU.

57 Pooling Layer Let us assume filter is an eye detector. Q.: how can we make the detection robust to the exact location of the eye? Ranzato 57

58 Pooling Layer By pooling (e.g., taking max) filter responses at different locations we gain robustness to the exact spatial location of features. Ranzato 58

59 Pooling Layer - makes the representations smaller and more manageable - operates over each activation map independently:

60 Max Pooling x Single depth slice max pool with 2x2 filters and stride y

61 ConvNets: Typical Stage One stage (zoom) Convol. Pooling courtesy of K. Kavukcuoglu Ranzato 61

62 Digit classification

63 ImageNet 1.2 million high-resolution images from ImageNet LSVRC-2010 contest 1000 different classes (sofmax layer) NN configuration NN contains 60 million parameters and 650,000 neurons, 5 convolutional layers, some of which are followed by max-pooling layers 3 fully-connected layers Krizhevsky, A., Sutskever, I. and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada

ImageNet Figure 3: 96 convolutional kernels of size 11 11 3 learned by the first convolutional layer on the 224 224 3 input images.

64 ImageNet Figure 3: 96 convolutional kernels of size learned by the first convolutional layer on the input images. The top 48 kernels were learned on GPU 1 while the bottom 48 kernels were learned on GPU 2. See Section 6.1 for details. Krizhevsky, A., Sutskever, I. and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada

65 ImageNet Eight ILSVRC-2010 test images and the five labels considered most probable by our model. The correct label is written under each image, and the probability assigned to the correct label is also shown with a red bar (if it happens to be in the top 5). Five ILSVRC-2010 test images in the first column. The remaining columns show the six training images that produce feature vectors in the last hidden layer with the smallest Euclidean distance from the feature vector for the test image. Krizhevsky, A., Sutskever, I. and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks NIPS 2012: Neural Information Processing Systems, Lake Tahoe, Nevada

66 CNN for Automatic Speech Recognition Convolution over frequencies Convolution over time

CNN-Recap Neural network with specialized connectivity structure Feed-forward: - Convolve input - Non-linearity (rectified linear) - Pooling (local max) Supervised training Train

67 CNN-Recap Neural network with specialized connectivity structure Feed-forward: - Convolve input - Non-linearity (rectified linear) - Pooling (local max) Supervised training Train convolutional filters by back-propagating error Convolution over time Adding memory to classical MLP network Recurrent neural network Feature maps Pooling Non-linearity Convolution (Learned) Input image

68 Recurrent Neural Networks (RNNs) Recurrent Neural Network Recurrent networks introduce (RNN) cycles and a notion of time. x t y t h t 1 h t One-step delay They are designed to process sequences of data x 1,, x n and can produce sequences of outputs y 1,, y m.

69 Elman Nets (1990) Simple Recurrent Neural Networks Elman nets are feed forward networks with partial recurrence Unlike feed forward nets, Elman nets have a memory or sense of time Can also be viewed as a Markovian NN

70 (Vanilla) Recurrent Neural Network Simple Recurrent Neural Network The state consists of a single hidden vector h: x t y t h t 1 h t One-step delay

71 Unrolling RNNs Recurrent Neural Network RNNs can be unrolled across multiple time steps. x t y t h t 1 h t y 0 y 1 y 2 One-step delay h 0 h 1 h 2 This produces a DAG which supports backpropagation. x 0 x 1 x 2 But its size depends on the input sequence length.

72 Learning time sequences Recurrent networks have one more or more feedback loops There are many tasks that require learning a temporal sequence of events Speech, video, Text, Market These problems can be broken into 3 distinct types of tasks 1. Sequence Recognition: Produce a particular output pattern when a specific input sequence is seen. Applications: speech recognition 2. Sequence Reproduction: Generate the rest of a sequence when the network sees only part of the sequence. Applications: Time series prediction (stock market, sun spots, etc) 3. Temporal Association: Produce a particular output sequence in response to a specific input sequence. Applications: speech generation

73 RNN structure Recurrent Neural Network Often layers are stacked vertically (deep RNNs): Abstraction - Higher level features y 00 h 00 x 0 Time y 10 y 11 y 12 h 10 h 11 h 12 x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Same parameters at this level Same parameters at this level

74 RNN structure Recurrent Neural Network Backprop still works: (it called Backpropagation Through Time) y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

75 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

76 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

77 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

78 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

79 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

80 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Activations

81 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Gradients

82 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Gradients

83 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Gradients

84 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features x 00 x 01 x 02 y 00 h 00 x 0 Time y 01 h 01 x 1 y 02 h 02 x 2 Gradients

85 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Gradients

86 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features y 00 h 00 x 0 Time x 00 x 01 x 02 y 01 h 01 x 1 y 02 h 02 x 2 Gradients

87 RNN structure Recurrent Neural Network Backprop still works: y 10 y 11 y 12 h 10 h 11 h 12 Abstraction - Higher level features x 00 x 01 x 02 y 00 h 00 x 0 Time y 01 h 01 x 1 y 02 h 02 x 2 Gradients

88 The memory problem with RNN RNN models signal context If very long context is used -> RNNs become unable to learn the context information

89 Standard RNNs to LSTM Standard LSTM

90 LSTM illustrated: input and forming new memory LSTM cell takes the following input the input x t past memory output h t 1 past memory C t 1 (all vectors) Cell state Forget gate Input gate New memory

91 LSTM illustrated: Output Forming the output of the cell by using output gate Overall picture:

LSTM Equations i: input gate, how much of the new information will be let through the memory cell. f: forget gate, responsible for information should be thrown away from memory cell.

92 LSTM Equations i: input gate, how much of the new information will be let through the memory cell. f: forget gate, responsible for information should be thrown away from memory cell. i = σ x t U i + s t 1 W i f = σ x t U f + s t 1 W f o = σ x t U o + s t 1 W o g = tanh x t U g + s t 1 W g c t = c t 1 f + g i s t = tanh c t o y = softmax Vs t o: output gate, how much of the information will be passed to expose to the next time step. g: self-recurrent which is equal to standard RNN c t : internal memory of the memory cell s t : hidden state LSTM Memory Cell y: final output 92

93 LSTM output synchronization

94 (NLP) Applications of RNNs Section overview Language Model Sentiment analysis / text classification Machine translation and conversation modeling Sentence skip-thought vectors

95 RNN for

96 Sentiment analysis / text classification A quick example, to see the idea. Given text collections and their labels. Predict labels for unseen texts.

Translating Videos to Natural Language Using Deep Recurrent Neural Networks Translating Videos to Natural Language Using Deep Recurrent Neural Networks Subhashini

97 Translating Videos to Natural Language Using Deep Recurrent Neural Networks Translating Videos to Natural Language Using Deep Recurrent Neural Networks Subhashini Venugopalan, Huijun Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, Kate Saenko North American Chapter of the Association for Computational Linguistics, Denver, Colorado, June 2015.

99 Composing music with RNN

100 CNN-LSTM-DNN for speech recognition Ensembles of RNN/LSTM, DNN, & Conv Nets (CNN) give huge gains (state of the art): T. Sainath, O. Vinyals, A. Senior, H. Sak. Convolutional, Long Short-Term Memory, Fully Connected Deep Neural Networks, ICASSP 2015.

101 The Impact of deep learning in speech technologies Cortana

102 Conclusions MLPs are Boolean machines They represent Boolean functions over linear boundaries They can represent arbitrary boundaries Perceptrons are correlation filters They detect patterns in the input MLPs are Boolean formulae over patterns detected by perceptron Higher-level perceptrons may also be viewed as feature detectors MLPs are universal approximators Can model any function to arbitrary precision Non linear PCA Convolute NN can handle shift invariance CNN Special NN can model sequential data RNN, LSTM

Deep Learning Sequence to Sequence models: Attention Models. 17 March 2018

Deep Learning Sequence to Sequence models: Attention Models 17 March 2018 1 Sequence-to-sequence modelling Problem: E.g. A sequence X 1 X N goes in A different sequence Y 1 Y M comes out Speech recognition: