Introduction to Convolutional Neural Networks (CNNs)

Size: px

Start display at page:

Download "Introduction to Convolutional Neural Networks (CNNs)"

Isabella Burke
5 years ago
Views:

1 Introduction to Convolutional Neural Networks (CNNs) Department of Transdisciplinary Studies Seoul National University, Korea Jan Many slides are from Fei-Fei Raquel Toronto, and Marc Aurelio

2 Traditional Pattern Recognition

3 Hierarchical Compositionality (DEEP)

4 Deep Learning What is deep learning? Nothing new! (Many) cascades of nonlinear transformations End-to-end learning (no human intervention / no fixed features)

5 Shallow(?) Learning Examples - Supervised

6 Deep Learning Examples - Supervised

7 Artificial Neural Networks (ANN) Biologically inspired models

8 Variants of ANNs

9 A Brief History of ANNs source of image: VUNO Inc.

10 A Brief History of ANNs First Generation: 1957 Perceptron: Rosenblatt, 1957 Adaline: Widrow and Hoff, 1960

11 A Brief History of ANNs First Generation: 1957 Perceptron: Rosenblatt, 1957 Adaline: Widrow and Hoff, 1960 Second Generation: 1986 MLP with BP: Rumelhart

12 A Brief History of ANNs Third Generation: 2006 RBM: Hinton and Salkhutdinov Reinvigorated research in Deep Learning

13 Classification problems Given inputs x, and outputs t { 1, 1} Find a hyperplane that divides the space into half (binary classification) y = sign(w T x + w 0 ) SVM tries to maximize the margin.

14 Classification problems Given inputs x, and outputs t { 1, 1} Find a hyperplane that divides the space into half (binary classification) y = sign(w T x + b) SVM tries to maximize the margin.

15 Nonlinear predictors How can we make our classifier more powerful?

16 Nonlinear predictors How can we make our classifier more powerful? Compute nonlinear functions of the input y = F(x, w)

17 Nonlinear predictors How can we make our classifier more powerful? Compute nonlinear functions of the input y = F(x, w) Two types of widely used approaches

18 Nonlinear predictors How can we make our classifier more powerful? Compute nonlinear functions of the input y = F(x, w) Two types of widely used approaches Kernel Trick: Fixed functions and optimize linear parameters on nonlinear mappings φ(x) y = sign(w T φ(x) + b)

19 Nonlinear predictors How can we make our classifier more powerful? Compute nonlinear functions of the input y = F(x, w) Two types of widely used approaches Kernel Trick: Fixed functions and optimize linear parameters on nonlinear mappings φ(x) y = sign(w T φ(x) + b) Deep Learning: Learn parametric nonlinear functions y = F(x, w) = (h 2 (w T 2 h 1 (w T 1 x + b 1 ) + b 2 ) h 1,2 : activation function at layer 1 or 2

20 Neural Networks Deep learning uses composite of simpler functions, e.g., ReLU, sigmoid, tanh, max Note: a composite of linear functions is linear! Example: 2 layer NNet (Convention: input and output layers are not taken as a layer)

21 Neural Networks Deep learning uses composite of simpler functions, e.g., ReLU, sigmoid, tanh, max Note: a composite of linear functions is linear! Example: 2 layer NNet (Convention: input and output layers are not taken as a layer) x is the input y is the output h i is the i-th hidden layer output W i is the set of parameters of the i-th layer

22 Nonlinearity - Activation function A singlue neuron can be used as a binary linear classifier Regularization has the interpretation of gradual forgetting Classical NNs used sigmoid or tanh function as an activation function. sigmoid: σ(x) = 1 1+e x tanh: tanh(x) = ex e x e x +e x

23 Sigmoid function Squashes numbers to range [0,1] Historically popular since they have nice interpretation as a saturating firing rate of a neuron

24 Sigmoid function Squashes numbers to range [0,1] Historically popular since they have nice interpretation as a saturating firing rate of a neuron 2 BIG problems: 1 Saturated neurons kill the gradients (cannot backprop further) Major bottleneck for the conventional NNs: not able to train more than 2 or 3 layers 2 Sigmoid outputs are not zero-centered Restriction on the gradient directions

25 ReLU: Rectified Linear Unit f (x) = max(0, x) Does not saturate Computationally very efficient Converges much faster than sigmoid/tanh in practice (e.g. 6x)

26 ReLU: Rectified Linear Unit f (x) = max(0, x) Does not saturate Computationally very efficient Converges much faster than sigmoid/tanh in practice (e.g. 6x) One annoying problem Dead neurons

27 ReLU: Rectified Linear Unit f (x) = max(0, x) Does not saturate Computationally very efficient Converges much faster than sigmoid/tanh in practice (e.g. 6x) One annoying problem Dead neurons Solution: leaky ReLU (small slope for negative input) Never dies. However, almost the same performance in practice.

28 Why ReLU? Piecewise linear tiling: mapping is locally linear Montufar et al. On the number of linear regions of DNNs, arxiv 2014

29 Forward Pass: Evaluating the function Forward propagation: compute the output y given the input x Fully connected layer Nonlinearity comes from ReLU Do it in a compositional way x h 1 h 2 y

30 Alternative graphical representation Slide from M. Ranzato

31 Why many layers? Hierarchically distributed representations Lee et al. Convolutional DBN s ICML 2009

32 Why many layers? Hierarchically distributed representations Lee et al. Convolutional DBN s ICML 2009

33 Training We want to estimate the parameters, biases and hyper-parameters (e.g., number of layers, number of neurons) for good predictions. Collect a training set of input-output pairs {x i, t i } N i=1. Encode the output with 1-K encoding t = [0,, 1,, 0].

34 Training We want to estimate the parameters, biases and hyper-parameters (e.g., number of layers, number of neurons) for good predictions. Collect a training set of input-output pairs {x i, t i } N i=1. Encode the output with 1-K encoding t = [0,, 1,, 0]. Define a loss per training example and minimize the empirical loss L(w) = 1 N l(w, x i, t i ) + R(w) N i=1 N : number of training examples R: regularizer w: set of all parameters

35 Loss functions L(w) = 1 N N l(w, x i, t i ) + R(w) i=1 Softmax (Probability of class k given input): p(c k = 1 x) = exp(y k ) Cj=1 exp(y j )

36 Loss functions L(w) = 1 N N l(w, x i, t i ) + R(w) i=1 Softmax (Probability of class k given input): p(c k = 1 x) = exp(y k ) Cj=1 exp(y j ) Cross entropy (most popular loss function for classification): l(w, x, t) = C t (k) log p(c k x) k=1

37 Loss functions L(w) = 1 N N l(w, x i, t i ) + R(w) i=1 Softmax (Probability of class k given input): p(c k = 1 x) = exp(y k ) Cj=1 exp(y j ) Cross entropy (most popular loss function for classification): l(w, x, t) = C t (k) log p(c k x) k=1 Gradient descent to train the network w = argmin L(w) w

38 Backpropagation Efficient way of computing gradient (Chain rule) Partial derivatives and gradients f (x, y) = xy f x = y df (x) f (x + h) f (x) = lim dx h 0 h f y = x df (x) f (x + h) = f (x) + h dx

39 Backpropagation Efficient way of computing gradient (Chain rule) Partial derivatives and gradients f (x, y) = xy f x = y df (x) f (x + h) f (x) = lim dx h 0 h f y = x Example: x = 4, y = 3 f (x, y) = 12 f x = 3 f x = 4 df (x) f (x + h) = f (x) + h dx [ f f = x, f ] y

40 Backpropagation Efficient way of computing gradient (Chain rule) Partial derivatives and gradients f (x, y) = xy f x = y df (x) f (x + h) f (x) = lim dx h 0 h f y = x Example: x = 4, y = 3 f (x, y) = 12 f x = 3 f x = 4 df (x) f (x + h) = f (x) + h dx [ f f = x, f ] y Question: If I increase x by h, how would the output f change?

41 Backpropagation Compound expressions with graphics (example from F.F. Li) q = x + y f = qz f q q x = 1, q x = 1 = z, f z = q Chain rule: f x = f q q x

42 Backpropagation: Another Example

43 Backpropagation: Another Example

44 Backpropagation: Another Example

45 Backpropagation: Another Example

46 Backpropagation: Another Example

47 Backpropagation: Another Example

48 Backpropagation: Another Example

49 Backpropagation: Key Idea

50 Backpropagation: Patterns in BP Add: gradient distributor Max: gradient router Mul: gradient switcher

51 Learning via Gradient Descent Gradient descent to train the network w = argmin w 1 N At each iteration, we need to compute N l(w, x i, t i ) + R(w) i=1 w n+1 = w n γ n L(w n ) Use the backward pass to compute L(w n ) efficiently Recall that the backward pass requires the forward pass first

52 Dealing with Big Data At each iteration, we need to compute w n+1 = w n γ n L(w n ) with L(w n ) = 1 N N l(w n, x i, t i ) + R(w n ) i=1 Too expensive when having millions of training examples

53 Dealing with Big Data At each iteration, we need to compute with L(w n ) = 1 N w n+1 = w n γ n L(w n ) N l(w n, x i, t i ) + R(w n ) i=1 Too expensive when having millions of training examples Instead, approximate the gradient with a mini-batch (subset of examples: 100 1,000) - called stochastic gradient descent 1 N N i=1 l(w n, x i, t i ) 1 S l(w n, x i, t i ) i S

54 SGD with momentum Stochastic Gradient Descent update with w n+1 = w n γ n L(w n ) L(w n ) = 1 S We can use momentum l(w n, x i, t i ) i S w w γ κ + L We can also decay learning rate γ as iterations goes on

55 Fully Connected Layer

56 Locally Connected Layer

57 Locally Connected Layer

58 Convolutional Neural Networks Idea: statistics are similar at different locations (Lecun 1998) Connect each hidden unit to a small input patch and share the weight across space This is called convolution layer and the network is a convolutional neural network

59 Convolutional Neural Networks Number of filters (neurons) is considered as a new dimension (depth) Volumetric representation

60 Convolutional Neural Networks Number of filters (neurons) is considered as a new dimension (depth) Volumetric representation

61 Convolutional Neural Networks CNNs are just neural nets BUT: 1. Local connectivity

62 Convolutional Neural Networks image: 32x32x3 volume CNNs are just neural nets BUT: 1. Local connectivity before: fully connected: 32x32x3 weights now: one neuron will connect to, e.g., 5x5x3 chunk (receptive field) and only have 5x5x3 weights connectivity is: local in space (5x5 instead of 32x32) but full in depth (all 3 depth channels)

63 Convolutional Neural Networks CNNs are just neural nets BUT: 1. Local connectivity

64 Convolutional Neural Networks CNNs are just neural nets BUT: 1. Local connectivity Multiple neurons all looking at the same region of the input volume, stacked along depth.

65 Convolutional Neural Networks CNNs are just neural nets BUT: 2. Weight sharing

66 Convolutional Neural Networks CNNs are just neural nets BUT: 2. Weight sharing Weights are shared across different locations Each depth slice is called one feature map

67 Convolutional Neural Networks

68 Convolutional Neural Networks

69 Convolutional Neural Networks

70 Convolutional Neural Networks

71 Convolutional Neural Networks

72 Convolutional Neural Networks

73 Convolutional Layer: Summary Input volume of size [W1 x H1 x D1] using K neurons with receptive fields F x F and applying them at strides of S gives Output volume: [W2, H2, D2] W2 = (W1-F)/S+1, H2 = (H1-F)/S+1, D2 = K

74 Feature (Filter) Visualization

more manageable without losing too much information Increased

75 Pooling In CNNs, Conv layers are often followed by Pool layers Pooling layer: makes the representations smaller and more manageable without losing too much information Increased receptive field Most common: MAX pooling Others: average, L2 pooling

76 Improving Generalization Weight sharing (Reduce the number of parameters) Data augmentation (e.g., jittering, noise injection, tranformations) Dropout [Hinton et al.]: randomly drop units (along with their connections) from the neural network during training. Use for the fully connected layers only Regularization: Weight decay (L2, L1) Sparsity in the hidden units Multi-task learning Transfer learning

77 Improving Generalization Dropout [Hinton et al.]: randomly drop units (along with their connections) from the neural network during training. Use for the fully connected layers only

78 Optimality Common knowledge: for non-convex problems, training does not work because we get stuck to local minima

79 Optimality Common knowledge: for non-convex problems, training does not work because we get stuck to local minima Not true!!! If the size of the network is large enough, local descent can reach a global minimizer from any initialization Haeffele, Vidal. Global Optimality in Tensor Factorization, Deep Learning and Beyond, arxiv 15

80 ConvNets Typical ConvNet: Image [Conv - ReLU] (Pool) [Conv - ReLU] (Pool) FC (fully-connected) Softmax

81 ConvNets - Flavours Lenet5 (Yann Lecun 1998) Alexnet (Alex Krizhevsky et. al., 2012)

82 ConvNets - Flavours ZFnet: Clarifai (Matt Zeiler and Rob Fergus, 2013) Googlenet (Google, 2014)

83 Complexity (Alexnet)

84 Recent Trend Human: 5.1% (Karpathy), Baidu cheating ( ) %

85 Recent Trend

86 Recent Trend

87 Conclusions Deep Learning = learning hierarhical models. ConvNets are the most successful example. Leverage large labeled datasets. Optimization Don t we get stuck in local minima? No, they are all the same! In large scale applications, local minima are even less of an issue. Scaling GPUs Distributed framework (Google) Better optimization techniques Generalization on small datasets (curse of dimensionality): data augmentation weight decay dropout unsupervised learning multi-task learning

ECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference

ECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Neural Networks: A brief touch Yuejie Chi Department of Electrical and Computer Engineering Spring 2018 1/41 Outline