Introduction to Machine Learning (67577)

Introduction to Machine Learning (67577) Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Deep Learning Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 1 / 38

Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 2 / 38

Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38

Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38

Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) The true and empirical risks are L D (θ) = E (x,y) D [l(θ; (x, y))], L S(θ) = 1 m m l(θ; (x i, y i )) i=1 Assumption: l is differentiable w.r.t. θ and we can calculate l(θ; (x, y)) efficiently Minimize L D or L S with Stochastic Gradient Descent (SGD): Start with θ (0) and update θ (t+1) = θ (t) η t l(θ (t) ; (x, y)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38

Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) The true and empirical risks are L D (θ) = E (x,y) D [l(θ; (x, y))], L S(θ) = 1 m m l(θ; (x i, y i )) i=1 Assumption: l is differentiable w.r.t. θ and we can calculate l(θ; (x, y)) efficiently Minimize L D or L S with Stochastic Gradient Descent (SGD): Start with θ (0) and update θ (t+1) = θ (t) η t l(θ (t) ; (x, y)) SGD converges for convex problems. It may work for non-convex problems if we initialize close enough to a good minimum Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38

Computation Graph A computation graph for a one dimensional Least Squares (numbering of nodes corresponds to topological sort): 5 Squared layer: s = r 2 4 Subtract layer: r = p y 1 Input layer: y 3 Linear layer: p = wx 0 Input layer: x 2 Variable layer: w Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 5 / 38

Gradient Calculation using the Chain Rule Fix x, y and write l as a function of w by l(w) = s(r y (p x (w))) = (s r y p x )(w). Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 6 / 38

Gradient Calculation using the Chain Rule Fix x, y and write l as a function of w by l(w) = s(r y (p x (w))) = (s r y p x )(w). Chain rule: l (w) = (s r y p x ) (w) = s (r y (p x (w))) (r y p x ) (w) = s (r y (p x (w))) r y(p x (w)) p x(w) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 6 / 38

Gradient Calculation using the Chain Rule Fix x, y and write l as a function of w by Chain rule: l(w) = s(r y (p x (w))) = (s r y p x )(w). l (w) = (s r y p x ) (w) = s (r y (p x (w))) (r y p x ) (w) = s (r y (p x (w))) r y(p x (w)) p x(w) Backpropagation: Calculate by a Forward-Backward pass over the graph Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 6 / 38

Computation Graph Forward For t = 0, 1,..., T 1 Layer[t]->output = Layer[t]->function(Layer[t]->inputs) 5 Squared layer: s = r 2 4 Subtract layer: r = p y 1 Input layer: y 3 Linear layer: p = wx 0 Input layer: x 2 Variable layer: w Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 7 / 38

Computation Graph Backward Recall: l (w) = s (r y (p x (w))) r y(p x (w)) p x(w) Layer[T-1]->delta = 1 For t = T 1, T 2,..., 0 For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) 5 Squared layer: s = r 2 4 Subtract layer: r = p y 1 Input layer: y 3 Linear layer: p = wx 0 Input layer: x 2 Variable layer: w Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 8 / 38

Layers Nodes in the computation graph are often called layers Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Logistic loss: f(x, y) = log(1 + exp( y ix i)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Logistic loss: f(x, y) = log(1 + exp( y ix i)) Main message Computation graph enables us to construct very complicated functions from simple building blocks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38

Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) derivative is the Jacobian matrix: The Jacobian of f : R n R m at x R n, denoted J x (f), is the m n matrix whose i, j element is the partial derivative of f i : R n R w.r.t. its j th variable at x. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38

Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) derivative is the Jacobian matrix: The Jacobian of f : R n R m at x R n, denoted J x (f), is the m n matrix whose i, j element is the partial derivative of f i : R n R w.r.t. its j th variable at x. The multiplication is matrix multiplication Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38

Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) derivative is the Jacobian matrix: The Jacobian of f : R n R m at x R n, denoted J x (f), is the m n matrix whose i, j element is the partial derivative of f i : R n R w.r.t. its j th variable at x. The multiplication is matrix multiplication The correctness of the algorithm follows from the multivariate chain rule J w (f g) = J g(w) (f)j w (g) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38

Jacobian Examples If f : R n R n is element-wise application of σ : R R then J x (f) = diag((σ (x 1 ),..., σ (x n ))). Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 11 / 38

Jacobian Examples If f : R n R n is element-wise application of σ : R R then J x (f) = diag((σ (x 1 ),..., σ (x n ))). Let f(x, w, b) = w x + b for w, x R n, b R 1. Then: J x (f) = w, J w (f) = x, J b (f) = 1 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 11 / 38

Jacobian Examples If f : R n R n is element-wise application of σ : R R then J x (f) = diag((σ (x 1 ),..., σ (x n ))). Let f(x, w, b) = w x + b for w, x R n, b R 1. Then: J x (f) = w, J w (f) = x, J b (f) = 1 Let f(w, x) = W x. Then: J x (f) = W, J W (f) = x 0 0 0 x 0....... 0 0 x Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 11 / 38

Sample Complexity If we learn d parameters, and each one is stored in, say, 32 bits, then the number of hypotheses in our class is at most 2 32d. It follows that the sample complexity is order of d. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 13 / 38

Sample Complexity If we learn d parameters, and each one is stored in, say, 32 bits, then the number of hypotheses in our class is at most 2 32d. It follows that the sample complexity is order of d. Other ways to improve generalization is all sort of regularization Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 13 / 38

Expressiveness So far in the course we considered hypotheses of the form x w x + b Now, consider the following computation graph, known as one hidden layer network : 9 Loss layer 1 Input layer: y 4 Variable layer: W (2) 8 Affine layer: p = W (2) h (1) + b (2) 5 Variable layer: b (2) 7 ReLU layer: h (1) = [a (1) ] + 2 Variable layer: W (1) 6 Affine layer: a (1) = W (1) x + b (1) 3 Variable layer: b (1) 0 Input layer: x Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 14 / 38

Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38

Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38

Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Theorem: For every n, let s(n) be the minimal integer such that there exists a one hidden layer network with s(n) hidden neurons that implements all functions from {0, 1} n to {0, 1}. Then, s(n) is exponential in n. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38

Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Theorem: For every n, let s(n) be the minimal integer such that there exists a one hidden layer network with s(n) hidden neurons that implements all functions from {0, 1} n to {0, 1}. Then, s(n) is exponential in n. Proof: Think on the VC dimension... Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38

Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Theorem: For every n, let s(n) be the minimal integer such that there exists a one hidden layer network with s(n) hidden neurons that implements all functions from {0, 1} n to {0, 1}. Then, s(n) is exponential in n. Proof: Think on the VC dimension... What type of functions can be implemented by small size networks? Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38

Geometric Intuition One hidden layer networks can express intersection of halfspaces Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 16 / 38

Geometric Intuition Two hidden layer networks can express unions of intersection of halfspaces Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 17 / 38

What can we express with T -depth networks? Theorem: Let T : N N and for every n, let F n be the set of functions that can be implemented using a Turing machine using runtime of at most T (n). Then, there exist constants b, c R + such that for every n, there is a network of depth at most T and size at most c T (n) 2 + b such that it implements all functions in F n. Sample complexity is order of number of variables (in our case polynomial in T ) Conclusion: A very weak notion of prior knowledge suffices if we only care about functions that can be implemented in time T (n), we can use neural networks of depth T and size O(T (n) 2 ), and the sample complexity is also bounded by polynomial in T (n)! Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 18 / 38

The ultimate hypothesis class expert system use prior knowledge to construct φ(x) and learn w, φ(x) net- deep works less prior knowledge more data No Free Lunch Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 19 / 38

Runtime of learning neural networks Theorem: It is NP hard to implement the ERM rule even for one hidden layer networks with just 4 neurons in the hidden layer. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 21 / 38

Runtime of learning neural networks Theorem: It is NP hard to implement the ERM rule even for one hidden layer networks with just 4 neurons in the hidden layer. But, maybe ERM is hard but some improper algorithm works? Theorem: Under some average case complexity assumption, it is hard to learn one hidden layer networks with ω(log(d)) hidden neurons even improperly Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 21 / 38

How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38

How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Often (but not always) still works fine, finds a good solution Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38

How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Often (but not always) still works fine, finds a good solution Easier than optimizing over Python programs... Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38

How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Often (but not always) still works fine, finds a good solution Easier than optimizing over Python programs... Need to apply some tricks (initialization, learning rate, mini-batching, architecture), and need some luck Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38

Deep Learning Golden age in Vision 2012-2014 Imagenet results: CNN non-cnn 2015 results: MSRA under 3.5% error. (using a CNN with 150 layers!) figures from Yann LeCun s CVPR 15 plenary Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 24 / 38

Convolution Layer Input: C images Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38

Convolution Layer Input: C images Output: C images Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38

Convolution Layer Input: C images Output: C images Calculation: O[c, h, w ] = b (c ) + C 1 k 1 k 1 c=0 h=0 w=0 W (c ) [c, h, w] X[c, h + h, w + w ] Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38

Convolution Layer Input: C images Output: C images Calculation: O[c, h, w ] = b (c ) + C 1 k 1 k 1 c=0 h=0 w=0 W (c ) [c, h, w] X[c, h + h, w + w ] Observe: equivalent to an Affine layer with weight sharing Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38

Convolution Layer Input: C images Output: C images Calculation: O[c, h, w ] = b (c ) + C 1 k 1 k 1 c=0 h=0 w=0 W (c ) [c, h, w] X[c, h + h, w + w ] Observe: equivalent to an Affine layer with weight sharing Observe: can be implemented as a combination of Im2Col layer and Affine layer Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38

Im2Col Layer Im2Col for 3 3 convolution 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 0 0 1 0 5 6 0 10 11 0 1 2 5 6 7 10 11 12 1 2 3 6 7 8 11 12 13 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 26 / 38

Im2Col Layer Im2Col for 3 3 convolution with 2 input channels Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 27 / 38

Parameters of Convolutions layer Kernel height and kernel width Stride height and stride width zero padding (True or False) Number of output channels Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 28 / 38

Pooling Layer Input: Image of size H W Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38

Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38

Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Calculation: Divide input image to k k windows and for each such window output the maximal value (or average value) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38

Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Calculation: Divide input image to k k windows and for each such window output the maximal value (or average value) Observe: equivalent to Im2Col + reduce operation Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38

Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Calculation: Divide input image to k k windows and for each such window output the maximal value (or average value) Observe: equivalent to Im2Col + reduce operation Discuss: how to calculate derivative? Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Output space: Y = {0, 1,..., 9} Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Output space: Y = {0, 1,..., 9} Multiclass categorization: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Logistic loss for multiclass categorization: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Logistic loss for multiclass categorization: SoftMax: i, p i = exp(hi(x)) j exp(hj(x)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} 28 28 Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Logistic loss for multiclass categorization: SoftMax: i, p i = exp(hi(x)) j exp(hj(x)) LogLoss: If the correct ( label is y then the loss is ) log(p y ) = log j exp(h j(x) h i (x)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38

Reduction Layers The complexity of convolutional layers is C in C out H W A reduction layer is a 1 1 convolution aiming at reducing C in It can greatly reduce the computational complexity (less time) and sample complexity (fewer parameters) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 33 / 38

Inception modules Filter concatenation 1x1 convolutions 3x3 convolutions 5x5 convolutions 3x3 max pooling Previous layer (a) Inception module, naïve version Filter concatenation 3x3 convolutions 5x5 convolutions 1x1 convolutions 1x1 convolutions 1x1 convolutions 1x1 convolutions 3x3 max pooling Previous layer (b) Inception module with dimensionality reduction Figure 2: Inception module ber of filters in the previous stage. The merging of output of the pooling layer with outputs of the convolutional layers would lead to an inevitable increase in the number of outputs from stage to stage. While this architecture might cover the optimal sparse structure, it would do it very inefficiently, leading to a computational blow up within a few stages. This leads to the second idea of the Inception architecture: judiciously reducing dimension wherever the compuefficiency during training), it seemed beneficial to sta ing Inception modules only at higher layers while ke the lower layers in traditional convolutional fashion. T not strictly necessary, simply reflecting some infrastru inefficiencies in our current implementation. A useful aspect of this architecture is that it allow increasing the number of units at each stage signific without an uncontrolled blow-up in computational plexity at later stages. This is achieved by the ubiqu use of dimensionality reduction prior to expensive con tions with larger patch sizes. Furthermore, the desig lows the practical intuition that visual information s be processed at various scales and then aggregated s the next stage can abstract features from the different s simultaneously. The improved use of computational resources allow increasing both the width of each stage as well as the ber of stages without getting into computational difficu One can utilize the Inception architecture to create sl inferior, but computationally cheaper versions of it. have found that all the available knobs and levers allo a controlled balancing of computational resources res in networks that are 3 10 faster than similarly per ing networks with non-inception architecture, howeve requires careful manual design at this point. Szegedy et al (Google) Won the ImageNet 2014 challenge (6.67% error) 5. GoogLeNet By the GoogLeNet name we refer to the particul carnation of the Inception architecture used in our su sion for the ILSVRC 2014 competition. We also use deeper and wider Inception network with slightly sup quality, but adding it to the ensemble seemed to improv results only marginally. We omit the details of that net as empirical evidence suggests that the influence of th act architectural parameters is relatively minor. Table Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 34 / 38

Residual Networks x weight layer F(x) relu x weight layer identity F(x) + x relu Figure 2. Residual learning: a building block. are comparably good or better than the constructed solution He, Zhang, Ren, Sun (Microsoft) Won the ImageNet 2015 challengetime). with a 152 layers network (3.57% (or unable to do so in feasible error) In this paper, we address the degradation problem by Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 35 / 38

Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38

Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster We don t pay a lot in time because of parallel implementation Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38

Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster We don t pay a lot in time because of parallel implementation Learning rate: Choice of learning rate is important. One way is to start with some fixed η and decrease it by 1/2 whenever the training stops making progress. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38

Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster We don t pay a lot in time because of parallel implementation Learning rate: Choice of learning rate is important. One way is to start with some fixed η and decrease it by 1/2 whenever the training stops making progress. Variants of SGD: There are plenty of variants that work better than vanilla SGD. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38

Failures of Deep Learning Parity of more than 30 bits Multiplication of large numbers Matrix inversion... Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 37 / 38

Summary Deep Learning can be used to construct the ultimate hypothesis class Worst-case complexity is exponential... but, empirically, it works reasonably well and leads to state-of-the-art on many real world problems Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 38 / 38