Introduction to Machine Learning (67577)
|
|
- Edwin Palmer
- 6 years ago
- Views:
Transcription
1 Introduction to Machine Learning (67577) Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Deep Learning Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 1 / 38
2 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 2 / 38
3 Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38
4 Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38
5 Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) The true and empirical risks are L D (θ) = E (x,y) D [l(θ; (x, y))], L S(θ) = 1 m m l(θ; (x i, y i )) i=1 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38
6 Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) The true and empirical risks are L D (θ) = E (x,y) D [l(θ; (x, y))], L S(θ) = 1 m m l(θ; (x i, y i )) i=1 Assumption: l is differentiable w.r.t. θ and we can calculate l(θ; (x, y)) efficiently Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38
7 Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) The true and empirical risks are L D (θ) = E (x,y) D [l(θ; (x, y))], L S(θ) = 1 m m l(θ; (x i, y i )) i=1 Assumption: l is differentiable w.r.t. θ and we can calculate l(θ; (x, y)) efficiently Minimize L D or L S with Stochastic Gradient Descent (SGD): Start with θ (0) and update θ (t+1) = θ (t) η t l(θ (t) ; (x, y)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38
8 Gradient-Based Learning Consider a hypothesis class which is parameterized by a vector θ R d Loss function of h θ on example (x, y) is denoted l(θ; (x, y)) The true and empirical risks are L D (θ) = E (x,y) D [l(θ; (x, y))], L S(θ) = 1 m m l(θ; (x i, y i )) i=1 Assumption: l is differentiable w.r.t. θ and we can calculate l(θ; (x, y)) efficiently Minimize L D or L S with Stochastic Gradient Descent (SGD): Start with θ (0) and update θ (t+1) = θ (t) η t l(θ (t) ; (x, y)) SGD converges for convex problems. It may work for non-convex problems if we initialize close enough to a good minimum Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 3 / 38
9 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 4 / 38
10 Computation Graph A computation graph for a one dimensional Least Squares (numbering of nodes corresponds to topological sort): 5 Squared layer: s = r 2 4 Subtract layer: r = p y 1 Input layer: y 3 Linear layer: p = wx 0 Input layer: x 2 Variable layer: w Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 5 / 38
11 Gradient Calculation using the Chain Rule Fix x, y and write l as a function of w by l(w) = s(r y (p x (w))) = (s r y p x )(w). Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 6 / 38
12 Gradient Calculation using the Chain Rule Fix x, y and write l as a function of w by l(w) = s(r y (p x (w))) = (s r y p x )(w). Chain rule: l (w) = (s r y p x ) (w) = s (r y (p x (w))) (r y p x ) (w) = s (r y (p x (w))) r y(p x (w)) p x(w) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 6 / 38
13 Gradient Calculation using the Chain Rule Fix x, y and write l as a function of w by Chain rule: l(w) = s(r y (p x (w))) = (s r y p x )(w). l (w) = (s r y p x ) (w) = s (r y (p x (w))) (r y p x ) (w) = s (r y (p x (w))) r y(p x (w)) p x(w) Backpropagation: Calculate by a Forward-Backward pass over the graph Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 6 / 38
14 Computation Graph Forward For t = 0, 1,..., T 1 Layer[t]->output = Layer[t]->function(Layer[t]->inputs) 5 Squared layer: s = r 2 4 Subtract layer: r = p y 1 Input layer: y 3 Linear layer: p = wx 0 Input layer: x 2 Variable layer: w Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 7 / 38
15 Computation Graph Backward Recall: l (w) = s (r y (p x (w))) r y(p x (w)) p x(w) Layer[T-1]->delta = 1 For t = T 1, T 2,..., 0 For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) 5 Squared layer: s = r 2 4 Subtract layer: r = p y 1 Input layer: y 3 Linear layer: p = wx 0 Input layer: x 2 Variable layer: w Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 8 / 38
16 Layers Nodes in the computation graph are often called layers Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
17 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
18 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
19 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
20 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
21 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
22 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
23 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
24 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
25 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
26 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
27 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Logistic loss: f(x, y) = log(1 + exp( y ix i)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
28 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Logistic loss: f(x, y) = log(1 + exp( y ix i)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
29 Layers Nodes in the computation graph are often called layers Each layer is a simple differentiable function Layers can implement multivariate functions Example of popular layers: Affine layer: O = W X + b 1 where W R m,n, x R n,c, b R m Unary layer: i, o i = f(x i ) for some f : R R e.g. Sigmoid: f(x) = (1 + exp( x)) 1 Rectified Linear Unit (ReLU): f(x) = max{0, x} (discuss: derivative?) Binary layer: i, o i = f(x i, y i ) for some f : R 2 R e.g. Add layer: f(x, y) = x + y Hinge loss: f(x, y) = [1 y ix i] + Logistic loss: f(x, y) = log(1 + exp( y ix i)) Main message Computation graph enables us to construct very complicated functions from simple building blocks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 9 / 38
30 Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38
31 Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38
32 Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) derivative is the Jacobian matrix: The Jacobian of f : R n R m at x R n, denoted J x (f), is the m n matrix whose i, j element is the partial derivative of f i : R n R w.r.t. its j th variable at x. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38
33 Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) derivative is the Jacobian matrix: The Jacobian of f : R n R m at x R n, denoted J x (f), is the m n matrix whose i, j element is the partial derivative of f i : R n R w.r.t. its j th variable at x. The multiplication is matrix multiplication Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38
34 Backpropgation for multivariate layers Recall the backpropagation rule: For i in Layer[t]->inputs: i->delta = Layer[t]->delta * Layer[t]->derivative(i,Layer[t]->inputs) delta is now a vector (same dimension as the output of the layer) derivative is the Jacobian matrix: The Jacobian of f : R n R m at x R n, denoted J x (f), is the m n matrix whose i, j element is the partial derivative of f i : R n R w.r.t. its j th variable at x. The multiplication is matrix multiplication The correctness of the algorithm follows from the multivariate chain rule J w (f g) = J g(w) (f)j w (g) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 10 / 38
35 Jacobian Examples If f : R n R n is element-wise application of σ : R R then J x (f) = diag((σ (x 1 ),..., σ (x n ))). Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 11 / 38
36 Jacobian Examples If f : R n R n is element-wise application of σ : R R then J x (f) = diag((σ (x 1 ),..., σ (x n ))). Let f(x, w, b) = w x + b for w, x R n, b R 1. Then: J x (f) = w, J w (f) = x, J b (f) = 1 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 11 / 38
37 Jacobian Examples If f : R n R n is element-wise application of σ : R R then J x (f) = diag((σ (x 1 ),..., σ (x n ))). Let f(x, w, b) = w x + b for w, x R n, b R 1. Then: J x (f) = w, J w (f) = x, J b (f) = 1 Let f(w, x) = W x. Then: J x (f) = W, J W (f) = x x x Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 11 / 38
38 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 12 / 38
39 Sample Complexity If we learn d parameters, and each one is stored in, say, 32 bits, then the number of hypotheses in our class is at most 2 32d. It follows that the sample complexity is order of d. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 13 / 38
40 Sample Complexity If we learn d parameters, and each one is stored in, say, 32 bits, then the number of hypotheses in our class is at most 2 32d. It follows that the sample complexity is order of d. Other ways to improve generalization is all sort of regularization Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 13 / 38
41 Expressiveness So far in the course we considered hypotheses of the form x w x + b Now, consider the following computation graph, known as one hidden layer network : 9 Loss layer 1 Input layer: y 4 Variable layer: W (2) 8 Affine layer: p = W (2) h (1) + b (2) 5 Variable layer: b (2) 7 ReLU layer: h (1) = [a (1) ] + 2 Variable layer: W (1) 6 Affine layer: a (1) = W (1) x + b (1) 3 Variable layer: b (1) 0 Input layer: x Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 14 / 38
42 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
43 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
44 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
45 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
46 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
47 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
48 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Theorem: For every n, let s(n) be the minimal integer such that there exists a one hidden layer network with s(n) hidden neurons that implements all functions from {0, 1} n to {0, 1}. Then, s(n) is exponential in n. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
49 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Theorem: For every n, let s(n) be the minimal integer such that there exists a one hidden layer network with s(n) hidden neurons that implements all functions from {0, 1} n to {0, 1}. Then, s(n) is exponential in n. Proof: Think on the VC dimension... Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
50 Expressiveness of One Hidden Layer Network Claim: Every Boolean function f : {±1} n {±1} can be expressed by a one hidden layer network. Proof: Show that for integer x we have sign(x) = 2([x + 1] + [x] + ) 1 Show that any f can be written as f(x) = i (x == u i ) for some vectors u 1,..., u k Show that sign(x u i (n 1)) is an indicator to (x == u i ) Conclude that we can adjust the weights so that yp(x) 1 for all examples (x, y) Theorem: For every n, let s(n) be the minimal integer such that there exists a one hidden layer network with s(n) hidden neurons that implements all functions from {0, 1} n to {0, 1}. Then, s(n) is exponential in n. Proof: Think on the VC dimension... What type of functions can be implemented by small size networks? Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 15 / 38
51 Geometric Intuition One hidden layer networks can express intersection of halfspaces Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 16 / 38
52 Geometric Intuition Two hidden layer networks can express unions of intersection of halfspaces Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 17 / 38
53 What can we express with T -depth networks? Theorem: Let T : N N and for every n, let F n be the set of functions that can be implemented using a Turing machine using runtime of at most T (n). Then, there exist constants b, c R + such that for every n, there is a network of depth at most T and size at most c T (n) 2 + b such that it implements all functions in F n. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 18 / 38
54 What can we express with T -depth networks? Theorem: Let T : N N and for every n, let F n be the set of functions that can be implemented using a Turing machine using runtime of at most T (n). Then, there exist constants b, c R + such that for every n, there is a network of depth at most T and size at most c T (n) 2 + b such that it implements all functions in F n. Sample complexity is order of number of variables (in our case polynomial in T ) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 18 / 38
55 What can we express with T -depth networks? Theorem: Let T : N N and for every n, let F n be the set of functions that can be implemented using a Turing machine using runtime of at most T (n). Then, there exist constants b, c R + such that for every n, there is a network of depth at most T and size at most c T (n) 2 + b such that it implements all functions in F n. Sample complexity is order of number of variables (in our case polynomial in T ) Conclusion: A very weak notion of prior knowledge suffices if we only care about functions that can be implemented in time T (n), we can use neural networks of depth T and size O(T (n) 2 ), and the sample complexity is also bounded by polynomial in T (n)! Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 18 / 38
56 The ultimate hypothesis class expert system use prior knowledge to construct φ(x) and learn w, φ(x) net- deep works less prior knowledge more data No Free Lunch Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 19 / 38
57 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 20 / 38
58 Runtime of learning neural networks Theorem: It is NP hard to implement the ERM rule even for one hidden layer networks with just 4 neurons in the hidden layer. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 21 / 38
59 Runtime of learning neural networks Theorem: It is NP hard to implement the ERM rule even for one hidden layer networks with just 4 neurons in the hidden layer. But, maybe ERM is hard but some improper algorithm works? Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 21 / 38
60 Runtime of learning neural networks Theorem: It is NP hard to implement the ERM rule even for one hidden layer networks with just 4 neurons in the hidden layer. But, maybe ERM is hard but some improper algorithm works? Theorem: Under some average case complexity assumption, it is hard to learn one hidden layer networks with ω(log(d)) hidden neurons even improperly Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 21 / 38
61 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
62 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
63 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
64 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
65 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Often (but not always) still works fine, finds a good solution Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
66 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Often (but not always) still works fine, finds a good solution Easier than optimizing over Python programs... Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
67 How to train neural network? So, neural networks can form an excellent hypothesis class, but it is intractable to train it. How is this different than the class of all Python programs that can be implemented in code length of b bits? Main technique: Gradient-based learning (using SGD) Not convex, no guarantees, can take a long time, but: Often (but not always) still works fine, finds a good solution Easier than optimizing over Python programs... Need to apply some tricks (initialization, learning rate, mini-batching, architecture), and need some luck Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 22 / 38
68 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 23 / 38
69 Deep Learning Golden age in Vision Imagenet results: CNN non-cnn 2015 results: MSRA under 3.5% error. (using a CNN with 150 layers!) figures from Yann LeCun s CVPR 15 plenary Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 24 / 38
70 Convolution Layer Input: C images Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38
71 Convolution Layer Input: C images Output: C images Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38
72 Convolution Layer Input: C images Output: C images Calculation: O[c, h, w ] = b (c ) + C 1 k 1 k 1 c=0 h=0 w=0 W (c ) [c, h, w] X[c, h + h, w + w ] Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38
73 Convolution Layer Input: C images Output: C images Calculation: O[c, h, w ] = b (c ) + C 1 k 1 k 1 c=0 h=0 w=0 W (c ) [c, h, w] X[c, h + h, w + w ] Observe: equivalent to an Affine layer with weight sharing Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38
74 Convolution Layer Input: C images Output: C images Calculation: O[c, h, w ] = b (c ) + C 1 k 1 k 1 c=0 h=0 w=0 W (c ) [c, h, w] X[c, h + h, w + w ] Observe: equivalent to an Affine layer with weight sharing Observe: can be implemented as a combination of Im2Col layer and Affine layer Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 25 / 38
75 Im2Col Layer Im2Col for 3 3 convolution Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 26 / 38
76 Im2Col Layer Im2Col for 3 3 convolution with 2 input channels Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 27 / 38
77 Parameters of Convolutions layer Kernel height and kernel width Stride height and stride width zero padding (True or False) Number of output channels Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 28 / 38
78 Pooling Layer Input: Image of size H W Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38
79 Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38
80 Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Calculation: Divide input image to k k windows and for each such window output the maximal value (or average value) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38
81 Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Calculation: Divide input image to k k windows and for each such window output the maximal value (or average value) Observe: equivalent to Im2Col + reduce operation Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38
82 Pooling Layer Input: Image of size H W Output: Image of size (H/k) (W/k) Calculation: Divide input image to k k windows and for each such window output the maximal value (or average value) Observe: equivalent to Im2Col + reduce operation Discuss: how to calculate derivative? Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 29 / 38
83 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 30 / 38
84 Example: LeNet for MNIST The task: Handwritten digits recognition Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
85 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
86 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
87 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
88 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
89 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
90 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
91 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
92 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Logistic loss for multiclass categorization: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
93 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Logistic loss for multiclass categorization: SoftMax: i, p i = exp(hi(x)) j exp(hj(x)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
94 Example: LeNet for MNIST The task: Handwritten digits recognition Input space: X = {0, 1,..., 255} Output space: Y = {0, 1,..., 9} Multiclass categorization: We take hypotheses of the form h : X R Y We interpret h(x) as a vector that gives scores for all the labels The actual prediction is the label with the highest score: argmax i h i (x) Network architecture: x Conv(5x5,1x1,no-pad,20) Pool(2x2) Conv(5x5,1x1,no-pad,50) Pool(2x2) Affine(500) ReLU Affine(10). Logistic loss for multiclass categorization: SoftMax: i, p i = exp(hi(x)) j exp(hj(x)) LogLoss: If the correct ( label is y then the loss is ) log(p y ) = log j exp(h j(x) h i (x)) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 31 / 38
95 Outline 1 Gradient-Based Learning 2 Computation Graph and Backpropagation 3 Expressiveness and Sample Complexity 4 Computational Complexity 5 Convolutional Networks 6 Solving MNIST with LeNet using Tensorflow 7 Tips and Tricks Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 32 / 38
96 Reduction Layers The complexity of convolutional layers is C in C out H W A reduction layer is a 1 1 convolution aiming at reducing C in It can greatly reduce the computational complexity (less time) and sample complexity (fewer parameters) Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 33 / 38
97 Inception modules Filter concatenation 1x1 convolutions 3x3 convolutions 5x5 convolutions 3x3 max pooling Previous layer (a) Inception module, naïve version Filter concatenation 3x3 convolutions 5x5 convolutions 1x1 convolutions 1x1 convolutions 1x1 convolutions 1x1 convolutions 3x3 max pooling Previous layer (b) Inception module with dimensionality reduction Figure 2: Inception module ber of filters in the previous stage. The merging of output of the pooling layer with outputs of the convolutional layers would lead to an inevitable increase in the number of outputs from stage to stage. While this architecture might cover the optimal sparse structure, it would do it very inefficiently, leading to a computational blow up within a few stages. This leads to the second idea of the Inception architecture: judiciously reducing dimension wherever the compuefficiency during training), it seemed beneficial to sta ing Inception modules only at higher layers while ke the lower layers in traditional convolutional fashion. T not strictly necessary, simply reflecting some infrastru inefficiencies in our current implementation. A useful aspect of this architecture is that it allow increasing the number of units at each stage signific without an uncontrolled blow-up in computational plexity at later stages. This is achieved by the ubiqu use of dimensionality reduction prior to expensive con tions with larger patch sizes. Furthermore, the desig lows the practical intuition that visual information s be processed at various scales and then aggregated s the next stage can abstract features from the different s simultaneously. The improved use of computational resources allow increasing both the width of each stage as well as the ber of stages without getting into computational difficu One can utilize the Inception architecture to create sl inferior, but computationally cheaper versions of it. have found that all the available knobs and levers allo a controlled balancing of computational resources res in networks that are 3 10 faster than similarly per ing networks with non-inception architecture, howeve requires careful manual design at this point. Szegedy et al (Google) Won the ImageNet 2014 challenge (6.67% error) 5. GoogLeNet By the GoogLeNet name we refer to the particul carnation of the Inception architecture used in our su sion for the ILSVRC 2014 competition. We also use deeper and wider Inception network with slightly sup quality, but adding it to the ensemble seemed to improv results only marginally. We omit the details of that net as empirical evidence suggests that the influence of th act architectural parameters is relatively minor. Table Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 34 / 38
98 Residual Networks x weight layer F(x) relu x weight layer identity F(x) + x relu Figure 2. Residual learning: a building block. are comparably good or better than the constructed solution He, Zhang, Ren, Sun (Microsoft) Won the ImageNet 2015 challengetime). with a 152 layers network (3.57% (or unable to do so in feasible error) In this paper, we address the degradation problem by Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 35 / 38
99 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
100 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
101 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
102 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
103 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster We don t pay a lot in time because of parallel implementation Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
104 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster We don t pay a lot in time because of parallel implementation Learning rate: Choice of learning rate is important. One way is to start with some fixed η and decrease it by 1/2 whenever the training stops making progress. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
105 Some Training Tricks Input normalization: divide each element of x by 255 to make sure it is in [0, 1] Initialization is important: One trick that works well in practice is to initialize the bias to be zero and initialize the rows of W to be random in [ 1/ n, 1/ n] Mini-batches: At each iteration of SGD we calculate the average loss on k random examples for k > 1. Advantages: Reduces the variance of the update direction (w.r.t. the full gradient), hence converges faster We don t pay a lot in time because of parallel implementation Learning rate: Choice of learning rate is important. One way is to start with some fixed η and decrease it by 1/2 whenever the training stops making progress. Variants of SGD: There are plenty of variants that work better than vanilla SGD. Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 36 / 38
106 Failures of Deep Learning Parity of more than 30 bits Multiplication of large numbers Matrix inversion... Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 37 / 38
107 Summary Deep Learning can be used to construct the ultimate hypothesis class Worst-case complexity is exponential... but, empirically, it works reasonably well and leads to state-of-the-art on many real world problems Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks 38 / 38
SGD and Deep Learning
SGD and Deep Learning Subgradients Lets make the gradient cheating more formal. Recall that the gradient is the slope of the tangent. f(w 1 )+rf(w 1 ) (w w 1 ) Non differentiable case? w 1 Subgradients
More informationIntroduction to Machine Learning (67577) Lecture 3
Introduction to Machine Learning (67577) Lecture 3 Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem General Learning Model and Bias-Complexity tradeoff Shai Shalev-Shwartz
More informationAccelerating Stochastic Optimization
Accelerating Stochastic Optimization Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem and Mobileye Master Class at Tel-Aviv, Tel-Aviv University, November 2014 Shalev-Shwartz
More informationDeep Learning (CNNs)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Deep Learning (CNNs) Deep Learning Readings: Murphy 28 Bishop - - HTF - - Mitchell
More informationNeural networks COMS 4771
Neural networks COMS 4771 1. Logistic regression Logistic regression Suppose X = R d and Y = {0, 1}. A logistic regression model is a statistical model where the conditional probability function has a
More informationNeural Networks and Deep Learning
Neural Networks and Deep Learning Professor Ameet Talwalkar November 12, 2015 Professor Ameet Talwalkar Neural Networks and Deep Learning November 12, 2015 1 / 16 Outline 1 Review of last lecture AdaBoost
More informationIntroduction to Convolutional Neural Networks (CNNs)
Introduction to Convolutional Neural Networks (CNNs) nojunk@snu.ac.kr http://mipal.snu.ac.kr Department of Transdisciplinary Studies Seoul National University, Korea Jan. 2016 Many slides are from Fei-Fei
More informationLecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
More informationIntroduction to Machine Learning (67577) Lecture 7
Introduction to Machine Learning (67577) Lecture 7 Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Solving Convex Problems using SGD and RLM Shai Shalev-Shwartz (Hebrew
More informationFeedforward Neural Networks
Feedforward Neural Networks Michael Collins 1 Introduction In the previous notes, we introduced an important class of models, log-linear models. In this note, we describe feedforward neural networks, which
More informationComments. Assignment 3 code released. Thought questions 3 due this week. Mini-project: hopefully you have started. implement classification algorithms
Neural networks Comments Assignment 3 code released implement classification algorithms use kernels for census dataset Thought questions 3 due this week Mini-project: hopefully you have started 2 Example:
More informationNeural Networks Learning the network: Backprop , Fall 2018 Lecture 4
Neural Networks Learning the network: Backprop 11-785, Fall 2018 Lecture 4 1 Recap: The MLP can represent any function The MLP can be constructed to represent anything But how do we construct it? 2 Recap:
More informationCSCI567 Machine Learning (Fall 2018)
CSCI567 Machine Learning (Fall 2018) Prof. Haipeng Luo U of Southern California Sep 12, 2018 September 12, 2018 1 / 49 Administration GitHub repos are setup (ask TA Chi Zhang for any issues) HW 1 is due
More informationConvolutional Neural Networks. Srikumar Ramalingam
Convolutional Neural Networks Srikumar Ramalingam Reference Many of the slides are prepared using the following resources: neuralnetworksanddeeplearning.com (mainly Chapter 6) http://cs231n.github.io/convolutional-networks/
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationEfficiently Training Sum-Product Neural Networks using Forward Greedy Selection
Efficiently Training Sum-Product Neural Networks using Forward Greedy Selection Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Greedy Algorithms, Frank-Wolfe and Friends
More informationNeural Networks. Nicholas Ruozzi University of Texas at Dallas
Neural Networks Nicholas Ruozzi University of Texas at Dallas Handwritten Digit Recognition Given a collection of handwritten digits and their corresponding labels, we d like to be able to correctly classify
More informationIntroduction to (Convolutional) Neural Networks
Introduction to (Convolutional) Neural Networks Philipp Grohs Summer School DL and Vis, Sept 2018 Syllabus 1 Motivation and Definition 2 Universal Approximation 3 Backpropagation 4 Stochastic Gradient
More informationFailures of Gradient-Based Deep Learning
Failures of Gradient-Based Deep Learning Shai Shalev-Shwartz, Shaked Shammah, Ohad Shamir The Hebrew University and Mobileye Representation Learning Workshop Simons Institute, Berkeley, 2017 Shai Shalev-Shwartz
More informationLecture 7 Convolutional Neural Networks
Lecture 7 Convolutional Neural Networks CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 17, 2017 We saw before: ŷ x 1 x 2 x 3 x 4 A series of matrix multiplications:
More informationJakub Hajic Artificial Intelligence Seminar I
Jakub Hajic Artificial Intelligence Seminar I. 11. 11. 2014 Outline Key concepts Deep Belief Networks Convolutional Neural Networks A couple of questions Convolution Perceptron Feedforward Neural Network
More informationConvolutional Neural Networks
Convolutional Neural Networks Books» http://www.deeplearningbook.org/ Books http://neuralnetworksanddeeplearning.com/.org/ reviews» http://www.deeplearningbook.org/contents/linear_algebra.html» http://www.deeplearningbook.org/contents/prob.html»
More informationBackpropagation Introduction to Machine Learning. Matt Gormley Lecture 12 Feb 23, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Backpropagation Matt Gormley Lecture 12 Feb 23, 2018 1 Neural Networks Outline
More informationNeural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
More information1 Machine Learning Concepts (16 points)
CSCI 567 Fall 2018 Midterm Exam DO NOT OPEN EXAM UNTIL INSTRUCTED TO DO SO PLEASE TURN OFF ALL CELL PHONES Problem 1 2 3 4 5 6 Total Max 16 10 16 42 24 12 120 Points Please read the following instructions
More informationConvolutional Neural Network Architecture
Convolutional Neural Network Architecture Zhisheng Zhong Feburary 2nd, 2018 Zhisheng Zhong Convolutional Neural Network Architecture Feburary 2nd, 2018 1 / 55 Outline 1 Introduction of Convolution Motivation
More informationLecture 2: Learning with neural networks
Lecture 2: Learning with neural networks Deep Learning @ UvA LEARNING WITH NEURAL NETWORKS - PAGE 1 Lecture Overview o Machine Learning Paradigm for Neural Networks o The Backpropagation algorithm for
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationLecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets)
COS 402 Machine Learning and Artificial Intelligence Fall 2016 Lecture 8: Introduction to Deep Learning: Part 2 (More on backpropagation, and ConvNets) Sanjeev Arora Elad Hazan Recap: Structure of a deep
More informationTopics in AI (CPSC 532L): Multimodal Learning with Vision, Language and Sound. Lecture 3: Introduction to Deep Learning (continued)
Topics in AI (CPSC 532L): Multimodal Learning with Vision, Language and Sound Lecture 3: Introduction to Deep Learning (continued) Course Logistics - Update on course registrations - 6 seats left now -
More informationNeural Networks. David Rosenberg. July 26, New York University. David Rosenberg (New York University) DS-GA 1003 July 26, / 35
Neural Networks David Rosenberg New York University July 26, 2017 David Rosenberg (New York University) DS-GA 1003 July 26, 2017 1 / 35 Neural Networks Overview Objectives What are neural networks? How
More informationLecture 3 Feedforward Networks and Backpropagation
Lecture 3 Feedforward Networks and Backpropagation CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 3, 2017 Things we will look at today Recap of Logistic Regression
More informationStatistical Machine Learning (BE4M33SSU) Lecture 5: Artificial Neural Networks
Statistical Machine Learning (BE4M33SSU) Lecture 5: Artificial Neural Networks Jan Drchal Czech Technical University in Prague Faculty of Electrical Engineering Department of Computer Science Topics covered
More informationMachine Learning for Computer Vision 8. Neural Networks and Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group
Machine Learning for Computer Vision 8. Neural Networks and Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group INTRODUCTION Nonlinear Coordinate Transformation http://cs.stanford.edu/people/karpathy/convnetjs/
More informationDeep Feedforward Networks. Seung-Hoon Na Chonbuk National University
Deep Feedforward Networks Seung-Hoon Na Chonbuk National University Neural Network: Types Feedforward neural networks (FNN) = Deep feedforward networks = multilayer perceptrons (MLP) No feedback connections
More informationNeural Networks: Introduction
Neural Networks: Introduction Machine Learning Fall 2017 Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others 1
More informationLecture 3 Feedforward Networks and Backpropagation
Lecture 3 Feedforward Networks and Backpropagation CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 3, 2017 Things we will look at today Recap of Logistic Regression
More informationArtificial Neural Networks
Artificial Neural Networks Oliver Schulte - CMPT 310 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will focus on
More informationCS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS
CS 179: LECTURE 16 MODEL COMPLEXITY, REGULARIZATION, AND CONVOLUTIONAL NETS LAST TIME Intro to cudnn Deep neural nets using cublas and cudnn TODAY Building a better model for image classification Overfitting
More informationNeural Networks. Bishop PRML Ch. 5. Alireza Ghane. Feed-forward Networks Network Training Error Backpropagation Applications
Neural Networks Bishop PRML Ch. 5 Alireza Ghane Neural Networks Alireza Ghane / Greg Mori 1 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of
More informationReading Group on Deep Learning Session 1
Reading Group on Deep Learning Session 1 Stephane Lathuiliere & Pablo Mesejo 2 June 2016 1/31 Contents Introduction to Artificial Neural Networks to understand, and to be able to efficiently use, the popular
More informationMachine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
More informationBayesian Networks (Part I)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Bayesian Networks (Part I) Graphical Model Readings: Murphy 10 10.2.1 Bishop 8.1,
More informationClassification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
More informationSupport vector machines Lecture 4
Support vector machines Lecture 4 David Sontag New York University Slides adapted from Luke Zettlemoyer, Vibhav Gogate, and Carlos Guestrin Q: What does the Perceptron mistake bound tell us? Theorem: The
More informationNeural Networks 2. 2 Receptive fields and dealing with image inputs
CS 446 Machine Learning Fall 2016 Oct 04, 2016 Neural Networks 2 Professor: Dan Roth Scribe: C. Cheng, C. Cervantes Overview Convolutional Neural Networks Recurrent Neural Networks 1 Introduction There
More informationTasks ADAS. Self Driving. Non-machine Learning. Traditional MLP. Machine-Learning based method. Supervised CNN. Methods. Deep-Learning based
UNDERSTANDING CNN ADAS Tasks Self Driving Localizati on Perception Planning/ Control Driver state Vehicle Diagnosis Smart factory Methods Traditional Deep-Learning based Non-machine Learning Machine-Learning
More informationCSE446: Neural Networks Spring Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer
CSE446: Neural Networks Spring 2017 Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer Human Neurons Switching time ~ 0.001 second Number of neurons 10 10 Connections per neuron 10 4-5 Scene
More informationBits of Machine Learning Part 1: Supervised Learning
Bits of Machine Learning Part 1: Supervised Learning Alexandre Proutiere and Vahan Petrosyan KTH (The Royal Institute of Technology) Outline of the Course 1. Supervised Learning Regression and Classification
More informationDeep Feedforward Networks
Deep Feedforward Networks Yongjin Park 1 Goal of Feedforward Networks Deep Feedforward Networks are also called as Feedforward neural networks or Multilayer Perceptrons Their Goal: approximate some function
More informationFeed-forward Networks Network Training Error Backpropagation Applications. Neural Networks. Oliver Schulte - CMPT 726. Bishop PRML Ch.
Neural Networks Oliver Schulte - CMPT 726 Bishop PRML Ch. 5 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will
More informationMachine Learning in the Data Revolution Era
Machine Learning in the Data Revolution Era Shai Shalev-Shwartz School of Computer Science and Engineering The Hebrew University of Jerusalem Machine Learning Seminar Series, Google & University of Waterloo,
More informationCSE 190 Fall 2015 Midterm DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO START!!!!
CSE 190 Fall 2015 Midterm DO NOT TURN THIS PAGE UNTIL YOU ARE TOLD TO START!!!! November 18, 2015 THE EXAM IS CLOSED BOOK. Once the exam has started, SORRY, NO TALKING!!! No, you can t even say see ya
More informationActive Learning: Disagreement Coefficient
Advanced Course in Machine Learning Spring 2010 Active Learning: Disagreement Coefficient Handouts are jointly prepared by Shie Mannor and Shai Shalev-Shwartz In previous lectures we saw examples in which
More informationCSC321 Lecture 9: Generalization
CSC321 Lecture 9: Generalization Roger Grosse Roger Grosse CSC321 Lecture 9: Generalization 1 / 26 Overview We ve focused so far on how to optimize neural nets how to get them to make good predictions
More informationCS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
More informationIntelligent Systems Discriminative Learning, Neural Networks
Intelligent Systems Discriminative Learning, Neural Networks Carsten Rother, Dmitrij Schlesinger WS2014/2015, Outline 1. Discriminative learning 2. Neurons and linear classifiers: 1) Perceptron-Algorithm
More informationNeural networks and optimization
Neural networks and optimization Nicolas Le Roux Criteo 18/05/15 Nicolas Le Roux (Criteo) Neural networks and optimization 18/05/15 1 / 85 1 Introduction 2 Deep networks 3 Optimization 4 Convolutional
More informationMulticlass Classification-1
CS 446 Machine Learning Fall 2016 Oct 27, 2016 Multiclass Classification Professor: Dan Roth Scribe: C. Cheng Overview Binary to multiclass Multiclass SVM Constraint classification 1 Introduction Multiclass
More informationOnline Learning Summer School Copenhagen 2015 Lecture 1
Online Learning Summer School Copenhagen 2015 Lecture 1 Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Online Learning Shai Shalev-Shwartz (Hebrew U) OLSS Lecture
More informationECE G: Special Topics in Signal Processing: Sparsity, Structure, and Inference
ECE 18-898G: Special Topics in Signal Processing: Sparsity, Structure, and Inference Neural Networks: A brief touch Yuejie Chi Department of Electrical and Computer Engineering Spring 2018 1/41 Outline
More informationMachine Learning Basics III
Machine Learning Basics III Benjamin Roth CIS LMU München Benjamin Roth (CIS LMU München) Machine Learning Basics III 1 / 62 Outline 1 Classification Logistic Regression 2 Gradient Based Optimization Gradient
More informationNeural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /9/17
3/9/7 Neural Networks Emily Fox University of Washington March 0, 207 Slides adapted from Ali Farhadi (via Carlos Guestrin and Luke Zettlemoyer) Single-layer neural network 3/9/7 Perceptron as a neural
More informationNeural Networks: Backpropagation
Neural Networks: Backpropagation Machine Learning Fall 2017 Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others
More informationNeural Networks: Backpropagation
Neural Networks: Backpropagation Seung-Hoon Na 1 1 Department of Computer Science Chonbuk National University 2018.10.25 eung-hoon Na (Chonbuk National University) Neural Networks: Backpropagation 2018.10.25
More information<Special Topics in VLSI> Learning for Deep Neural Networks (Back-propagation)
Learning for Deep Neural Networks (Back-propagation) Outline Summary of Previous Standford Lecture Universal Approximation Theorem Inference vs Training Gradient Descent Back-Propagation
More informationCSC321 Lecture 9: Generalization
CSC321 Lecture 9: Generalization Roger Grosse Roger Grosse CSC321 Lecture 9: Generalization 1 / 27 Overview We ve focused so far on how to optimize neural nets how to get them to make good predictions
More informationStochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization
Stochastic Dual Coordinate Ascent Methods for Regularized Loss Minimization Shai Shalev-Shwartz and Tong Zhang School of CS and Engineering, The Hebrew University of Jerusalem Optimization for Machine
More informationA summary of Deep Learning without Poor Local Minima
A summary of Deep Learning without Poor Local Minima by Kenji Kawaguchi MIT oral presentation at NIPS 2016 Learning Supervised (or Predictive) learning Learn a mapping from inputs x to outputs y, given
More information(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationTraining Neural Networks Practical Issues
Training Neural Networks Practical Issues M. Soleymani Sharif University of Technology Fall 2017 Most slides have been adapted from Fei Fei Li and colleagues lectures, cs231n, Stanford 2017, and some from
More informationDeep Learning & Artificial Intelligence WS 2018/2019
Deep Learning & Artificial Intelligence WS 2018/2019 Linear Regression Model Model Error Function: Squared Error Has no special meaning except it makes gradients look nicer Prediction Ground truth / target
More informationAutomatic Differentiation and Neural Networks
Statistical Machine Learning Notes 7 Automatic Differentiation and Neural Networks Instructor: Justin Domke 1 Introduction The name neural network is sometimes used to refer to many things (e.g. Hopfield
More informationGrundlagen der Künstlichen Intelligenz
Grundlagen der Künstlichen Intelligenz Neural networks Daniel Hennes 21.01.2018 (WS 2017/18) University Stuttgart - IPVS - Machine Learning & Robotics 1 Today Logistic regression Neural networks Perceptron
More informationBased on the original slides of Hung-yi Lee
Based on the original slides of Hung-yi Lee Google Trends Deep learning obtains many exciting results. Can contribute to new Smart Services in the Context of the Internet of Things (IoT). IoT Services
More informationLearning Deep Architectures for AI. Part II - Vijay Chakilam
Learning Deep Architectures for AI - Yoshua Bengio Part II - Vijay Chakilam Limitations of Perceptron x1 W, b 0,1 1,1 y x2 weight plane output =1 output =0 There is no value for W and b such that the model
More informationNeural Networks. Learning and Computer Vision Prof. Olga Veksler CS9840. Lecture 10
CS9840 Learning and Computer Vision Prof. Olga Veksler Lecture 0 Neural Networks Many slides are from Andrew NG, Yann LeCun, Geoffry Hinton, Abin - Roozgard Outline Short Intro Perceptron ( layer NN) Multilayer
More informationMachine Learning. Linear Models. Fabio Vandin October 10, 2017
Machine Learning Linear Models Fabio Vandin October 10, 2017 1 Linear Predictors and Affine Functions Consider X = R d Affine functions: L d = {h w,b : w R d, b R} where ( d ) h w,b (x) = w, x + b = w
More informationCS 1674: Intro to Computer Vision. Final Review. Prof. Adriana Kovashka University of Pittsburgh December 7, 2016
CS 1674: Intro to Computer Vision Final Review Prof. Adriana Kovashka University of Pittsburgh December 7, 2016 Final info Format: multiple-choice, true/false, fill in the blank, short answers, apply an
More informationApprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning
Apprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning Nicolas Thome Prenom.Nom@cnam.fr http://cedric.cnam.fr/vertigo/cours/ml2/ Département Informatique Conservatoire
More informationCS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning
CS 229 Project Final Report: Reinforcement Learning for Neural Network Architecture Category : Theory & Reinforcement Learning Lei Lei Ruoxuan Xiong December 16, 2017 1 Introduction Deep Neural Network
More informationDeep Learning. Convolutional Neural Network (CNNs) Ali Ghodsi. October 30, Slides are partially based on Book in preparation, Deep Learning
Convolutional Neural Network (CNNs) University of Waterloo October 30, 2015 Slides are partially based on Book in preparation, by Bengio, Goodfellow, and Aaron Courville, 2015 Convolutional Networks Convolutional
More informationNeural networks. Chapter 20. Chapter 20 1
Neural networks Chapter 20 Chapter 20 1 Outline Brains Neural networks Perceptrons Multilayer networks Applications of neural networks Chapter 20 2 Brains 10 11 neurons of > 20 types, 10 14 synapses, 1ms
More informationMachine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function.
Bayesian learning: Machine learning comes from Bayesian decision theory in statistics. There we want to minimize the expected value of the loss function. Let y be the true label and y be the predicted
More informationRegML 2018 Class 8 Deep learning
RegML 2018 Class 8 Deep learning Lorenzo Rosasco UNIGE-MIT-IIT June 18, 2018 Supervised vs unsupervised learning? So far we have been thinking of learning schemes made in two steps f(x) = w, Φ(x) F, x
More information1 What a Neural Network Computes
Neural Networks 1 What a Neural Network Computes To begin with, we will discuss fully connected feed-forward neural networks, also known as multilayer perceptrons. A feedforward neural network consists
More informationLecture 2: Modular Learning
Lecture 2: Modular Learning Deep Learning @ UvA MODULAR LEARNING - PAGE 1 Announcement o C. Maddison is coming to the Deep Vision Seminars on the 21 st of Sep Author of concrete distribution, co-author
More informationDeep Feedforward Networks. Sargur N. Srihari
Deep Feedforward Networks Sargur N. srihari@cedar.buffalo.edu 1 Topics Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation and Other Differentiation
More informationIntro to Neural Networks and Deep Learning
Intro to Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi UVA CS 6316 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions Backpropagation Nonlinearity Functions NNs
More informationAdvanced Machine Learning
Advanced Machine Learning Lecture 4: Deep Learning Essentials Pierre Geurts, Gilles Louppe, Louis Wehenkel 1 / 52 Outline Goal: explain and motivate the basic constructs of neural networks. From linear
More informationNeural Networks, Computation Graphs. CMSC 470 Marine Carpuat
Neural Networks, Computation Graphs CMSC 470 Marine Carpuat Binary Classification with a Multi-layer Perceptron φ A = 1 φ site = 1 φ located = 1 φ Maizuru = 1 φ, = 2 φ in = 1 φ Kyoto = 1 φ priest = 0 φ
More informationECE521 Lectures 9 Fully Connected Neural Networks
ECE521 Lectures 9 Fully Connected Neural Networks Outline Multi-class classification Learning multi-layer neural networks 2 Measuring distance in probability space We learnt that the squared L2 distance
More informationMachine Learning Lecture 10
Machine Learning Lecture 10 Neural Networks 26.11.2018 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Today s Topic Deep Learning 2 Course Outline Fundamentals Bayes
More informationDEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY
DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY 1 On-line Resources http://neuralnetworksanddeeplearning.com/index.html Online book by Michael Nielsen http://matlabtricks.com/post-5/3x3-convolution-kernelswith-online-demo
More informationMore on Neural Networks
More on Neural Networks Yujia Yan Fall 2018 Outline Linear Regression y = Wx + b (1) Linear Regression y = Wx + b (1) Polynomial Regression y = Wφ(x) + b (2) where φ(x) gives the polynomial basis, e.g.,
More information9 Classification. 9.1 Linear Classifiers
9 Classification This topic returns to prediction. Unlike linear regression where we were predicting a numeric value, in this case we are predicting a class: winner or loser, yes or no, rich or poor, positive
More informationEVERYTHING YOU NEED TO KNOW TO BUILD YOUR FIRST CONVOLUTIONAL NEURAL NETWORK (CNN)
EVERYTHING YOU NEED TO KNOW TO BUILD YOUR FIRST CONVOLUTIONAL NEURAL NETWORK (CNN) TARGETED PIECES OF KNOWLEDGE Linear regression Activation function Multi-Layers Perceptron (MLP) Stochastic Gradient Descent
More informationTopic 3: Neural Networks
CS 4850/6850: Introduction to Machine Learning Fall 2018 Topic 3: Neural Networks Instructor: Daniel L. Pimentel-Alarcón c Copyright 2018 3.1 Introduction Neural networks are arguably the main reason why
More informationCOMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: Ryan Lowe (ryan.lowe@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted,
More information