Neural networks COMS 4771
|
|
- Timothy Samson McKinney
- 5 years ago
- Views:
Transcription
1 Neural networks COMS 4771
2 1. Logistic regression
3 Logistic regression Suppose X = R d and Y = {0, 1}. A logistic regression model is a statistical model where the conditional probability function has a particular form: Y X = x Bern(logistic(x T w)), x R d, with logistic(z) := 1 ez = 1 + e z 1 + e, z R. z Parameters: w = (w 1,..., w d ) R d. Conditional probability function: η w(x) = logistic(x T w). 1 / 22
4 Logistic regression Network diagram for η w: x 1 x 2 w 1 w 2 v. w d x d d v := g(z), z := w jx j, j=1 (g = logistic). Here, g is called the link function. 2 / 22
5 Learning w from data Training data ((x i, y i)) n i=1 from R d {0, 1}. Could use MLE to learn w from data. 3 / 22
6 Learning w from data Training data ((x i, y i)) n i=1 from R d {0, 1}. Could use MLE to learn w from data. Another option: Squared loss ERM (with link function g) 1 min w R d n n (g(x T i w) y i) 2. i=1 3 / 22
7 Learning w from data Training data ((x i, y i)) n i=1 from R d {0, 1}. Could use MLE to learn w from data. Another option: Squared loss ERM (with link function g) 1 min w R d n n (g(x T i w) y i) 2. i=1 Observe that for any (X, Y ) P (not necessarily logistic regression), [ ( ) ] 2 ( 2 E g(x T w) Y X = x = g(x T w) η(x)) + var(y X = x) where η(x) = P(Y = 1 X = x). 3 / 22
8 Learning w from data Training data ((x i, y i)) n i=1 from R d {0, 1}. Could use MLE to learn w from data. Another option: Squared loss ERM (with link function g) 1 min w R d n n (g(x T i w) y i) 2. i=1 Observe that for any (X, Y ) P (not necessarily logistic regression), [ ( ) ] 2 ( 2 E g(x T w) Y X = x = g(x T w) η(x)) + var(y X = x) where η(x) = P(Y = 1 X = x). Algorithm for Squared loss ERM with link function g? 3 / 22
9 Stochastic gradient method w {(g(x T w) y) 2} = 2(g(x T w) y) g (x T w) x. 4 / 22
10 Stochastic gradient method w {(g(x T w) y) 2} = 2(g(x T w) y) g (x T w) x. Stochastic gradient method for squared loss ERM with link function g 1: Start with some initial w (1) R d. 2: for t = 1, 2,... until some stopping condition is satisfied do 3: Pick (X (t), Y (t) ) uniformly at random from (x 1, y 1 ),..., (x n, y n ). 4: Update: w (t+1) := w (t) 2η t (g( X (t), w (t) ) Y (t) ) g ( X (t), w (t) ) X (t). 5: end for 4 / 22
11 Extensions Other loss functions (e.g., y ln (1 y) ln ). p 1 p Other link functions (e.g., g(z) = some polynomial in z). 5 / 22
12 Extensions Other loss functions (e.g., y ln (1 y) ln ). p 1 p Other link functions (e.g., g(z) = some polynomial in z). Is the overall objective function convex? 5 / 22
13 Extensions Other loss functions (e.g., y ln (1 y) ln ). p 1 p Other link functions (e.g., g(z) = some polynomial in z). Is the overall objective function convex? Somtimes, but not always. 5 / 22
14 Extensions Other loss functions (e.g., y ln (1 y) ln ). p 1 p Other link functions (e.g., g(z) = some polynomial in z). Is the overall objective function convex? Somtimes, but not always. Nevertheless, stochastic gradient method is still often effective at finding approximate local minima. 5 / 22
15 2. Multilayer neural networks
16 Two-output network x 1 v 1 x 2. v 2 x d d v j := g(z j), z j := W i,jx i, j {1, 2}. i=1 6 / 22
17 k-output network x 1 v 1 x 2 v 2. x d v k d v j := g(z j), z j := W i,jx i, i=1 j {1,..., k}. 7 / 22
18 A motivating example: multitask learning k binary prediction tasks with a single feature vector (e.g., predicting tags for images). Labeled examples are of the form (x i, (y i,1,..., y i,k )) R d {0, 1} k. 8 / 22
19 A motivating example: multitask learning k binary prediction tasks with a single feature vector (e.g., predicting tags for images). Labeled examples are of the form (x i, (y i,1,..., y i,k )) R d {0, 1} k. Option 1: k independent logistic regression models; learn w 1,..., w k by minimizing (e.g.) 1 n k (g(x T i w j) y i,j) 2. n i=1 j=1 8 / 22
20 A motivating example: multitask learning k binary prediction tasks with a single feature vector (e.g., predicting tags for images). Labeled examples are of the form (x i, (y i,1,..., y i,k )) R d {0, 1} k. Option 1: k independent logistic regression models; learn w 1,..., w k by minimizing (e.g.) 1 n k (g(x T i w j) y i,j) 2. n i=1 j=1 Suppose labels y i,1,..., y i,k are not independent. E.g., if y i,1 = 1, then also more likely that y i,2 = 1. 8 / 22
21 A motivating example: multitask learning k binary prediction tasks with a single feature vector (e.g., predicting tags for images). Labeled examples are of the form (x i, (y i,1,..., y i,k )) R d {0, 1} k. Option 1: k independent logistic regression models; learn w 1,..., w k by minimizing (e.g.) 1 n k (g(x T i w j) y i,j) 2. n i=1 j=1 Suppose labels y i,1,..., y i,k are not independent. E.g., if y i,1 = 1, then also more likely that y i,2 = 1. Option 2: Do Option 1, but also learn to combine predictions of y i,1,..., y i,k to get better predictions for each y i,j. 8 / 22
22 Multilayer neural network x 1 W (1) W (2) v (1) 1 v (2) 1 x 2 v (1) 2 v (2) 2. x d v (1) k v (2) k Columns of W 1 R d k : params. of original logistic regression models. Columns of W 2 R k k : params. of new logistic regression models to combine predictions of original models. 9 / 22
23 Multilayer neural network x 1 W (1) W (2) v (1) 1 v (2) 1 x 2 v (1) 2 v (2) 2. x d v (1) k v (2) k Columns of W 1 R d k : params. of original logistic regression models. Columns of W 2 R k k : params. of new logistic regression models to combine predictions of original models. Each node is called a unit. 9 / 22
24 Multilayer neural network x 1 W (1) W (2) v (1) 1 v (2) 1 x 2 v (1) 2 v (2) 2. x d v (1) k v (2) k Columns of W 1 R d k : params. of original logistic regression models. Columns of W 2 R k k : params. of new logistic regression models to combine predictions of original models. Each node is called a unit. Non-input and non-output units are called hidden. 9 / 22
25 Compositional structure Suppose we have two functions f W 1 : R d R k, (W 2 R d k ), f W 2 : R k R l, (W 2 R k l ), where f W (x) := g(w T x), and g applies the link function g coordinate-wise to a vector. 10 / 22
26 Compositional structure Suppose we have two functions f W 1 : R d R k, (W 2 R d k ), f W 2 : R k R l, (W 2 R k l ), where f W (x) := g(w T x), and g applies the link function g coordinate-wise to a vector. Composition: f W 1,W 2 := f W 2 f W 1 is defined by f W 1,W 2 (x) := f W 2 ( fw 1 (x) ). 10 / 22
27 Compositional structure Suppose we have two functions f W 1 : R d R k, (W 2 R d k ), f W 2 : R k R l, (W 2 R k l ), where f W (x) := g(w T x), and g applies the link function g coordinate-wise to a vector. Composition: f W 1,W 2 := f W 2 f W 1 is defined by This is a two-layer neural network. f W 1,W 2 (x) := f W 2 ( fw 1 (x) ). x 1 W (1) W (2) v (1) 1 v (2) 1 x 2 v (1) 2 v (2) 2. x d v (1) k v (2) l 10 / 22
28 Necessity of multiple layers One-layer neural network with a monotonic link function is a linear (or affine) classifier. Cannot represent XOR function (Minsky and Papert, 1969). x 1 1 x 1 x 1 1 1? x x x 2 (a) x 1 and x 2 (b) x1 or x2 (c) x1 xor x2 (Figure from Stuart Russell.) 11 / 22
29 Approximation power of multilayer neural networks Theorem (Cybenko, 1989; Hornik, 1991; Barron, 1993). Any continuous function f can be approximated arbitrarily well by a two-layer neural network f f W 2 }{{} R k R f W }{{} 1. R d R k However: may need a very large number of hidden units. 12 / 22
30 Approximation power of multilayer neural networks Theorem (Cybenko, 1989; Hornik, 1991; Barron, 1993). Any continuous function f can be approximated arbitrarily well by a two-layer neural network f f W 2 }{{} R k R f W }{{} 1. R d R k However: may need a very large number of hidden units. Theorem (Telgarsky, 2015; Eldan and Shamir, 2015). Some functions can be approximated with exponentially fewer hidden units by using more than two layers. 12 / 22
31 Approximation power of multilayer neural networks Theorem (Cybenko, 1989; Hornik, 1991; Barron, 1993). Any continuous function f can be approximated arbitrarily well by a two-layer neural network f f W 2 }{{} R k R f W }{{} 1. R d R k However: may need a very large number of hidden units. Theorem (Telgarsky, 2015; Eldan and Shamir, 2015). Some functions can be approximated with exponentially fewer hidden units by using more than two layers. Note: none of this speaks directly to learning neural networks from data. 12 / 22
32 3. Computation and learning with neural networks
33 General structure of neural network Neural network for f : R d R. (Easy to generalize to f : R d R k.) 13 / 22
34 General structure of neural network Neural network for f : R d R. (Easy to generalize to f : R d R k.) Directed acyclic graph G = (V, E); vertices regarded as formal variables. ŷ w u,v u v x 1 x 2 x d 13 / 22
35 General structure of neural network Neural network for f : R d R. (Easy to generalize to f : R d R k.) Directed acyclic graph G = (V, E); vertices regarded as formal variables. d source vertices, one per input variable, called x 1,..., x d. w u,v u ŷ v x 1 x 2 x d 13 / 22
36 General structure of neural network Neural network for f : R d R. (Easy to generalize to f : R d R k.) Directed acyclic graph G = (V, E); vertices regarded as formal variables. d source vertices, one per input variable, called x 1,..., x d. Single sink vertex, called ŷ. w u,v u ŷ v x 1 x 2 x d 13 / 22
37 General structure of neural network Neural network for f : R d R. (Easy to generalize to f : R d R k.) Directed acyclic graph G = (V, E); vertices regarded as formal variables. d source vertices, one per input variable, called x 1,..., x d. Single sink vertex, called ŷ. Each edge (u, v) E has a weight w u,v R. x 1 w u,v u x 2 ŷ v x d 13 / 22
38 General structure of neural network Neural network for f : R d R. (Easy to generalize to f : R d R k.) Directed acyclic graph G = (V, E); vertices regarded as formal variables. d source vertices, one per input variable, called x 1,..., x d. Single sink vertex, called ŷ. w u,v u ŷ v Each edge (u, v) E has a weight w u,v R. x 1 x 2 x d Value of vertex v given values of parents π G(v) := {u V : (u, v) E} is v := g(z v), z v := w u,v u. (g is link function, e.g., logistic function.) u π G (v) 13 / 22
39 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1, / 22
40 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. 14 / 22
41 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. 14 / 22
42 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. Final layer only consists of sink vertex, ŷ. 14 / 22
43 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. Final layer only consists of sink vertex, ŷ. Given input values x = (x 1,..., x d ) R d, how to compute f(x)? 14 / 22
44 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. Final layer only consists of sink vertex, ŷ. Given input values x = (x 1,..., x d ) R d, how to compute f(x)? 1. Compute values of all vertices in V 1, given values of vertices in V 0 (i.e., input variables). v := g(z v), z v := w u,v u. (All parents of v V 1 are in V 0.) u π G (v) 14 / 22
45 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. Final layer only consists of sink vertex, ŷ. Given input values x = (x 1,..., x d ) R d, how to compute f(x)? 1. Compute values of all vertices in V 1, given values of vertices in V 0 (i.e., input variables). v := g(z v), z v := w u,v u. (All parents of v V 1 are in V 0.) u π G (v) 2. Compute values of all vertices in V 2, given values of vertices in V 0 V 1. (All parents of v V 2 are in V 0 V 1.) 14 / 22
46 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. Final layer only consists of sink vertex, ŷ. Given input values x = (x 1,..., x d ) R d, how to compute f(x)? 1. Compute values of all vertices in V 1, given values of vertices in V 0 (i.e., input variables). v := g(z v), z v := w u,v u. (All parents of v V 1 are in V 0.) u π G (v) 2. Compute values of all vertices in V 2, given values of vertices in V 0 V 1. (All parents of v V 2 are in V 0 V 1.) 3. Etc., until ŷ = f(x) is computed. 14 / 22
47 Organizing and evaluating a neural network Vertices in V partitioned into layers V 0, V 1,.... V 0 := {x 1,..., x d }, just the input variables. Put v in V l if longest path in G from some x i to v has l edges. Final layer only consists of sink vertex, ŷ. Given input values x = (x 1,..., x d ) R d, how to compute f(x)? 1. Compute values of all vertices in V 1, given values of vertices in V 0 (i.e., input variables). v := g(z v), z v := w u,v u. (All parents of v V 1 are in V 0.) u π G (v) 2. Compute values of all vertices in V 2, given values of vertices in V 0 V 1. (All parents of v V 2 are in V 0 V 1.) 3. Etc., until ŷ = f(x) is computed. This is called forward propagataion. 14 / 22
48 Training a neural network How to fit a neural network to data? 15 / 22
49 Training a neural network How to fit a neural network to data? Use stochastic gradient method! 15 / 22
50 Training a neural network How to fit a neural network to data? Use stochastic gradient method! Basic computational problem: compute partial derivative of loss on a labeled example with respect to a parameter. 15 / 22
51 Training a neural network How to fit a neural network to data? Use stochastic gradient method! Basic computational problem: compute partial derivative of loss on a labeled example with respect to a parameter. Mid-to-late 20th century researchers discovered how to use chain rule to organize gradient computation: backpropagation algorithm. 15 / 22
52 Training a neural network How to fit a neural network to data? Use stochastic gradient method! Basic computational problem: compute partial derivative of loss on a labeled example with respect to a parameter. Mid-to-late 20th century researchers discovered how to use chain rule to organize gradient computation: backpropagation algorithm. Given: labeled example (x, y) R d R; current weights w u,v R for all (u, v) E; 15 / 22
53 Training a neural network How to fit a neural network to data? Use stochastic gradient method! Basic computational problem: compute partial derivative of loss on a labeled example with respect to a parameter. Mid-to-late 20th century researchers discovered how to use chain rule to organize gradient computation: backpropagation algorithm. Given: labeled example (x, y) R d R; current weights w u,v R for all (u, v) E; values v and z v for all non-source v V on input x. (Can first run forward propagation to get v s and z v s.) 15 / 22
54 Training a neural network How to fit a neural network to data? Use stochastic gradient method! Basic computational problem: compute partial derivative of loss on a labeled example with respect to a parameter. Mid-to-late 20th century researchers discovered how to use chain rule to organize gradient computation: backpropagation algorithm. Given: labeled example (x, y) R d R; current weights w u,v R for all (u, v) E; values v and z v for all non-source v V on input x. (Can first run forward propagation to get v s and z v s.) Let l denote loss of prediction ŷ = f(x) (e.g., l := (ŷ y) 2 ). 15 / 22
55 Training a neural network How to fit a neural network to data? Use stochastic gradient method! Basic computational problem: compute partial derivative of loss on a labeled example with respect to a parameter. Mid-to-late 20th century researchers discovered how to use chain rule to organize gradient computation: backpropagation algorithm. Given: labeled example (x, y) R d R; current weights w u,v R for all (u, v) E; values v and z v for all non-source v V on input x. (Can first run forward propagation to get v s and z v s.) Let l denote loss of prediction ŷ = f(x) (e.g., l := (ŷ y) 2 ). Goal: Compute l w u,v, (u, v) E. 15 / 22
56 Backpropagation: exploiting the chain rule Strategy: use chain rule. l = l w u,v ŷ ŷ v v. w u,v 16 / 22
57 Backpropagation: exploiting the chain rule Strategy: use chain rule. l = l w u,v ŷ ŷ v v. w u,v For squared loss l = (ŷ y) 2, l = 2(ŷ y). ŷ Easy to compute with other losses as well. (ŷ is computed in forward propagation.) 16 / 22
58 Backpropagation: exploiting the chain rule Strategy: use chain rule. l = l w u,v ŷ ŷ v v. w u,v For squared loss l = (ŷ y) 2, l = 2(ŷ y). ŷ Easy to compute with other losses as well. (ŷ is computed in forward propagation.) Since v = g(z v) where z v = w u,v u + (terms not involving w u,v), v = v z v w u,v z v w u,v = g (z v) u. w u,v (z v and u are computed in forward propagation.) v u 16 / 22
59 Backpropagation: the recursive part Key trick: compute ŷ v for all v V l, in decreasing order of layer l. 17 / 22
60 Backpropagation: the recursive part Key trick: compute ŷ v for all v V l, in decreasing order of layer l. Strategy: for v ŷ, use multivariate chain rule. Let k = out-deg(v), (v, v 1),..., (v, v k ) E: ŷ. ŷ v = k i=1 ŷ v i vi v. v 1 v k w v,v1 w v,vk v 17 / 22
61 Backpropagation: the recursive part Key trick: compute ŷ v for all v V l, in decreasing order of layer l. Strategy: for v ŷ, use multivariate chain rule. Let k = out-deg(v), (v, v 1),..., (v, v k ) E: ŷ. ŷ v = k i=1 ŷ v i vi v. v 1 v k w v,v1 w v,vk v Since v i = g(z vi ) where z vi = w v,vi v + (terms not involving v), v i v = g (z vi ) w v,vi. (The z vi s are computed in forward propagation.) 17 / 22
62 Backpropagation: the recursive part Key trick: compute ŷ v for all v V l, in decreasing order of layer l. Strategy: for v ŷ, use multivariate chain rule. Let k = out-deg(v), (v, v 1),..., (v, v k ) E: ŷ. ŷ v = k i=1 ŷ v i vi v. v 1 v k w v,v1 w v,vk v Since v i = g(z vi ) where z vi = w v,vi v + (terms not involving v), v i v = g (z vi ) w v,vi. (The z vi s are computed in forward propagation.) Since v i are in a higher layer than v, ŷ v i has already been computed! 17 / 22
63 Matrix view of forward/backward propagation Recall general neural network function f : R d R can be written as f(x) = g L (W L g 2 (W 2g 1 (W 1x)) ), where W i R d i d i 1 is weight matrix for layer i, and g i : R d i R d i collects the non-linear link functions for layer i. 18 / 22
64 Matrix view of forward/backward propagation Recall general neural network function f : R d R can be written as f(x) = g L (W L g 2 (W 2g 1 (W 1x)) ), where W i R d i d i 1 is weight matrix for layer i, and g i : R d i R d i collects the non-linear link functions for layer i. The first layer thus computes and i-th layer computes (Convention: f 0 (x) := x.) f 1 (x) := g 1 (z 1) where z 1 := W 1x, f i (x) := g i (z i) where z i := W if i 1 (x). 18 / 22
65 Matrix view of forward/backward propagation Recall general neural network function f : R d R can be written as f(x) = g L (W L g 2 (W 2g 1 (W 1x)) ), where W i R d i d i 1 is weight matrix for layer i, and g i : R d i R d i collects the non-linear link functions for layer i. The first layer thus computes and i-th layer computes (Convention: f 0 (x) := x.) f 1 (x) := g 1 (z 1) where z 1 := W 1x, f i (x) := g i (z i) where z i := W if i 1 (x). Previous backprop equations prove correctness of the following matrix derivative formula: [ ( l ( ) )] = g W i(z i) W T i+1 g L 1(z L 1) W T L g L(z L)l (f L(x)) f i 1(x) T i where denotes element-wise product. 18 / 22
66 Practical tips Apply stochastic gradient method to examples in random order. Can use several examples to form gradient estimate: mini-batch. λ (t) := 1 b tb i=(t 1)b+1 where l i is loss on i-th training example. l i W W =W (t) 19 / 22
67 Practical tips Apply stochastic gradient method to examples in random order. Can use several examples to form gradient estimate: mini-batch. λ (t) := 1 b tb i=(t 1)b+1 where l i is loss on i-th training example. l i W W =W (t) Standardize inputs (i.e., center and divide by standard deviation). Doing this at every layer during training: batch normalization. (Must also apply same/similar normalization at test time.) 19 / 22
68 Practical tips Apply stochastic gradient method to examples in random order. Can use several examples to form gradient estimate: mini-batch. λ (t) := 1 b tb i=(t 1)b+1 where l i is loss on i-th training example. l i W W =W (t) Standardize inputs (i.e., center and divide by standard deviation). Doing this at every layer during training: batch normalization. (Must also apply same/similar normalization at test time.) Initialization: Take care so initial weights not too large or small. E.g., for node with d inputs, draw weights iid from N(0, 1/d). 19 / 22
69 Modern networks Two kinds of linear layers: dense / fully-connected layers W if i 1 (x) as before, convolutional layers, which have a special sparse representation. 20 / 22
70 Modern networks Two kinds of linear layers: dense / fully-connected layers W if i 1 (x) as before, convolutional layers, which have a special sparse representation. Common link functions: ReLU z i max{0, z i} (applied element-wise) SoftMax z i ez i j ez j (applied element-wise) max-pooling z max i z i (after convolution layers; R d R), 20 / 22
71 Modern networks Two kinds of linear layers: dense / fully-connected layers W if i 1 (x) as before, convolutional layers, which have a special sparse representation. Common link functions: ReLU z i max{0, z i} (applied element-wise) SoftMax z i ez i j ez j (applied element-wise) max-pooling z max i z i (after convolution layers; R d R), Trend in architectures (in some domains, like vision and speech): Few convolutional layers then many dense layers (AlexNet) More convolutional layers (VGGNet) Nearly purely convolutional layers in many (100+) layers with varienty of identity connections throughout (ResNet, DenseNet). 20 / 22
72 Modern networks Two kinds of linear layers: dense / fully-connected layers W if i 1 (x) as before, convolutional layers, which have a special sparse representation. Common link functions: ReLU z i max{0, z i} (applied element-wise) SoftMax z i ez i j ez j (applied element-wise) max-pooling z max i z i (after convolution layers; R d R), Trend in architectures (in some domains, like vision and speech): Few convolutional layers then many dense layers (AlexNet) More convolutional layers (VGGNet) Nearly purely convolutional layers in many (100+) layers with varienty of identity connections throughout (ResNet, DenseNet). Can use intermediate computation (e.g., f i (x)) as feature expansion! Indespensible in visual and audio tasks; application attempts are constant in all other disciplines. 20 / 22
73 Wrap-up Frontier of experimental machine learning research. 21 / 22
74 Wrap-up Frontier of experimental machine learning research. Basic ideas same as from the 1980s. 21 / 22
75 Wrap-up Frontier of experimental machine learning research. Basic ideas same as from the 1980s. What s new? Fast hardware. 21 / 22
76 Wrap-up Frontier of experimental machine learning research. Basic ideas same as from the 1980s. What s new? Fast hardware. Tons of data. 21 / 22
77 Wrap-up Frontier of experimental machine learning research. Basic ideas same as from the 1980s. What s new? Fast hardware. Tons of data. Many-layered networks (made possible through many adjustments). 21 / 22
78 Wrap-up Frontier of experimental machine learning research. Basic ideas same as from the 1980s. What s new? Fast hardware. Tons of data. Many-layered networks (made possible through many adjustments). Applications: visual detection and recognition, speech recognition, general function fitting (e.g., learning reward functions of different actions of video games), etc. 21 / 22
79 Wrap-up Frontier of experimental machine learning research. Basic ideas same as from the 1980s. What s new? Fast hardware. Tons of data. Many-layered networks (made possible through many adjustments). Applications: visual detection and recognition, speech recognition, general function fitting (e.g., learning reward functions of different actions of video games), etc. In cutting-edge applications, training is still delicate. 21 / 22
80 Key takeaways 1. Structure of neural networks; concept of link functions. 2. Motivation for multilayer neural networks and arguments for depth. 3. Forward and backward propagation algorithms. 22 / 22
<Special Topics in VLSI> Learning for Deep Neural Networks (Back-propagation)
Learning for Deep Neural Networks (Back-propagation) Outline Summary of Previous Standford Lecture Universal Approximation Theorem Inference vs Training Gradient Descent Back-Propagation
More informationRapid Introduction to Machine Learning/ Deep Learning
Rapid Introduction to Machine Learning/ Deep Learning Hyeong In Choi Seoul National University 1/62 Lecture 1b Logistic regression & neural network October 2, 2015 2/62 Table of contents 1 1 Bird s-eye
More informationCSE446: Neural Networks Spring Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer
CSE446: Neural Networks Spring 2017 Many slides are adapted from Carlos Guestrin and Luke Zettlemoyer Human Neurons Switching time ~ 0.001 second Number of neurons 10 10 Connections per neuron 10 4-5 Scene
More informationLecture 5: Logistic Regression. Neural Networks
Lecture 5: Logistic Regression. Neural Networks Logistic regression Comparison with generative models Feed-forward neural networks Backpropagation Tricks for training neural networks COMP-652, Lecture
More informationCSCI567 Machine Learning (Fall 2018)
CSCI567 Machine Learning (Fall 2018) Prof. Haipeng Luo U of Southern California Sep 12, 2018 September 12, 2018 1 / 49 Administration GitHub repos are setup (ask TA Chi Zhang for any issues) HW 1 is due
More informationCh.6 Deep Feedforward Networks (2/3)
Ch.6 Deep Feedforward Networks (2/3) 16. 10. 17. (Mon.) System Software Lab., Dept. of Mechanical & Information Eng. Woonggy Kim 1 Contents 6.3. Hidden Units 6.3.1. Rectified Linear Units and Their Generalizations
More informationMachine Learning for Large-Scale Data Analysis and Decision Making A. Neural Networks Week #6
Machine Learning for Large-Scale Data Analysis and Decision Making 80-629-17A Neural Networks Week #6 Today Neural Networks A. Modeling B. Fitting C. Deep neural networks Today s material is (adapted)
More informationFeedforward Neural Nets and Backpropagation
Feedforward Neural Nets and Backpropagation Julie Nutini University of British Columbia MLRG September 28 th, 2016 1 / 23 Supervised Learning Roadmap Supervised Learning: Assume that we are given the features
More informationDeep Feedforward Networks
Deep Feedforward Networks Yongjin Park 1 Goal of Feedforward Networks Deep Feedforward Networks are also called as Feedforward neural networks or Multilayer Perceptrons Their Goal: approximate some function
More informationNeural Networks Learning the network: Backprop , Fall 2018 Lecture 4
Neural Networks Learning the network: Backprop 11-785, Fall 2018 Lecture 4 1 Recap: The MLP can represent any function The MLP can be constructed to represent anything But how do we construct it? 2 Recap:
More informationDeep Feedforward Networks. Seung-Hoon Na Chonbuk National University
Deep Feedforward Networks Seung-Hoon Na Chonbuk National University Neural Network: Types Feedforward neural networks (FNN) = Deep feedforward networks = multilayer perceptrons (MLP) No feedback connections
More informationIntroduction to Machine Learning (67577)
Introduction to Machine Learning (67577) Shai Shalev-Shwartz School of CS and Engineering, The Hebrew University of Jerusalem Deep Learning Shai Shalev-Shwartz (Hebrew U) IML Deep Learning Neural Networks
More informationNONLINEAR CLASSIFICATION AND REGRESSION. J. Elder CSE 4404/5327 Introduction to Machine Learning and Pattern Recognition
NONLINEAR CLASSIFICATION AND REGRESSION Nonlinear Classification and Regression: Outline 2 Multi-Layer Perceptrons The Back-Propagation Learning Algorithm Generalized Linear Models Radial Basis Function
More informationA summary of Deep Learning without Poor Local Minima
A summary of Deep Learning without Poor Local Minima by Kenji Kawaguchi MIT oral presentation at NIPS 2016 Learning Supervised (or Predictive) learning Learn a mapping from inputs x to outputs y, given
More informationDEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY
DEEP LEARNING AND NEURAL NETWORKS: BACKGROUND AND HISTORY 1 On-line Resources http://neuralnetworksanddeeplearning.com/index.html Online book by Michael Nielsen http://matlabtricks.com/post-5/3x3-convolution-kernelswith-online-demo
More informationMachine Learning Basics III
Machine Learning Basics III Benjamin Roth CIS LMU München Benjamin Roth (CIS LMU München) Machine Learning Basics III 1 / 62 Outline 1 Classification Logistic Regression 2 Gradient Based Optimization Gradient
More informationArtificial Neural Networks. MGS Lecture 2
Artificial Neural Networks MGS 2018 - Lecture 2 OVERVIEW Biological Neural Networks Cell Topology: Input, Output, and Hidden Layers Functional description Cost functions Training ANNs Back-Propagation
More informationLecture 3 Feedforward Networks and Backpropagation
Lecture 3 Feedforward Networks and Backpropagation CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 3, 2017 Things we will look at today Recap of Logistic Regression
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationNeural Networks with Applications to Vision and Language. Feedforward Networks. Marco Kuhlmann
Neural Networks with Applications to Vision and Language Feedforward Networks Marco Kuhlmann Feedforward networks Linear separability x 2 x 2 0 1 0 1 0 0 x 1 1 0 x 1 linearly separable not linearly separable
More informationBackpropagation Introduction to Machine Learning. Matt Gormley Lecture 12 Feb 23, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Backpropagation Matt Gormley Lecture 12 Feb 23, 2018 1 Neural Networks Outline
More informationLecture 3 Feedforward Networks and Backpropagation
Lecture 3 Feedforward Networks and Backpropagation CMSC 35246: Deep Learning Shubhendu Trivedi & Risi Kondor University of Chicago April 3, 2017 Things we will look at today Recap of Logistic Regression
More informationFeedforward Neural Networks
Feedforward Neural Networks Michael Collins 1 Introduction In the previous notes, we introduced an important class of models, log-linear models. In this note, we describe feedforward neural networks, which
More informationNeural Networks and Deep Learning
Neural Networks and Deep Learning Professor Ameet Talwalkar November 12, 2015 Professor Ameet Talwalkar Neural Networks and Deep Learning November 12, 2015 1 / 16 Outline 1 Review of last lecture AdaBoost
More informationNeural Networks. Yan Shao Department of Linguistics and Philology, Uppsala University 7 December 2016
Neural Networks Yan Shao Department of Linguistics and Philology, Uppsala University 7 December 2016 Outline Part 1 Introduction Feedforward Neural Networks Stochastic Gradient Descent Computational Graph
More informationNeural Networks. Nicholas Ruozzi University of Texas at Dallas
Neural Networks Nicholas Ruozzi University of Texas at Dallas Handwritten Digit Recognition Given a collection of handwritten digits and their corresponding labels, we d like to be able to correctly classify
More informationLecture 17: Neural Networks and Deep Learning
UVA CS 6316 / CS 4501-004 Machine Learning Fall 2016 Lecture 17: Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions
More informationCSC 411 Lecture 10: Neural Networks
CSC 411 Lecture 10: Neural Networks Roger Grosse, Amir-massoud Farahmand, and Juan Carrasquilla University of Toronto UofT CSC 411: 10-Neural Networks 1 / 35 Inspiration: The Brain Our brain has 10 11
More informationDeep Feedforward Networks
Deep Feedforward Networks Liu Yang March 30, 2017 Liu Yang Short title March 30, 2017 1 / 24 Overview 1 Background A general introduction Example 2 Gradient based learning Cost functions Output Units 3
More informationAdvanced Machine Learning
Advanced Machine Learning Lecture 4: Deep Learning Essentials Pierre Geurts, Gilles Louppe, Louis Wehenkel 1 / 52 Outline Goal: explain and motivate the basic constructs of neural networks. From linear
More informationNeural Networks. Single-layer neural network. CSE 446: Machine Learning Emily Fox University of Washington March 10, /9/17
3/9/7 Neural Networks Emily Fox University of Washington March 0, 207 Slides adapted from Ali Farhadi (via Carlos Guestrin and Luke Zettlemoyer) Single-layer neural network 3/9/7 Perceptron as a neural
More informationMachine Learning and Data Mining. Multi-layer Perceptrons & Neural Networks: Basics. Prof. Alexander Ihler
+ Machine Learning and Data Mining Multi-layer Perceptrons & Neural Networks: Basics Prof. Alexander Ihler Linear Classifiers (Perceptrons) Linear Classifiers a linear classifier is a mapping which partitions
More informationMachine Learning Lecture 10
Machine Learning Lecture 10 Neural Networks 26.11.2018 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Today s Topic Deep Learning 2 Course Outline Fundamentals Bayes
More informationDeep Learning (CNNs)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Deep Learning (CNNs) Deep Learning Readings: Murphy 28 Bishop - - HTF - - Mitchell
More informationConvolutional Neural Network Architecture
Convolutional Neural Network Architecture Zhisheng Zhong Feburary 2nd, 2018 Zhisheng Zhong Convolutional Neural Network Architecture Feburary 2nd, 2018 1 / 55 Outline 1 Introduction of Convolution Motivation
More informationDeep Feedforward Networks. Sargur N. Srihari
Deep Feedforward Networks Sargur N. srihari@cedar.buffalo.edu 1 Topics Overview 1. Example: Learning XOR 2. Gradient-Based Learning 3. Hidden Units 4. Architecture Design 5. Backpropagation and Other Differentiation
More informationMachine Learning (CSE 446): Neural Networks
Machine Learning (CSE 446): Neural Networks Noah Smith c 2017 University of Washington nasmith@cs.washington.edu November 6, 2017 1 / 22 Admin No Wednesday office hours for Noah; no lecture Friday. 2 /
More informationLogistic Regression & Neural Networks
Logistic Regression & Neural Networks CMSC 723 / LING 723 / INST 725 Marine Carpuat Slides credit: Graham Neubig, Jacob Eisenstein Logistic Regression Perceptron & Probabilities What if we want a probability
More informationMachine Learning (CSE 446): Backpropagation
Machine Learning (CSE 446): Backpropagation Noah Smith c 2017 University of Washington nasmith@cs.washington.edu November 8, 2017 1 / 32 Neuron-Inspired Classifiers correct output y n L n loss hidden units
More informationNeural Networks: Backpropagation
Neural Networks: Backpropagation Seung-Hoon Na 1 1 Department of Computer Science Chonbuk National University 2018.10.25 eung-hoon Na (Chonbuk National University) Neural Networks: Backpropagation 2018.10.25
More information(Feed-Forward) Neural Networks Dr. Hajira Jabeen, Prof. Jens Lehmann
(Feed-Forward) Neural Networks 2016-12-06 Dr. Hajira Jabeen, Prof. Jens Lehmann Outline In the previous lectures we have learned about tensors and factorization methods. RESCAL is a bilinear model for
More informationCS 6501: Deep Learning for Computer Graphics. Basics of Neural Networks. Connelly Barnes
CS 6501: Deep Learning for Computer Graphics Basics of Neural Networks Connelly Barnes Overview Simple neural networks Perceptron Feedforward neural networks Multilayer perceptron and properties Autoencoders
More informationComments. Assignment 3 code released. Thought questions 3 due this week. Mini-project: hopefully you have started. implement classification algorithms
Neural networks Comments Assignment 3 code released implement classification algorithms use kernels for census dataset Thought questions 3 due this week Mini-project: hopefully you have started 2 Example:
More informationReading Group on Deep Learning Session 1
Reading Group on Deep Learning Session 1 Stephane Lathuiliere & Pablo Mesejo 2 June 2016 1/31 Contents Introduction to Artificial Neural Networks to understand, and to be able to efficiently use, the popular
More informationMachine Learning
Machine Learning 10-315 Maria Florina Balcan Machine Learning Department Carnegie Mellon University 03/29/2019 Today: Artificial neural networks Backpropagation Reading: Mitchell: Chapter 4 Bishop: Chapter
More informationMachine Learning for Computer Vision 8. Neural Networks and Deep Learning. Vladimir Golkov Technical University of Munich Computer Vision Group
Machine Learning for Computer Vision 8. Neural Networks and Deep Learning Vladimir Golkov Technical University of Munich Computer Vision Group INTRODUCTION Nonlinear Coordinate Transformation http://cs.stanford.edu/people/karpathy/convnetjs/
More informationClassification objectives COMS 4771
Classification objectives COMS 4771 1. Recap: binary classification Scoring functions Consider binary classification problems with Y = { 1, +1}. 1 / 22 Scoring functions Consider binary classification
More informationMachine Learning Lecture 12
Machine Learning Lecture 12 Neural Networks 30.11.2017 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de leibe@vision.rwth-aachen.de Course Outline Fundamentals Bayes Decision Theory Probability
More informationBack-Propagation Algorithm. Perceptron Gradient Descent Multilayered neural network Back-Propagation More on Back-Propagation Examples
Back-Propagation Algorithm Perceptron Gradient Descent Multilayered neural network Back-Propagation More on Back-Propagation Examples 1 Inner-product net =< w, x >= w x cos(θ) net = n i=1 w i x i A measure
More informationBACKPROPAGATION. Neural network training optimization problem. Deriving backpropagation
BACKPROPAGATION Neural network training optimization problem min J(w) w The application of gradient descent to this problem is called backpropagation. Backpropagation is gradient descent applied to J(w)
More informationCSC321 Lecture 5: Multilayer Perceptrons
CSC321 Lecture 5: Multilayer Perceptrons Roger Grosse Roger Grosse CSC321 Lecture 5: Multilayer Perceptrons 1 / 21 Overview Recall the simple neuron-like unit: y output output bias i'th weight w 1 w2 w3
More informationComputational Graphs, and Backpropagation. Michael Collins, Columbia University
Computational Graphs, and Backpropagation Michael Collins, Columbia University A Key Problem: Calculating Derivatives where and p(y x; θ, v) = exp (v(y) φ(x; θ) + γ y ) y Y exp (v(y ) φ(x; θ) + γ y ) φ(x;
More information8-1: Backpropagation Prof. J.C. Kao, UCLA. Backpropagation. Chain rule for the derivatives Backpropagation graphs Examples
8-1: Backpropagation Prof. J.C. Kao, UCLA Backpropagation Chain rule for the derivatives Backpropagation graphs Examples 8-2: Backpropagation Prof. J.C. Kao, UCLA Motivation for backpropagation To do gradient
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationIntroduction to Machine Learning
Introduction to Machine Learning Neural Networks Varun Chandola x x 5 Input Outline Contents February 2, 207 Extending Perceptrons 2 Multi Layered Perceptrons 2 2. Generalizing to Multiple Labels.................
More informationNeed for Deep Networks Perceptron. Can only model linear functions. Kernel Machines. Non-linearity provided by kernels
Need for Deep Networks Perceptron Can only model linear functions Kernel Machines Non-linearity provided by kernels Need to design appropriate kernels (possibly selecting from a set, i.e. kernel learning)
More informationRegML 2018 Class 8 Deep learning
RegML 2018 Class 8 Deep learning Lorenzo Rosasco UNIGE-MIT-IIT June 18, 2018 Supervised vs unsupervised learning? So far we have been thinking of learning schemes made in two steps f(x) = w, Φ(x) F, x
More informationStatistical Machine Learning (BE4M33SSU) Lecture 5: Artificial Neural Networks
Statistical Machine Learning (BE4M33SSU) Lecture 5: Artificial Neural Networks Jan Drchal Czech Technical University in Prague Faculty of Electrical Engineering Department of Computer Science Topics covered
More informationArtificial Neural Networks
Artificial Neural Networks 鮑興國 Ph.D. National Taiwan University of Science and Technology Outline Perceptrons Gradient descent Multi-layer networks Backpropagation Hidden layer representations Examples
More informationApprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning
Apprentissage, réseaux de neurones et modèles graphiques (RCP209) Neural Networks and Deep Learning Nicolas Thome Prenom.Nom@cnam.fr http://cedric.cnam.fr/vertigo/cours/ml2/ Département Informatique Conservatoire
More informationCS489/698: Intro to ML
CS489/698: Intro to ML Lecture 03: Multi-layer Perceptron Outline Failure of Perceptron Neural Network Backpropagation Universal Approximator 2 Outline Failure of Perceptron Neural Network Backpropagation
More informationAN INTRODUCTION TO NEURAL NETWORKS. Scott Kuindersma November 12, 2009
AN INTRODUCTION TO NEURAL NETWORKS Scott Kuindersma November 12, 2009 SUPERVISED LEARNING We are given some training data: We must learn a function If y is discrete, we call it classification If it is
More informationMachine Learning
Machine Learning 10-601 Maria Florina Balcan Machine Learning Department Carnegie Mellon University 02/10/2016 Today: Artificial neural networks Backpropagation Reading: Mitchell: Chapter 4 Bishop: Chapter
More informationFeedforward Neural Networks. Michael Collins, Columbia University
Feedforward Neural Networks Michael Collins, Columbia University Recap: Log-linear Models A log-linear model takes the following form: p(y x; v) = exp (v f(x, y)) y Y exp (v f(x, y )) f(x, y) is the representation
More informationStatistical NLP for the Web
Statistical NLP for the Web Neural Networks, Deep Belief Networks Sameer Maskey Week 8, October 24, 2012 *some slides from Andrew Rosenberg Announcements Please ask HW2 related questions in courseworks
More informationNeural Networks: Backpropagation
Neural Networks: Backpropagation Machine Learning Fall 2017 Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others
More informationIntroduction to (Convolutional) Neural Networks
Introduction to (Convolutional) Neural Networks Philipp Grohs Summer School DL and Vis, Sept 2018 Syllabus 1 Motivation and Definition 2 Universal Approximation 3 Backpropagation 4 Stochastic Gradient
More information10 NEURAL NETWORKS Bio-inspired Multi-Layer Networks. Learning Objectives:
10 NEURAL NETWORKS TODO The first learning models you learned about (decision trees and nearest neighbor models) created complex, non-linear decision boundaries. We moved from there to the perceptron,
More informationMachine Learning: Chenhao Tan University of Colorado Boulder LECTURE 6
Machine Learning: Chenhao Tan University of Colorado Boulder LECTURE 6 Slides adapted from Jordan Boyd-Graber, Chris Ketelsen Machine Learning: Chenhao Tan Boulder 1 of 39 HW1 turned in HW2 released Office
More informationDeep Feedforward Networks
Deep Feedforward Networks Liu Yang March 30, 2017 Liu Yang Short title March 30, 2017 1 / 24 Overview 1 Background A general introduction Example 2 Gradient based learning Cost functions Output Units 3
More informationNeural Networks Lecturer: J. Matas Authors: J. Matas, B. Flach, O. Drbohlav
Neural Networks 30.11.2015 Lecturer: J. Matas Authors: J. Matas, B. Flach, O. Drbohlav 1 Talk Outline Perceptron Combining neurons to a network Neural network, processing input to an output Learning Cost
More informationUnderstanding Neural Networks : Part I
TensorFlow Workshop 2018 Understanding Neural Networks Part I : Artificial Neurons and Network Optimization Nick Winovich Department of Mathematics Purdue University July 2018 Outline 1 Neural Networks
More informationLogistic regression and linear classifiers COMS 4771
Logistic regression and linear classifiers COMS 4771 1. Prediction functions (again) Learning prediction functions IID model for supervised learning: (X 1, Y 1),..., (X n, Y n), (X, Y ) are iid random
More informationSolutions. Part I Logistic regression backpropagation with a single training example
Solutions Part I Logistic regression backpropagation with a single training example In this part, you are using the Stochastic Gradient Optimizer to train your Logistic Regression. Consequently, the gradients
More informationMachine Learning. Neural Networks. Le Song. CSE6740/CS7641/ISYE6740, Fall Lecture 7, September 11, 2012 Based on slides from Eric Xing, CMU
Machine Learning CSE6740/CS7641/ISYE6740, Fall 2012 Neural Networks Le Song Lecture 7, September 11, 2012 Based on slides from Eric Xing, CMU Reading: Chap. 5 CB Learning highly non-linear functions f:
More informationIntroduction to Machine Learning Spring 2018 Note Neural Networks
CS 189 Introduction to Machine Learning Spring 2018 Note 14 1 Neural Networks Neural networks are a class of compositional function approximators. They come in a variety of shapes and sizes. In this class,
More informationDeep Learning: Self-Taught Learning and Deep vs. Shallow Architectures. Lecture 04
Deep Learning: Self-Taught Learning and Deep vs. Shallow Architectures Lecture 04 Razvan C. Bunescu School of Electrical Engineering and Computer Science bunescu@ohio.edu Self-Taught Learning 1. Learn
More informationSGD and Deep Learning
SGD and Deep Learning Subgradients Lets make the gradient cheating more formal. Recall that the gradient is the slope of the tangent. f(w 1 )+rf(w 1 ) (w w 1 ) Non differentiable case? w 1 Subgradients
More informationMachine Learning. Neural Networks. (slides from Domingos, Pardo, others)
Machine Learning Neural Networks (slides from Domingos, Pardo, others) For this week, Reading Chapter 4: Neural Networks (Mitchell, 1997) See Canvas For subsequent weeks: Scaling Learning Algorithms toward
More informationArtificial Neural Networks
Artificial Neural Networks Oliver Schulte - CMPT 310 Neural Networks Neural networks arise from attempts to model human/animal brains Many models, many claims of biological plausibility We will focus on
More informationNeural Networks: Introduction
Neural Networks: Introduction Machine Learning Fall 2017 Based on slides and material from Geoffrey Hinton, Richard Socher, Dan Roth, Yoav Goldberg, Shai Shalev-Shwartz and Shai Ben-David, and others 1
More informationCSC321 Lecture 6: Backpropagation
CSC321 Lecture 6: Backpropagation Roger Grosse Roger Grosse CSC321 Lecture 6: Backpropagation 1 / 21 Overview We ve seen that multilayer neural networks are powerful. But how can we actually learn them?
More informationAn overview of deep learning methods for genomics
An overview of deep learning methods for genomics Matthew Ploenzke STAT115/215/BIO/BIST282 Harvard University April 19, 218 1 Snapshot 1. Brief introduction to convolutional neural networks What is deep
More informationNeural networks. Chapter 20, Section 5 1
Neural networks Chapter 20, Section 5 Chapter 20, Section 5 Outline Brains Neural networks Perceptrons Multilayer perceptrons Applications of neural networks Chapter 20, Section 5 2 Brains 0 neurons of
More information17 Neural Networks NEURAL NETWORKS. x XOR 1. x Jonathan Richard Shewchuk
94 Jonathan Richard Shewchuk 7 Neural Networks NEURAL NETWORKS Can do both classification & regression. [They tie together several ideas from the course: perceptrons, logistic regression, ensembles of
More informationMachine Learning. Neural Networks
Machine Learning Neural Networks Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 Biological Analogy Bryan Pardo, Northwestern University, Machine Learning EECS 349 Fall 2007 THE
More informationBackpropagation Introduction to Machine Learning. Matt Gormley Lecture 13 Mar 1, 2018
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Backpropagation Matt Gormley Lecture 13 Mar 1, 2018 1 Reminders Homework 5: Neural
More informationIntro to Neural Networks and Deep Learning
Intro to Neural Networks and Deep Learning Jack Lanchantin Dr. Yanjun Qi UVA CS 6316 1 Neurons 1-Layer Neural Network Multi-layer Neural Network Loss Functions Backpropagation Nonlinearity Functions NNs
More information4. Multilayer Perceptrons
4. Multilayer Perceptrons This is a supervised error-correction learning algorithm. 1 4.1 Introduction A multilayer feedforward network consists of an input layer, one or more hidden layers, and an output
More informationMultilayer Perceptrons (MLPs)
CSE 5526: Introduction to Neural Networks Multilayer Perceptrons (MLPs) 1 Motivation Multilayer networks are more powerful than singlelayer nets Example: XOR problem x 2 1 AND x o x 1 x 2 +1-1 o x x 1-1
More informationLecture 2: Learning with neural networks
Lecture 2: Learning with neural networks Deep Learning @ UvA LEARNING WITH NEURAL NETWORKS - PAGE 1 Lecture Overview o Machine Learning Paradigm for Neural Networks o The Backpropagation algorithm for
More informationBayesian Networks (Part I)
10-601 Introduction to Machine Learning Machine Learning Department School of Computer Science Carnegie Mellon University Bayesian Networks (Part I) Graphical Model Readings: Murphy 10 10.2.1 Bishop 8.1,
More informationDeep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, Spis treści
Deep learning / Ian Goodfellow, Yoshua Bengio and Aaron Courville. - Cambridge, MA ; London, 2017 Spis treści Website Acknowledgments Notation xiii xv xix 1 Introduction 1 1.1 Who Should Read This Book?
More informationCOMP 551 Applied Machine Learning Lecture 14: Neural Networks
COMP 551 Applied Machine Learning Lecture 14: Neural Networks Instructor: Ryan Lowe (ryan.lowe@mail.mcgill.ca) Slides mostly by: Class web page: www.cs.mcgill.ca/~hvanho2/comp551 Unless otherwise noted,
More informationECE521 Lectures 9 Fully Connected Neural Networks
ECE521 Lectures 9 Fully Connected Neural Networks Outline Multi-class classification Learning multi-layer neural networks 2 Measuring distance in probability space We learnt that the squared L2 distance
More information10-701/ Machine Learning, Fall
0-70/5-78 Machine Learning, Fall 2003 Homework 2 Solution If you have questions, please contact Jiayong Zhang .. (Error Function) The sum-of-squares error is the most common training
More informationClassification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses about the label (Top-5 error) No Bounding Box
ImageNet Classification with Deep Convolutional Neural Networks Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Motivation Classification goals: Make 1 guess about the label (Top-1 error) Make 5 guesses
More informationGradient Descent Training Rule: The Details
Gradient Descent Training Rule: The Details 1 For Perceptrons The whole idea behind gradient descent is to gradually, but consistently, decrease the output error by adjusting the weights. The trick is
More informationRestricted Boltzmann Machines
Restricted Boltzmann Machines Boltzmann Machine(BM) A Boltzmann machine extends a stochastic Hopfield network to include hidden units. It has binary (0 or 1) visible vector unit x and hidden (latent) vector
More information