Lecture 12. Neural Networks Bastian Leibe RWTH Aachen

Advanced Machine Learning Lecture 12 Neural Networks 10.12.2015 Bastian Leibe RWTH Aachen http://www.vision.rwth-aachen.de/ leibe@vision.rwth-aachen.de

This Lecture: Advanced Machine Learning Regression Approaches Linear Regression Regularization (Ridge, Lasso) Gaussian Processes Learning with Latent Variables Prob. Distributions & Approx. Inference Mixture Models EM and Generalizations Deep Learning Linear Discriminants Neural Networks Backpropagation CNNs, RNNs, RBMs, etc.

Recap: Generalized Linear Discriminants Extension with non-linear basis functions Transform vector x with M nonlinear basis functions Á j (x): Basis functions Á j (x) allow non-linear decision boundaries. Activation function g( ) bounds the influence of outliers. Disadvantage: minimization no longer in closed form. Notation with Á 0 (x) = 1 Slide adapted from Bernt Schiele 3

Recap: Gradient Descent Iterative minimization w (0) kj Start with an initial guess for the parameter values. Move towards a (local) minimum by following the gradient. Basic strategies Batch learning w ( +1) kj = w ( ) kj @E(w) @w kj w( ) Sequential updating where w ( +1) kj = w ( ) kj E(w) = @E n(w) @w kj NX E n (w) n=1 w( ) 4

Recap: Gradient Descent Example: Quadratic error function NX E(w) = (y(x n ; w) t n ) 2 n=1 Sequential updating leads to delta rule (=LMS rule) w ( +1) kj = w ( ) kj (y k(x n ; w) t kn ) Á j (x n ) = w ( ) kj ± kná j (x n ) where ± kn = y k (x n ; w) t kn Simply feed back the input data point, weighted by the classification error. Slide adapted from Bernt Schiele 5

Recap: Probabilistic Discriminative Models Consider models of the form p(c 1 já) = y(á) = ¾(w T Á) with p(c 2 já) = 1 p(c 1 já) This model is called logistic regression. Properties Probabilistic interpretation But discriminative method: only focus on decision hyperplane Advantageous for high-dimensional spaces, requires less parameters than explicitly modeling p(á C k ) and p(c k ). 6

Recap: Logistic Regression Let s consider a data set {Á n,t n } with n = 1,,N, Á n = Á(x n ) t n 2 f0; 1g t = (t 1 ; : : : ; t N ) T where and,. With y n = p(c 1 Á n ), we can write the likelihood as p(tjw) = NY n=1 y t n n f1 y n g 1 t n Define the error function as the negative log-likelihood E(w) = ln p(tjw) NX = ft n ln y n + (1 t n ) ln(1 y n )g n=1 This is the so-called cross-entropy error function. 8

Recap: Gradient of the Error Function Gradient for logistic regression NX re(w) = (y n t n )Á n n=1 This is the same result as for the Delta (=LMS) rule w ( +1) kj = w ( ) kj (y k(x n ; w) t kn )Á j (x n ) We can use this to derive a sequential estimation algorithm. However, this will be quite slow More efficient to use 2 nd -order Newton-Raphson IRLS 9

Recap: Softmax Regression Multi-class generalization of logistic regression In logistic regression, we assumed binary labels t n 2 f0; 1g Softmax generalizes this to K values in 1-of-K notation. 2 3 2 P (y = 1jx; w) exp(w > 3 1 x) P (y = 2jx; w) y(x; w) = 6 4 7. 5 = 1 exp(w 2 > P K j=1 exp(w> j x) 6 x) 7 4. 5 P (y = Kjx; w) exp(w K > x) This uses the softmax function Note: the resulting distribution is normalized. 11

Recap: Softmax Regression Cost Function Logistic regression Alternative way of writing the cost function E(w) = = Softmax regression NX ft n ln y n + (1 t n ) ln(1 y n )g n=1 NX 1X n=1 k=0 fi (t n = k) ln P (y n = kjx n ; w)g Generalization to K classes using indicator functions. E(w) = r wk E(w) = NX KX n=1 k=1 ( I (t n = k) ln ) exp(w k > P x) K j=1 exp(w> j x) NX [I (t n = k) ln P (y n = kjx n ; w)] n=1 12

Side Note: Support Vector Machine (SVM) Basic idea The SVM tries to find a classifier which maximizes the margin between pos. and neg. data points. Up to now: consider linear classifiers w T x + b = 0 Margin Formulation as a convex optimization problem Find the hyperplane satisfying arg min w;b under the constraints 1 2 kwk2 t n (w T x n + b) 1 based on training data points x n and target values. 8n t n 2 f 1; 1g 13

SVM Analysis Traditional soft-margin formulation 1 NX min w2r D ;» n 2R + 2 kwk2 + C subject to the constraints Different way of looking at it We can reformulate the constraints into the objective function. 1 min w2r D 2 kwk2 + C n=1» n NX [1 t n y(x n )] + n=1 Maximize the margin Most points should be on the correct side of the margin L 2 regularizer where [x] + := max{0,x}. Slide adapted from Christoph Lampert Hinge loss 14

SVM Error Function (Loss Function) Ideal misclassification error Squared error Hinge error Robust to outliers! Not differentiable! Favors sparse solutions! Hinge error used in SVMs Zero error for points outside the margin (z n > 1). Linearly increasing error for misclassified points (z n < 1). Leads to sparse solutions, not sensitive to outliers. Not differentiable around z n = 1 Cannot be optimized directly. 15 Image source: Bishop, 2006

SVM Discussion SVM optimization function 1 min w2r D 2 kwk2 + C L 2 regularizer NX n=1 [1 t n y(x n )] + Hinge loss Hinge loss enforces sparsity Only a subset of training data points actually influences the decision boundary. This is different from sparsity obtained through the regularizer! There, only a subset of input dimensions are used. Unconstrained optimization, but non-differentiable function. Solve, e.g. by subgradient descent Currently most efficient: stochastic gradient descent Slide adapted from Christoph Lampert 16

Topics of This Lecture A Short History of Neural Networks Perceptrons Definition Loss functions Regularization Limits Multi-Layer Perceptrons Definition Learning 17

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron And a cool learning algorithm: Perceptron Learning Hardware implementation Mark I Perceptron for 20 20 pixel image analysis The embryo of an electronic computer that [...] will be able to walk, talk, see, write, reproduce itself and be conscious of its existence. 18 Image source: Wikipedia, clipartpanda.com

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert They showed that (single-layer) Perceptrons cannot solve all problems. This was misunderstood by many that they were worthless. Neural Networks don t work! 19 Image source: colourbox.de, thinkstock

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert 1980s Resurgence of Neural Networks Some notable successes with multi-layer perceptrons. Backpropagation learning algorithm Oh no! Killer robots will achieve world domination! OMG! They work like the human brain! 20 Image sources: clipartpanda.com, cliparts.co

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert 1980s Resurgence of Neural Networks Some notable successes with multi-layer perceptrons. Backpropagation learning algorithm But they are hard to train, tend to overfit, and have unintuitive parameters. So, the excitement fades again. sigh! 21 Image source: clipartof.com, colourbox.de

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert 1980s Resurgence of Neural Networks 1995+ Interest shifts to other learning methods Notably Support Vector Machines Machine Learning becomes a discipline of its own. I can do research, me! 22 Image source: clipartof.com

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert 1980s Resurgence of Neural Networks 1995+ Interest shifts to other learning methods Notably Support Vector Machines Machine Learning becomes a discipline of its own. The general public and the press still love Neural Networks. I m doing Machine Learning. So, you re using Neural Networks? Actually... 23

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert 1980s Resurgence of Neural Networks 1995+ Interest shifts to other learning methods 2005+ Gradual progress Better understanding how to successfully train deep networks Availability of large datasets and powerful GPUs Still largely under the radar for many disciplines applying ML Are you using Neural Networks? Come on. Get real! 24

A Brief History of Neural Networks 1957 Rosenblatt invents the Perceptron 1969 Minsky & Papert 1980s Resurgence of Neural Networks 1995+ Interest shifts to other learning methods 2005+ Gradual progress 2012 Breakthrough results ImageNet Large Scale Visual Recognition Challenge A ConvNet halves the error rate of dedicated vision approaches. Deep Learning is widely adopted. It works! 25 Image source: clipartpanda.com, clipartof.com

Topics of This Lecture A Short History of Neural Networks Perceptrons Definition Loss functions Regularization Limits Multi-Layer Perceptrons Definition Learning 26

Perceptrons (Rosenblatt 1957) Standard Perceptron Output layer Input Layer Hand-designed features based on common sense Outputs Weights Input layer Linear outputs Logistic outputs Learning = Determining the weights w 27

Extension: Multi-Class Networks One output node per class Output layer Weights Input layer Outputs Linear outputs Logistic outputs Can be used to do multidimensional linear regression or multiclass classification. Slide adapted from Stefan Roth 28

Extension: Non-Linear Basis Functions Straightforward generalization Output layer Weights Feature layer Mapping (fixed) Input layer Outputs Linear outputs Logistic outputs 29

Extension: Non-Linear Basis Functions Straightforward generalization Output layer Weights Feature layer Mapping (fixed) Input layer Remarks Perceptrons are generalized linear discriminants! Everything we know about the latter can also be applied here. Note: feature functions Á(x) are kept fixed, not learned! 30

Perceptron Learning Very simple algorithm Process the training cases in some permutation If the output unit is correct, leave the weights alone. If the output unit incorrectly outputs a zero, add the input vector to the weight vector. If the output unit incorrectly outputs a one, subtract the input vector from the weight vector. This is guaranteed to converge to a correct solution if such a solution exists. Slide adapted from Geoff Hinton 31

Perceptron Learning Let s analyze this algorithm... Process the training cases in some permutation If the output unit is correct, leave the weights alone. If the output unit incorrectly outputs a zero, add the input vector to the weight vector. If the output unit incorrectly outputs a one, subtract the input vector from the weight vector. Translation w ( +1) kj = w ( ) kj (y k(x n ; w) t kn ) Á j (x n ) Slide adapted from Geoff Hinton 32

Loss Functions We can now also apply other loss functions L2 loss Least-squares regression L1 loss: Median regression Cross-entropy loss Logistic regression Hinge loss SVM classification Softmax loss L(t; y(x)) = P n P k Multi-class probabilistic classification n I (t n = k) ln exp(y k(x)) P j exp(y j(x)) o 34

Regularization In addition, we can apply regularizers E.g., an L2 regularizer This is known as weight decay in Neural Networks. We can also apply other regularizers, e.g. L1 sparsity Since Neural Networks often have many parameters, regularization becomes very important in practice. We will see more complex regularization techniques later on... 35

Limitations of Perceptrons What makes the task difficult? Perceptrons with fixed, hand-coded input features can model any separable function perfectly......given the right input features. For some tasks this requires an exponential number of input features. E.g., by enumerating all possible binary input vectors as separate feature units (similar to a look-up table). But this approach won t generalize to unseen test cases! It is the feature design that solves the task! Once the hand-coded features have been determined, there are very strong limitations on what a perceptron can learn. Classic example: XOR function. 36

Wait... Didn t we just say that... Perceptrons correspond to generalized linear discriminants And Perceptrons are very limited... Doesn t this mean that what we have been doing so far in this lecture has the same problems??? Yes, this is the case. A linear classifier cannot solve certain problems (e.g., XOR). However, with a non-linear classifier based on the right kind of features, the problem becomes solvable. So far, we have solved such problems by hand-designing good features Á and kernels Á > Á. Can we also learn such feature representations? 37

Topics of This Lecture A Short History of Neural Networks Perceptrons Definition Loss functions Regularization Limits Multi-Layer Perceptrons Definition Learning 38

Multi-Layer Perceptrons Adding more layers Output layer Hidden layer Input layer Output Slide adapted from Stefan Roth 39

Multi-Layer Perceptrons Activation functions g (k) : For example: g (2) (a) = ¾(a), g (1) (a) = a The hidden layer can have an arbitrary number of nodes There can also be multiple hidden layers. Universal approximators A 2-layer network (1 hidden layer) can approximate any continuous function of a compact domain arbitrarily well! (assuming sufficient hidden nodes) Slide credit: Stefan Roth 40

Learning with Hidden Units Networks without hidden units are very limited in what they can learn More layers of linear units do not help still linear Fixed output non-linearities are not enough. We need multiple layers of adaptive non-linear hidden units. But how can we train such nets? Need an efficient way of adapting all weights, not just the last layer. Learning the weights to the hidden units = learning features This is difficult, because nobody tells us what the hidden units should do. Next lecture Slide adapted from Geoff Hinton 41

References and Further Reading More information on Neural Networks can be found in Chapters 6 and 7 of the Goodfellow & Bengio book Ian Goodfellow, Aaron Courville, Yoshua Bengio Deep Learning MIT Press, in preparation https://goodfeli.github.io/dlbook/ 42