Neural Networks and Deep Learning www.cs.wisc.edu/~dpage/cs760/ 1
Goals for the lecture you should understand the following concepts perceptrons the perceptron training rule linear separability hidden units multilayer neural networks gradient descent stochastic (online) gradient descent sigmoid function gradient descent with a linear output unit gradient descent with a sigmoid output unit backpropagation 2
Goals for the lecture you should understand the following concepts weight initialization early stopping the role of hidden units input encodings for neural networks output encodings recurrent neural networks autoencoders stacked autoencoders 3
Neural networks a.k.a. artificial neural networks, connectionist models inspired by interconnected neurons in biological systems simple processing units each unit receives a number of real-valued inputs each unit produces a single real-valued output 4
Perceptrons [McCulloch & Pitts, 1943; Rosenblatt, 1959; Widrow & Hoff, 1960] 1 w 0 x 1 x 2 w 1 w 2 o = " $ # $ % 1 if w 0 + w i x i > 0 n i=1 0 otherwise x n w n input units: represent given x output unit: represents binary classification 5
Learning a perceptron: the perceptron training rule 1. randomly initialize weights 2. iterate through training instances until convergence 2a. calculate the output for the given instance o = " $ # $ % 1 if w 0 + w i x i > 0 n i=1 0 otherwise 2b. update each weight Δw i = η( y o)x i η is learning rate; set to value << 1 w i w i + Δw i 6
Representational power of perceptrons perceptrons can represent only linearly separable concepts " $ o = # $ % decision boundary given by: 1 if w 0 + w i x i > 0 n i=1 0 otherwise w 1 if w 0 + w 1 x 1 + w 2 x 2 > 0 also write as: wx > 0 w 1 x 1 + w 2 x 2 = w 0 x 2 = w 1 w 2 x 1 w 0 w 2 x 2 + + + + + + - + - - + + + + - + + - - - + + - - + - - - - - x 1 7
Representational power of perceptrons in previous example, feature space was 2D so decision boundary was a line in higher dimensions, decision boundary is a hyperplane 8
Some linearly separable functions AND a b c d x 1 x 2 0 0 0 1 1 0 1 1 y 0 0 0 1 x 1 1 b d a c 0 1 x 2 OR a b c d x 1 x 2 0 0 0 1 1 0 1 1 y 0 1 1 1 x 1 1 b d a c 0 1 x 2 9
XOR is not linearly separable a b c x 1 x 2 0 0 0 1 1 0 1 1 y 0 1 1 0 a c d 0 1 x 1 1 b d x 2 a multilayer perceptron can represent XOR x 1 1-1 -1 x 2 1 1 1 assume w 0 = 0 for all nodes 10
Example multilayer neural network output units hidden units input units figure from Huang & Lippmann, NIPS 1988 input: two features from spectral analysis of a spoken sound output: vowel sound occurring in the context h d 11
Decision regions of a multilayer neural network figure from Huang & Lippmann, NIPS 1988 input: two features from spectral analysis of a spoken sound output: vowel sound occurring in the context h d 12
Learning in multilayer networks work on neural nets fizzled in the 1960 s single layer networks had representational limitations (linear separability) no effective methods for training multilayer networks x 1 how to determine error signal for hidden units? x 2 revived again with the invention of backpropagation method [Rumelhart & McClelland, 1986; also Werbos, 1975] key insight: require neural network to be differentiable; use gradient descent 13
Gradient descent in weight space { } D = (x (1), y (1) ) (x (m), y (m) ) Given a training set we can specify an error measure that is a function of our weight vector w E(w) = 1 2 ( y (d ) o (d ) ) 2 d D figure from Cho & Chow, Neurocomputing 1999 w 2 w 1 This error measure defines a surface over the hypothesis (i.e. weight) space 14
Gradient descent in weight space gradient descent is an iterative process aimed at finding a minimum in the error surface on each iteration current weights define a point in this space find direction in which error surface descends most steeply take a step (i.e. update weights) in that direction Error w 1 w 2 15
Gradient descent in weight space calculate the gradient of E: E(w) = # % $ E, E,, E w 0 w 1 w n & ( ' take a step in the opposite direction Δw = η E( w) Δw i = η E w i Error E w 1 w 1 w 2 16 E w 2
The sigmoid function to be able to differentiate E with respect to w i, our network must represent a continuous function to do this, we use sigmoid functions instead of threshold functions in our hidden and output units f (x) = 1 1+ e x x 17
The sigmoid function for the case of a single-layer network f (x) = 1+ e 1 # % w 0 + $ n i=1 & w i x i ( ' n w 0 + w i x i 18 i=1
Batch neural network training given: network structure and a training set initialize all weights in w to small random numbers until stopping criteria met do initialize the error E(w) = 0 for each (x (d), y (d) ) in the training set input x (d) to the network and compute output o (d) increment the error calculate the gradient # E(w) = E, E,, E & % w 0 w 1 w ( $ n ' update the weights { } D = (x (1), y (1) ) (x (m), y (m) ) ( ) 2 E(w) = E(w) + 1 2 y(d ) o (d ) Δw = η E( w) 19
Online vs. batch training Standard gradient descent (batch training): calculates error gradient for the entire training set, before taking a step in weight space Stochastic gradient descent (online training): calculates error gradient for a single instance, then takes a step in weight space much faster convergence less susceptible to local minima 20
Online neural network training (stochastic gradient descent) given: network structure and a training set initialize all weights in w to small random numbers until stopping criteria met do for each (x (d), y (d) ) in the training set input x (d) to the network and compute output o (d) calculate the error calculate the gradient { } D = (x (1), y (1) ) (x (m), y (m) ) ( ) 2 E(w) = 1 2 y(d ) o (d ) E(w) = # % E, E,, E $ w 0 w 1 w n & ( ' update the weights Δw = η E w ( ) 21
Convergence of gradient descent gradient descent will converge to a minimum in the error function for a multi-layer network, this may be a local minimum (i.e. there may be a better solution elsewhere in weight space) for a single-layer network, this will be a global minimum (i.e. gradient descent will find the best solution) 22
Taking derivatives in neural nets recall the chain rule from calculus y = f (u) u = g(x) y x = y u u x we ll make use of this as follows E w i = E o o net net w i net = w 0 + n w i x i i=1 23
Gradient descent: simple case Consider a simple case of a network with one linear output unit and no hidden units: o (d ) = w 0 + w i x (d ) i n i=1 1 w 0 x w 1 1 let s learn w i s that minimize squared error E(w) = 1 2 ( y (d ) o (d ) ) 2 d D x 2 x n w 2 w n batch case E w i = 1 w i 2 ( y (d ) o (d ) ) 2 d D online case E (d ) = 1 ( w i w i 2 y(d ) o (d )) 2 24
Stochastic gradient descent: simple case let s focus on the online case (stochastic gradient descent): E (d ) w i = 1 ( w i 2 y(d ) o (d )) 2 = ( y (d ) o (d ) ) ( y (d ) o (d )) w i ( ) o(d ) = y (d ) o (d ) # $ % w i & ' ( = ( y (d ) o (d ) ) o(d ) net (d ) = ( y (d ) o (d )) net (d ) net (d ) w i w i = ( y (d ) o (d ) )( (d ) x ) i 25
Gradient descent with a sigmoid Now let s consider the case in which we have a sigmoid output unit and no hidden units: net (d ) = w 0 + w i x (d ) i n i=1 1 o (d ) = 1+ e net(d ) 1 w 0 x 1 w 1 x 2 w 2 useful property: o (d ) net (d ) = o(d ) (1 o (d ) ) x n w n 26
Stochastic GD with sigmoid output unit E (d ) w i = 1 ( w i 2 y(d ) o (d )) 2 = ( y (d ) o (d ) ) ( y (d ) o (d )) w i ( ) o(d ) = y (d ) o (d ) # $ % w i & ' ( = ( y (d ) o (d ) ) o(d ) net (d ) net (d ) w i = ( y (d ) o (d ) )o (d ) (1 o (d ) ) net (d ) w i = ( y (d ) o (d ) )o (d ) (1 o (d ) (d ) )x i 27
Backpropagation now we ve covered how to do gradient descent for single-layer networks with linear output units sigmoid output units E how can we calculate for every weight in a multilayer network? w i è backpropagate errors from the output units to the hidden units 28
Backpropagation notation let s consider the online case, but drop the (d) superscripts for simplicity we ll use subscripts on y, o, net to indicate which unit they refer to subscripts to indicate the unit a weight emanates from and goes to i w ji j o j 29
Backpropagation each weight is changed by Δw ji = η E w ji = η E net j net j w ji = η δ j o i where δ j = E net j x i if i is an input unit 30
Backpropagation each weight is changed by Δw ji = η δ j o i where δ j = E net j δ j = o j (1 o j )(y j o j ) δ j = o j (1 o j ) k δ k w kj if j is an output unit if j is a hidden unit same as single-layer net with sigmoid output sum of backpropagated contributions to error 31
Backpropagation illustrated 1. calculate error of output units δ j = o j (1 o j )(y j o j ) 2. determine updates for weights going to output units Δw ji = η δ j o i j 32
Backpropagation illustrated 3. calculate error for hidden units δ j = o j (1 o j ) k δ k w kj 4. determine updates for weights to hidden units using hidden-unit errors Δw ji = η δ j o i j j 33
Neural network jargon activation: the output value of a hidden or output unit epoch: one pass through the training instances during gradient descent transfer function: the function used to compute the output of a hidden/ output unit from the net input Minibatch: in practice, randomly partition data into many parts (e.g., 10 examples each), and compute SGD gradient for one step with one of these parts (minibatches) instead of just one example 34
Initializing weights Weights should be initialized to small values so that the sigmoid activations are in the range where the derivative is large (learning will be quicker) random values to ensure symmetry breaking (i.e. if all weights are the same, the hidden units will all represent the same thing) typical initial weight range [-0.01, 0.01] 35
Setting the learning rate convergence depends on having an appropriate learning rate Error E w ij w ij η too small (error goes down a little) η too large (error goes up) 36
Stopping criteria conventional gradient descent: train until local minimum reached empirically better approach: early stopping use a validation set to monitor accuracy during training iterations return the weights that result in minimum validation-set error error stop training here training iterations 37
Input (feature) encoding for neural networks nominal features are usually represented using a 1-of-k encoding A = 1 0 0 0! " # # # # $ % & & & & C = 0 1 0 0! " # # # # $ % & & & & G = 0 0 1 0! " # # # # $ % & & & & T = 0 0 0 1! " # # # # $ % & & & & ordinal features can be represented using a thermometer encoding small = 1 0 0! " # # # $ % & & & medium = 1 1 0! " # # # $ % & & & large = 1 1 1! " # # # $ % & & & precipitation = 0.68 [ ] real-valued features can be represented using individual input units (we may want to scale/normalize them first though) 38
Output encoding for neural networks regression tasks usually use output units with linear transfer functions binary classification tasks usually use one sigmoid output unit k-ary classification tasks usually use k sigmoid or softmax output units o i = e net i e net j j outputs 39
Recurrent neural networks recurrent networks are sometimes used for tasks that involve making sequences of predictions Elman networks: recurrent connections go from hidden units to inputs Jordan networks: recurrent connections go from output units to inputs 40
Alternative approach to training deep networks use unsupervised learning to to find useful hidden unit representations 41
Learning representations the feature representation provided is often the most significant factor in how well a learning system works an appealing aspect of multilayer neural networks is that they are able to change the feature representation can think of the nodes in the hidden layer as new features constructed from the original features in the input layer consider having more levels of constructed features, e.g., pixels -> edges -> shapes -> faces or other objects 42
Competing intuitions Only need a 2-layer network (input, hidden layer, output) Representation Theorem (1989): Using sigmoid activation functions (more recently generalized to others as well), can represent any continuous function with a single hidden layer Empirically, adding more hidden layers does not improve accuracy, and it often degrades accuracy, when training by standard backpropagation Deeper networks are better More efficient representationally, e.g., can represent n-variable parity function with polynomially many (in n) nodes using multiple hidden layers, but need exponentially many (in n) nodes when limited to a single hidden layer More structure, should be able to construct more interesting derived features 43
The role of hidden units Hidden units transform the input space into a new space where perceptrons suffice They numerically represent constructed features Consider learning the target function using the network structure below: 44
The role of hidden units In this task, hidden units learn a compressed numerical coding of the inputs/outputs 45
How many hidden units should be used? conventional wisdom in the early days of neural nets: prefer small networks because fewer parameters (i.e. weights & biases) will be less likely to overfit somewhat more recent wisdom: if early stopping is used, larger networks often behave as if they have fewer effective hidden units, and find better solutions 4 HUs test set error 15 HUs Figure from Weigend, Proc. of the CMSS 1993 training epochs 46
Another way to avoid overfitting Allow many hidden units but force each hidden unit to output mostly zeroes: tend to meaningful concepts Gradient descent solves an optimization problem add a regularizing term to the objective function Let X be vector of random variables, one for each hidden unit, giving average output of unit over data set. Let target distribution s have variables independent with low probability of outputting one (say 0.1), and let ŝ be empirical distribution in the data set. Add to the backpropagation target function (that minimizes δ s) a penalty of KL(s(X) ŝ(x)) 47
Backpropagation with multiple hidden layers in principle, backpropagation can be used to train arbitrarily deep networks (i.e. with multiple hidden layers) in practice, this doesn t usually work well there are likely to be lots of local minima diffusion of gradients leads to slow training in lower layers gradients are smaller, less pronounced at deeper levels errors in credit assignment propagate as you go back 48
Autoencoders one approach: use autoencoders to learn hidden-unit representations in an autoencoder, the network is trained to reconstruct the inputs 49
Autoencoder variants how to encourage the autoencoder to generalize bottleneck: use fewer hidden units than inputs sparsity: use a penalty function that encourages most hidden unit activations to be near 0 [Goodfellow et al. 2009] denoising: train to predict true input from corrupted input [Vincent et al. 2008] contractive: force encoder to have small derivatives (of hidden unit output as input varies) [Rifai et al. 2011] 50
Stacking Autoencoders can be stacked to form highly nonlinear representations [Bengio et al. NIPS 2006] train autoencoder to represent x Discard output layer; train autoencoder to represent h 1 Repeat for k layers discard output layer; train weights on last layer for supervised task each W i here represents the matrix of weights between layers 51
Fine-Tuning After completion, run backpropagation on the entire network to fine-tune weights for the supervised task Because this backpropagation starts with good structure and weights, its credit assignment is better and so its final results are better than if we just ran backpropagation initially 52
Why does the unsupervised training step work well? regularization hypothesis: representations that are good for P(x) are good for P(y x) optimization hypothesis: unsupervised initializations start near better local minima of supervised training error 53
Deep learning not limited to neural networks First developed by Geoff Hinton and colleagues for belief networks, a kind of hybrid between neural nets and Bayes nets Hinton motivates the unsupervised deep learning training process by the credit assignment problem, which appears in belief nets, Bayes nets, neural nets, restricted Boltzmann machines, etc. d-separation: the problem of evidence at a converging connection creating competing explanations backpropagation: can t choose which neighbors get the blame for an error at this node 54
Room for Debate many now arguing that unsupervised pre-training phase not really needed backprop is sufficient if done better wider diversity in initial weights, try with many initial settings until you get learning don t worry much about exact learning rate, but add momentum: if moving fast in a given direction, keep it up for awhile Need a lot of data for deep net backprop 55
Problems with Backprop for Deep Neural Networks Overfits both training data and the particular starting point Converges too quickly to a suboptimal solution, even with SGD (gradient from one example or minibatch of examples at one time) Need more training data and/or fewer weights to estimate, or other regularizer 56
Trick 1: Data Augmentation Deep learning depends critically on Big Data need many more training examples than features Turn one positive (negative) example into many positive (negative) examples Image data: rotate, re-scale, or shift image, or flip image about axis; image still contains the same objects, exhibits the same event or action 57
Trick 2: Parameter (Weight) Tying Normally all neurons at one layer are connected to next layer Instead, have only n features feed to one specific neuron at next level (e.g., 4 or 9 pixels of image go to one hidden unit summarizing this super-pixel ) Tie the 4 (or 9) input weights across all superpixels more data per weight 58
Weight Tying Example: Convolution Have a sliding window (e.g., square of 4 pixels, set of 5 consecutive items in a sequence, etc), and only the neurons for these inputs feed into one neuron, N1, at the next layer. Slide this window over by some amount and repeat, feeding into another neuron, N2, etc. Tie the input weights for N1, N2, etc., so they will all learn the same concept (e.g., diagonal edge). Repeat into new neurons N1, N2, etc., to learn other concepts. 59
Alternate Convolutional Layer with Pooling Layer Mean pooling: k nodes (e.g., corresponding to 4 pixels constituting a square in an image) are averaged to create one node (e.g., corresponding to one pixel) at the next layer. Max pooling: replace average with maximum 60
Used in Convolutional Neural Networks for Vision Applications 61
Trick 3: Alternative Activation Search MathWorld Tanh: (e 2x -1)/(e 2x +1) hyperbolic tangent eural networks) - Wikipedia https://en.wikipedia.org/wiki/rectifier_(neural_networks) ectifier (neural networks) e ReLU: max(0,x) or 1/(1+e -x ) om Wikipedia, the free encyclopedia the context of artificial neural networks, the rectifier is an tivation function defined as rectified linear unit or softplus here x is the input to a neuron. This is also known as a ramp nction and is analogous to half-wave rectification in electrical gineering. This activation function was first introduced to a namical network by Hahnloser et al. in a 2000 paper in Nature [1] ith strong biological motivations and mathematical justifications. [2] has been used in convolutional networks [3] more effectively than e widely used logistic sigmoid (which is inspired by probability Plot of the rectifier (blue) and softplus (green) functions near x = 0 62
Trick 4: Alternative Error Function Example: Cross-entropy C = 1 [y ln o a + (1 y) ln(1 o a)] n x 63
Trick 5: Momentum 64
Trick 6: Dropout Training Build some redundancy into the hidden units Essentially create an ensemble of neural networks, but without high cost of training many deep networks Dropout training 65
Dropout training On each training iteration, drop out (ignore) 50% of the units (or other 90%, or other) by forcing output to 0 during forward pass Ignore for forward & backprop (all training) Figures from Srivastava et al., Journal of Machine Learning Research 2014 66
At Test Time Final model uses all nodes Multiply each weight from a node by fraction of times node was used during training Figures from Srivastava et al., Journal of Machine Learning Research 2014 67
Trick 7: Batch Normalization If outputs of earlier layers are uniform or change greatly on one round for one mini-batch, then neurons at next levels can t keep up: they output all high (or all low) values Next layer doesn t have ability to change its outputs with learning-rate-sized changes to its input weights We say the layer has saturated 68
Another View of Problem In ML, we assume future data will be drawn from same probability distribution as training data For a hidden unit, after training, the earlier layers have new weights and hence generate input data for this hidden unit from a new distribution Want to reduce this internal covariate shift for the benefit of later layers 69
Input: Values of x over a mini-batch: B = {x 1...m }; Parameters to be learned:, Output: {y i = BN, (x i )} µ B 2 B bx i 1 m 1 m mx i=1 x i // mini-batch mean mx (x i µ B ) 2 // mini-batch variance i=1 x i µ B p 2B + // normalize y i bx i + BN, (x i ) // scale and shift Algorithm 1: Batch Normalizing Transform, applied to activation x over a mini-batch. 70
Comments on Batch Normalization First three steps are just like standardization of input data, but with respect to only the data in mini-batch. Can take derivative and incorporate the learning of last step parameters into backpropagation. Note last step can completely un-do previous 3 steps But if so this un-doing is driven by the later layers, not the earlier layers; later layers get to choose whether they want standard normal inputs or not 71
Some Deep Learning Resources Nature, Jan 8, 2014: http://www.nature.com/news/computer-science-thelearning-machines-1.14481 Ng Tutorial: http://deeplearning.stanford.edu/wiki/index.php/ UFLDL_Tutorial Hinton Tutorial: http://videolectures.net/jul09_hinton_deeplearn/ LeCun & Ranzato Tutorial: http://www.cs.nyu.edu/ ~yann/talks/lecun-ranzato-icml2013.pdf 72
Comments on neural networks stochastic gradient descent often works well for very large data sets backpropagation generalizes to arbitrary numbers of output and hidden units arbitrary layers of hidden units (in theory) arbitrary connection patterns other transfer (i.e. output) functions other error measures backprop doesn t usually work well for networks with multiple layers of hidden units; recent work in deep networks addresses this limitation 73