100 inference steps doesn't seem like enough. Many neuron-like threshold switching units. Many weighted interconnections among units

Transcription

1 Connectionist Models Consider humans: Neuron switching time ~ :001 second Number of neurons ~ Connections per neuron ~ Scene recognition time ~ :1 second 100 inference steps doesn't seem like enough! much parallel computation Properties of articial neural nets (ANN's): Many neuron-like threshold switching units Many weighted interconnections among units Highly parallel, distributed process Emphasis on tuning weights automatically 75 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

2 When to Consider Neural Networks Input is high-dimensional discrete or real-valued (e.g. raw sensor input) Output is discrete or real valued Output is a vector of values Possibly noisy data Form of target function is unknown Human readability of result is unimportant Examples: Speech phoneme recognition [Waibel] Image classication [Kanade, Baluja, Rowley] Financial prediction 76 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

3 ALVINN drives 70 mph on highways Sharp Left Straight Ahead Sharp Right 30 Output Units 4 Hidden Units 30x32 Sensor Input Retina 77 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

4 Perceptron x 1 w 1 x 0 =1 x 2 x n... w 2 w n o(x 1 ; : : : ; x n ) = w 0 8>< >: Σ n Σ w i x i i=0 { n 1 if Σ w > 0 o = i x i=0 i -1 otherwise 1 if w 0 + w 1 x w n x n > 0 1 otherwise. Sometimes we'll use simpler vector notation: o(~x) = 8>< >: 1 if ~w ~x > 0 1 otherwise. 78 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

5 Decision Surface of a Perceptron x 2 + x x x 1 (a) (b) Represents some useful functions What weights represent g(x 1 ; x 2 ) = AND(x 1 ; x 2 )? But some functions not representable e.g., not linearly separable Therefore, we'll want networks of these lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

6 Perceptron training rule where w i w i + w i w i = (t o)x i Where: t = c(~x) is target value o is perceptron output is small constant (e.g.,.1) called learning rate 80 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

7 Perceptron training rule Can prove it will converge If training data is linearly separable and suciently small 81 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

8 Gradient Descent Gradient-Descent(training examples; ) Each training example is a pair of the form h~x; ti, where ~x is the vector of input values, and t is the target output value. is the learning rate (e.g.,.05). Initialize each w i to some small random value Until the termination condition is met, Do { Initialize each w i to zero. { For each h~x; ti in training examples, Do Input the instance ~x to the unit and compute the output o For each linear unit weight w i, Do w i w i + (t o)x i { For each linear unit weight w i, Do w i w i + w i 85 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

9 Summary Perceptron training rule guaranteed to succeed if Training examples are linearly separable Suciently small learning rate Linear unit training rule uses gradient descent Guaranteed to converge to hypothesis with minimum squared error Given suciently small learning rate Even when training data contains noise Even when training data not separable by H 86 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

10 Multilayer Networks of Sigmoid Units head hid who d hood F1 F2 88 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

11 Sigmoid Unit x 1 w 1 x 0 = 1 x 2 x n... w 2 w n w 0 Σ n net = Σ w i x i=0 i o = σ(net) = ē net (x) is the sigmoid function Nice property: d(x) dx e x = (x)(1 (x)) We can derive gradient decent rules to train One sigmoid unit Multilayer networks of sigmoid units! Backpropagation 89 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

12 Backpropagation Algorithm Initialize all weights to small random numbers. Until satised, Do For each training example, Do 1. Input the training example to the network and compute the network outputs 2. For each output unit k 3. For each hidden unit h k o k (1 o k )(t k o k ) h o h (1 o h ) X k2outputs 4. Update each network weight w i;j w h;k k where w i;j w i;j + w i;j w i;j = j x i;j 91 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

13 More on Backpropagation Gradient descent over entire network weight vector Easily generalized to arbitrary directed graphs Will nd a local, not necessarily global error minimum { In practice, often works well (can run multiple times) Often include weight momentum w i;j (n) = j x i;j + w i;j (n 1) Minimizes error over training examples { Will it generalize well to subsequent examples? Training can take thousands of iterations! slow! Using network after training is very fast 92 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

14 Learning Hidden Layer Representations Inputs Outputs A target function: Input Output ! ! ! ! ! ! ! ! Can this be learned?? 93 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

15 Learning Hidden Layer Representations A network: Inputs Outputs Learned hidden layer representation: Input Hidden Output Values ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

16 Overtting in ANNs Error Error Error versus weight updates (example 1) Training set error Validation set error Number of weight updates Error versus weight updates (example 2) Training set error Validation set error Number of weight updates 100 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

17 Neural Nets for Face Recognition left strt rght up x32 inputs Typical input images 90% accurate learning head pose, and recognizing 1-of-20 faces 101 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

18 Learned Hidden Unit Weights left strt rght up Learned Weights x32 inputs Typical input images lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997

19 Recurrent Networks y(t + 1) y(t + 1) b x(t) x(t) c(t) (a) Feedforward network (b) Recurrent network y(t + 1) x(t) c(t) y(t) x(t 1) c(t 1) y(t 1) (c) Recurrent network unfolded in time x(t 2) c(t 2) 104 lecture slides for textbook Machine Learning, T. Mitchell, McGraw Hill, 1997