Chapter ML:VI (continued)

Transcription

1 Chapter ML:VI (continued) VI. Neural Networks Perceptron Learning Gradient Descent Multilayer Perceptron Radial asis Functions ML:VI-56 Neural Networks STEIN

2 Definition 1 (Linear Separability) Two sets of feature vectors, X 0, X 1, of a p-dimensional feature space are called linearly separable, if p + 1 real numbers, θ, w 1,..., w p, exist such that holds: 1. x X 0 : p j=1 w jx j < θ 2. x X 1 : p j=1 w jx j θ ML:VI-57 Neural Networks STEIN

3 Definition 1 (Linear Separability) Two sets of feature vectors, X 0, X 1, of a p-dimensional feature space are called linearly separable, if p + 1 real numbers, θ, w 1,..., w p, exist such that holds: 1. x X 0 : p j=1 w jx j < θ 2. x X 1 : p j=1 w jx j θ x 2 x 1 x 2 x 1 linearly separable not linearly separable ML:VI-58 Neural Networks STEIN

4 Separability The XOR function defines the smallest example for two not linearly separable sets: x 2 = 1 x 1 x 2 XOR Class x 2 = 0 x 1 = 0 x 1 = 1 ML:VI-59 Neural Networks STEIN

5 Separability (continued) The XOR function defines the smallest example for two not linearly separable sets: x 2 = 1 x 1 x 2 XOR Class x 2 = 0 x 1 = 0 x 1 = 1 specification of several hyperplanes combination of several perceptrons ML:VI-60 Neural Networks STEIN

6 Separability (continued) Layered combination of several perceptrons: the multilayer perceptron. Minimum multilayer perceptron that is able to handle the XOR problem: x 1 = Σ, θ Σ, θ {, } x 2 = Σ, θ ML:VI-61 Neural Networks STEIN

7 Remarks: The multilayer perceptron was presented by Rumelhart and McClelland in Earlier, but unnoticed, was a similar research work of Werbos and Parker [1974, 1982]. Compared to a single perceptron the multilayer perceptron poses a significantly more challenging training (= learning) problem, which requires continuous threshold functions and sophisticated learning strategies. Marvin Minsky and Seymour Papert showed 1969 with the XOR problem the limitations of single perceptrons. Moreover, they assumed that extensions of the perceptron architecture (such as the multilayer perceptron) would be similarly limited as a single perceptron. fatal mistake. In fact, they brought the research in this field to a halt that lasted 17 years. [erkeley] [Marvin Minsky] ML:VI-62 Neural Networks STEIN

8 Computation in the Network [Heaviside] perceptron with a continuous, non-linear threshold function: x 0 =1 w 0 = θ x 1 w 1 x 2. w 2. w p Σ 0θ y x p Inputs Output The sigmoid function σ(z) as threshold function: σ(z) = 1 where dσ(z) 1 + e z dz = σ(z) (1 σ(z)) ML:VI-63 Neural Networks STEIN

9 Computation in the Network (continued) Computation of the perceptron output y(x) via the sigmoid function σ: y(x) = σ(w T x) = e wt x 1 0 Σ w j x j j=0 p n alternative to the sigmoid function is the tanh function: 1 tanh(x) = ex e x e x + e = e2x 1 x e 2x Σ w j x j j=0 p ML:VI-64 Neural Networks STEIN

10 Computation in the Network (continued) Distinguish units (nodes, perceptrons) of type input, hidden, and output: U I U H U O x 0 =1 = Σ y 1 x 1 x p =. = Σ Σ. Σ Σ. y 2 y k feed forward U I, U H, U O x u v w uv, w uv δ u y u Sets with units of type input, hidden, and output Input value for unit v, provided via output of unit u Weight and weight adaptation for edge connecting units u and v Classification error of unit u Output value of unit u ML:VI-65 Neural Networks STEIN

11 Remarks: The units of the input layer, U I, perform no computations at all. They distribute the input values to the next layer. The non-linear characteristic of the sigmoid function make networks possible that approximate every function. To achieve this flexibility, only three active layers are required, i.e., two layers with hidden units and one layer with output units. Keywords: universal approximator, [Kolmogorov Theorem, 1957] The generic delta rule allows for a backpropagagtion of the classification error and hence the training of multi-layered networks. Gradient descent is based on the classification error of thee entire network and hence considers the entire network weight vector. Multilayer perceptrons are also called multilayer networks or (artificial) neural networks, abbreviated as NN. ML:VI-66 Neural Networks STEIN

12 Classification Error The classification error Err(w) is computed as sum over the U O = k network outputs: Err(w) = 1 (c v (x) y v (x)) 2 2 v U O (x,c(x)) D Due its complex form, Err(w) may contain various local minima: Err(w) ML:VI-67 Neural Networks STEIN

13 Weight daptation: Incremental Gradient Descent [network] lgorithm: MPT Multilayer Perceptron Training Input: D Training examples of the form (x, c(x)) with x = p + 1, c(x) {0, 1} k. η Learning rate, a small positive constant. Output: w Weights for the units in U I, U H, U O. 1. initialize_random_weights(u I, U H, U O ), t = 0 2. REPET 3. t = t FORECH (x, c(x)) D DO 5. FORECH u U H DO y u = propagate(x, u) // layer 1 6. FORECH v U O DO y v = propagate(y H, v) // layer 2 7. FORECH v U O DO δ v = y v (1 y v ) (c v (x) y v ) // backpropagate layer 2 8. FORECH u U H DO δ u = y u (1 y u ) v U o w uv δ v // backpropagate layer 1 9. FORECH w jk, (j, k) (U I U H ) (U H U O ) DO 10. w jk = η δ k x j k 11. w jk = w jk + w jk 12. ENDDO 13. ENDDO 14. UNTIL(convergence(D, y O (D)) OR t > t max ) 15. return(w) ML:VI-68 Neural Networks STEIN

14 Weight daptation: Incremental Gradient Descent [network] lgorithm: MPT Multilayer Perceptron Training Input: D Training examples of the form (x, c(x)) with x = p + 1, c(x) {0, 1} k. η Learning rate, a small positive constant. Output: w Weights for the units in U I, U H, U O. 1. initialize_random_weights(u I, U H, U O ), t = 0 2. REPET 3. t = t FORECH (x, c(x)) D DO 5. FORECH u U H DO y u = propagate(x, u) // layer 1 6. FORECH v U O DO y v = propagate(y H, v) // layer 2 7. FORECH v U O DO δ v = y v (1 y v ) (c v (x) y v ) // backpropagate layer 2 8. FORECH u U H DO δ u = y u (1 y u ) v U o w uv δ v // backpropagate layer 1 9. FORECH w jk, (j, k) (U I U H ) (U H U O ) DO 10. w jk = η δ k x j k 11. w jk = w jk + w jk 12. ENDDO 13. ENDDO 14. UNTIL(convergence(D, y O (D)) OR t > t max ) 15. return(w) ML:VI-69 Neural Networks STEIN

15 Weight daptation: Incremental Gradient Descent [network] lgorithm: MPT Multilayer Perceptron Training Input: D Training examples of the form (x, c(x)) with x = p + 1, c(x) {0, 1} k. η Learning rate, a small positive constant. Output: w Weights for the units in U I, U H, U O. 1. initialize_random_weights(u I, U H, U O ), t = 0 2. REPET 3. t = t FORECH (x, c(x)) D DO 5. FORECH u U H DO y u = propagate(x, u) // layer 1 6. FORECH v U O DO y v = propagate(y H, v) // layer 2 7. FORECH v U O DO δ v = y v (1 y v ) (c v (x) y v ) // backpropagate layer 2 8. FORECH u U H DO δ u = y u (1 y u ) v U o w uv δ v // backpropagate layer 1 9. FORECH w jk, (j, k) (U I U H ) (U H U O ) DO 10. w jk = η δ k x j k 11. w jk = w jk + w jk 12. ENDDO 13. ENDDO 14. UNTIL(convergence(D, y O (D)) OR t > t max ) 15. return(w) ML:VI-70 Neural Networks STEIN

16 Weight daptation: Momentum Term Momentum idea: a weight adaptation in iteration t considers the adaptation in iteration t 1: w uv (t) = η δ v x u v + α w uv (t 1) The term α, 0 α < 1, is called momentum. ML:VI-71 Neural Networks STEIN

17 Weight daptation: Momentum Term Momentum idea: a weight adaptation in iteration t considers the adaptation in iteration t 1: w uv (t) = η δ v x u v + α w uv (t 1) The term α, 0 α < 1, is called momentum. Effects: due the adaptation inertia local minima can be overcome if the direction of the descent does not change, the adaptation increment and, as a consequence, the speed of convergence is increased. ML:VI-72 Neural Networks STEIN

18 Neural Networks dditional Sources on the Web pplication and implementation: JNNS. Java Neural Network Simulator. SNNS. Stuttgart Neural Network Simulator. ML:VI-73 Neural Networks STEIN