Chapter 4 Supervised learning:

Transcription

1 Chapter 4 Spervised learning: Mltilayer Networks II

2 Madaline Other Feedforward Networks Mltiple adalines of a sort as hidden nodes Weight change follows minimm distrbance principle Adaptive mlti-layer layer networks Dynamically change the network size # of hidden nodes Prediction networks BP nets for prediction Recrrent nets Networks of radial basis fnction RBF e.g., Gassian fnction Perform better than sigmoid fnction e.g., interpolation in fnction approximation Some other selected types of layered NN

3 Architectres Madaline Hidden layers of adaline nodes Otpt nodes differ Learning Error driven, bt not by gradient descent Minimm distrbance: smaller change of weights is preferred, provided it can redce the error Three Madaline models Different node fnctions Different learning rles MR I, II, and III MR I and II developed in 60 s, MR III mch later 88

4 Madaline MRI net: Otpt nodes with logic fnction MRII net: Otpt nodes are adalines MRIII net: Same as MRII, except the nodes with sigmoid fnction

5 MR II rle Madaline Only change weights associated with nodes which have small net j Bottom p, layer by layer Otline of algorithm 1. At layer h: sort all nodes in order of increasing net vales, pt those with net <θ, in S. For each A j in S if reversing its otpt change x j to -x j improves the otpt error, then change the weight vector leading into A j by LMS of Adaline or other ways w x net i ji, j j ji,

6 MR III rle Madaline Even thogh node fnction is sigmoid, do not se gradient descent do not assme its derivative is known Use trial adaptation E: total sqare error at otpt nodes E k : total sqare error at otpt nodes if net k at node k is increased by ε > 0 Change weight leading to node k according to w i Ek E / or w ie E E / Update weight to one node at a time It can be shown to be eqivalent to BP Since it is not explicitly dependent on derivatives, this method can be sed for hardware devices that inaccrately ate implement e sigmoid fnction k

7 Adaptive Mltilayer Networks Smaller nets are often preferred Compting is faster Generalize better Training is faster Fewer weights to be trained Smaller # of training samples needed Heristics for optimal net size Prning: start with a large net, then prne it by removing nimportant nodes and associated connections/weights Growing: start with a very small net, then continosly increase its size with small increments ntil the performance becomes satisfactory Combining the above two: a cycle of prning and growing g ntil performance is satisfied and no more prning is possible

8 Adaptive Mltilayer Networks Prning a network by removing Weights with small magnitde e.g., 0 Nodes with small incoming weights Weights whose existence does not significantly affect network otpt If o / w is negligible By examining the second derivative E 1 '' E w E w where E'' E w w w when E approaches a local minimm, E / w 0, then 1 E E'' w effect of removing w is to change it to 0, ie i.e., w w 1 whether to remove w depends on if E E'' w is sfficiently small E Inpt nodes can also be prned if the reslting change of is negligible

9 Adaptive Mltilayer Networks Cascade correlation example of growing net size Cascade architectre development Start with a net withot hidden nodes Each time one hidden node is added between the otpt nodes and all other nodes The new node is connected to otpt nodes, and from all other nodes inpt and all existing hidden nodes Not strictly feedforward

10 Adaptive Mltilayer Networks Correlation learning: when a new node n is added first train all inpt weights to node n from all nodes below maximize covariance with crrent error of otpt nodes E then train all weight to otpt t nodes minimize E qickprop is sed all other weights to lower hidden nodes are not changes so it trains fast

11 Adaptive Mltilayer Networks Train w new to maximize covariance covariance between x and E old S w x x E E new new new k, k K k 1 P p1 x new, p x new E k, p E k, where is the otpt of for th, p x p sample, the mean vale of x over all samples the error on th otpt node for th p k p sample with old weights, its mean vale over all samples when Sw new i is maximized, i variance of x p from x mirrors that tof error E k, p from Ek, Sw new ismaximized by gradient ascent S K P ' wi Sk Ek, p Ek f pii, p, where wi k 1 p1 ' Sk is the sign of correlation between xnewand Ek, f p is the th th derivative of x's node fnction, and I the i inpt of p sample i, p x new w new

12 Adaptive Mltilayer Networks Example: corner isolation problem Hidden nodes are with sigmoid fnction [-0.5, 0.5] When trained withot t hidden node: 4 ot of 1 patterns are misclassified After adding 1 hidden node, only patterns are misclassified After adding the second hidden node, all 1 patterns are correctly classified At least 4 hidden nodes are reqired with BP learning X X X X

13 Prediction Prediction Networks Predict ft based on vales of ft 1, ft, Two NN models: feedforward dand recrrent A simple example section Forecasting commodity price at month t based on its prices at previos months Using a BP net with a single hidden layer 1 otpt node: forecasted price for month t k inpt nodes sing price of previos k months for prediction k hidden nodes Training sample: for k = : {x t-, x t-1 x t } Raw data: flor prices for 100 consective months, 90 for training, 10 for cross validation testing one-lag forecasting: predict x t based on x t- and x t-1 1 mltilag: sing predicted vales for frther forecasting

14 Training: 90 inpt data vales Last 10 prices for validation test Three attempts: t k =, 4, 6 Learning rate = 0.3, momentm = ,000 50,000 epochs -- net with good prediction Twolarger nets over-trained d with larger prediction errors for validation data Prediction Networks Reslts Network MSE --1 Training one-lag mltilag Training one-lag mltilag Training one-lag mltilag

15 Prediction Networks Generic NN model for prediction Preprocessor prepares p training samples xt from time series data xt Train predictor sing samples xt e.g., by BP learning Preprocessor In the previos example, Let k = d + 1 sing previos d + 1 data points to predict inpt sample at time t: xt xt d,..., xt 1, xt the desired otpt e.g., prediction: xt 1 More general: c i is called a kernel fnction for different memory model how previos data are remembered Examples: exponential trace memory; gamma memory see p.141

16 Recrrent NN architectre Cycles in the net Prediction Networks Otpt nodes with connections to hidden/inpt nodes Connections between nodes at the same layer Nd Node may connect tto itself Each node receives external inpt as well as inpt from other nodes Each node may be affected by otpt of every other node With a given external inpt vector, the net often converges to an eqilibrim state after a nmber of iterations otpt t of every er node stops to change An alternative NN model for fnction approximation Fewer nodes, more flexible/complicated connections Learning procedre is often more complicated

17 Prediction Networks Approach I: nfolding to a feedforward net Each layer represents a time delay of the network evoltion Weights in different layers are identical A flly connected net of 3 nodes Cannot directly apply BP learning becase weights in different layers are constrained to be identical How many layers to nfold to? Hard to determine Eqivalent FF net of k layers

18 Approach II: gradient descent A more general approach Prediction Networks Error driven: for a given external inpt t d t o t e t E k k k k k where k are otpt nodes desired otpt are known Weight pdate wi, j t 1 wi, j t wi, j t E t ok t wi, j t k dk t ok t wi, j wi, j t o t1 z t k l f ' net t[ ] where k w t z t l k, l i, k l w t1 w t i, j i, j 1 if i k, 0 otherwise and z t is inpt to node k from either ik, l inpt nodes or other nodes o ok 0 0 w 0 i, j

19 NN of Radial Basis Fnctions Motivations: better performance than sigmoid fnctions For some classification problems Fnction interpolation Definition A fnction is radial symmetric or is RBF if its otpt depends on the distance between the inpt vector and a stored vector related to that fnction Distance i where i is the inpt vector, is the vector associated with the RBF Otpt 1 whenever 1 NN with RBF node fnction are called RBF-nets

20 NN of Radial Basis Fnctions Gassian fnction is the most widely sed RBF abellshaped fnction centered at =0 / c e a bell-shaped fnction centered at = 0. Continos and differentiable g e ' / then if / ' / c e e c c Other RBF In erse q adratic f nction h persh]pheric f nction etc / then if / / c c e e g c g c g Inverse qadratic fnction, hypersh]pheric fnction, etc Gassian fnction μ Inverse qadratic μ hyperspheric fnction μ Gassian fnction fnction 0, β for c hyperspheric fnction c c s if 0 if 1

21 e c / Consider Gassian fnction again gives the center of the region for activating this nit gives the max otpt c determines the size of the region ex: for e / c g 0.9 c = 0.1 = c = 1.0 = c = 10. = 3.46 g Small c Large c

22 NN of Radial Basis Fnctions Pattern classification 4 or 5 sigmoid hidden nodes are reqired for a good classification Only 1 RBF node is reqired if the fnction can approximate the circle x x x x x x x x x x x

23 XOR problem --1 network ewo NN of Radial Basis Fnctions hidden nodes are RBF: xt ρ1 x e 1, t1 [1,1] xt ρ x e, t [0,0] Otpt node can be step or sigmoid When inpt x is applied Hidden node calclates distance x then its otpt All weights to hidden nodes set to 1 Weights to otpt node trained by LMS t 1 and t can also been trained xt j ρ x ρ x x x 1, , , , , 0 0, 1 1, 0 1 x 1, 1

24 NN of Radial Basis Fnctions Fnction interpolation Sppose yo know f x 1 and x x, to approximate x x by linear interpolation: 1 0 x f f xx f x0 f x1 f x f x1 x0 x1 / x x1 Let D x x, D x x be the distances of x0 from 1and then f x0 [ f x1 D1D f x f x1 D1]/ D1D [ f x1 D f x D1]/ D1D [ D f x D f x ]/[ D D ] x x i.e., sm of fnction vales, weighted and normalized dby distances Generalized to interpolating by more than known f vales D1 f x1 D f x D P f xp D1 D D P 0 0 f x where e P is the nm ber of neighbors to x 0 0 Only those x with small distance to x are sefl f i 0 0

25 Example: NN of Radial Basis Fnctions 8 samples with known fnction vales f x 0 can be interpolated sing only 4 nearest neighbors x, x, x, 3 4 x5 f x D f x D f x D f x D f x D D D D D 9D 3D 8D D D D D 3 4 5

26 NN of Radial Basis Fnctions Using RBF node to achieve neighborhood One hidden node per sample x p: = x p p,, and D D Network otpt for approximating f x is proportional to where d f x p p 1 weights w p = d p /P otpt node with net P p1 w p x x p hidden RBF nodes: Otpt x x p x

27 NN of Radial Basis Fnctions Clstering samples Too many hidden nodes when # of samples is large Groping similar samples having similar inpt and similar desired otpt together into N clsters, each with The center: vector i Mean desired otpt: Nt Network otpt: t i Sppose we know how to determine N and how to clster all P samples not a easy task itself, i and i can be determined by learning

28 NN of Radial Basis Fnctions Learning in RBF net Objective: learning to minimize i i Gradient descent approach seqential mode where fnction R is defined as R D D One can also obtain i by other clstering techniqes, then se GD learning for only i

29 D x x, o w x, E E d o n N P P p i j1 p, j i, j p i1 i p i p1 p p1 p p Learning wi E o p p d o d o x, w d o x p p p p p i i i p p p i w w i i Learning i, j E o x i d o w d o p p p p p i p p i, j i, j i, j x x x p i p i p i R x x p i p, j i, j i, j x p i i, j = ' where RD D w d o R' x x i, j i, j i p p p i p, j i, j For Gassian fnctions: D exp D /, RD exp D/, R' D 1/ exp D/ exp / i i p p p i w d o x w d o x exp x / i, j i, j i p p p, j i, j p i

30 NN of Radial Basis Fnctions A strategy for learning RBF net Start with a single RBF hidden node for a single clster containing only the first training sample. For each of the new training samples x If it is close to any of the existing clsters, do the gradient descent based pdates of the w and φ for all existing clsters/hidden nodes Otherwise, adding a new hidden node for a clster containing only x RBF networks are niversal approximators same representational power as BP networks

31 Polynomial networks Polynomial Networks Node fnctions allow direct compting of polynomials of inpts Approximating higher order fnctions with fewer nodes even withot hidden nodes Each node has more connection weights Higher-order networks n n kn # of weights per node: 1 1 k Can be trained by LMS General fnction approximator

32 Sigma-pi networks Polynomial Networks Does not allow terms with higher powers of inpts, so they are not a general fnction approximator # of weights per node: n n n 1 1k Can be trained by LMS Pi-sigma networks One hidden layer with Sigma fnction: Otpt nodes with Pi fnction: Prodct nits: Node comptes prodct: Integer power P j,i can be learned Often mix with other nits e.g., sigmoid