Defining Feedforward Network Architecture. net = newff([pn],[s1 S2... SN],{TF1 TF2... TFN},BTF,LF,PF);

Transcription

1 Appendix D MATLAB Programs for Neural Systems D.. Defining Feedforward Network Architecture Feedforward networks often have one or more hidden layers of sigmoid neurons followed by an output layer of linear neurons. Multiple layers of neurons with nonlinear transfer functions allow the network to learn nonlinear and linear relationships between input and output vectors. The function newff() creates a feedforward backpropagation network architecture with desired number of layers and neurons. The general form of use of the function is given below, which returns an N-layer feedforward backpropagation network object: net = newff([pn],[s S2... SN],{TF TF2... TFN},BTF,LF,PF); where the first input PN is an N 2 matrix of minimum and maximum values for N input elements. S S2... SN are the sizes (number of neurons) of the layers of the network architecture. TFi is the transfer function of the ith layer; the default is tansig. The transfer functions TFi can be any differentiable transfer function such as tansig, logsig or purelin. BTF is the backpropagation network training function; the default is trainlm. Different training functions with their features are described in Section of Chapter 4. LF is the backpropagation weight/bias learning function with gradient descent, such as learngd, learngdm. The default is learngdm. The function learngdm is used to calculate the weight change dw for a given neuron from the neuron s input P and error E. Learning occurs according to learngdm s learning parameters such as the weight (or bias) W, learning rate and momentum constant, according to gradient descent with momentum and returns weight change and new learning states. PF is the performance function such as mse (mean squared error), mae (mean absolute error) and msereg (mean squared error with regularization). The default is mse. For example: net=newff([- 2; 5],[3,],{'tansig','purelin'},'traingd', 'learngdm', 'mae'); The function creates a two-input single-output feedforward network with single hidden layer. The first input [ 2; 5] specifies the minimum and maximum values for each of the input vectors. The second input is an array containing the sizes of each layer, i.e., the network Computational Intelligence: Synergies of Fuzzy Logic, Neural Networks and Evolutionary Computing, First Edition. Nazmul Siddique and Hojjat Adeli. 23 John Wiley & Sons, Ltd. Published 23 by John Wiley & Sons, Ltd.

2 462 Computational Intelligence has 3 neurons in the hidden layer and neuron in the output layer. The third input is a cell array containing the names of the transfer functions to be used in each layer, i.e., tansig for hidden layer and purelin (linear) activation function for output layer. There are other different activation functions with distinct features, such as logsig, hardlim. The final input contains the name of the training function to be used. traingd is one of the training functions used by the network. newff() will also automatically initialize the weights and biases of the network. D... Creating RBF Network Architecture In an RBF network, there can be a maximum of M inputs and a maximum of N radial basis neurons in the hidden layer. There are no weights between inputs and hidden neurons. Each radial basis neuron is connected to the output neuron through the weight matrix W, which has to be learned. Using the MATLAB R functions, the architecture of an RBF network with m =, 2, 3,...,M input elements and n =, 2, 3,...,N radial basis neurons (in the hidden layer) can be created. All details of designing a radial basis function network are built into the design functions newrbe() and newrb(), and their outputs can be obtained with sim(). The functions are called in the following way: net = newrbe(p, T, Spread); The function newrbe() takes matrices of input vectors P and target vectors T, and a spread for the radial basis layer, and returns a network with weights and biases such that the outputs are exactly T when the inputs are P. The value for the spread constant should be larger than the distance between adjacent input vectors, so as to get a good generalization, but smaller than the distance across the whole input space. The function newrbe() creates as many radial basis neurons as there are input vectors in P. The drawback to newrbe() is that it produces a network with as many hidden neurons as there are input vectors. For this reason, newrbe() does not return an acceptable solution when many input vectors are needed to properly define a network, as is typically the case (Demuth and Beale, 2). newrb() is a more efficient design function, which creates a radial basis network one neuron at a time. Neurons are added to the network until the sum-squared error falls beneath an error goal or a maximum number of neurons have been reached. The call for this function is net = newrb(p, T, Goal, Spread); The function newrb() takes matrices of input vectors P and target vectors T, and design parameters goal and spread for radial basis layer, and returns the desired network with weights and biases such that the outputs are exactly T when the inputs are P. The design method of newrb() is similar to that of newrbe(). The difference is that newrb() creates neurons one at a time. The error of the new network is checked, and if low enough newrb() is finished. Otherwise the next neuron is added. This procedure is repeated until the error goal is met or the maximum number of neurons is reached. Thus, newrbe() creates a network with zero error on training vectors. The only condition required is to make sure that SPREAD is large enough so that the active input regions of the radbas neurons overlap enough so that several radbas neurons always have fairly large outputs at any given moment. This makes the network function smoother and results in better generalization for new input vectors occurring between input vectors used in the design. (However, SPREAD should not be so large that each neuron is effectively responding in the same large area of the input space.)

3 Appendix D: MATLAB Programs for Neural Systems 463 RBF networks, even when designed effectively with newrbe(), tend to have many times more neurons than a comparable MLP network with tansig or logsig neurons in the hidden layer. This is because sigmoid neurons can have outputs over a large region of the input space, while RBF neurons only respond to relatively small regions of the input space. The result is that the larger the input space (in terms of number of inputs, and the ranges those inputs vary over) the more RBF neurons are required. On the other hand, designing an RBF network often takes much less time than training a sigmoid/linear network, and can sometimes result in fewer neurons being used. D...2 Creating GRNN Network Architecture A generalized regression neural network is often used for function approximation. A GRNN network with m =, 2, 3,..., M input elements and n =, 2, 3,..., N radial basis neurons (in the hidden layer) can be created using the function newgrnn() where the first layer is just like that of newrbe() or newrb() but has a slightly different second layer. It has as many neurons as there are input/target vectors in P. Specifically, the first-layer weights are set to P. The bias b is set to a column vector of.8326/spread. The user chooses Spread, the distance an input vector must be from a neuron s weight vector to be.5. Each neuron s weighted input is the distance between the input vector and its weight vector. The second layer also has as many neurons as input/target vectors, but there the weights are set to target T. The function is called in the following way: net = newgrnn(p, T, Spread); The function newgrnn() takes matrices of input vectors P and target vectors T, and a spread for the radial basis layer, and returns a network with weights and biases such that the outputs are exactly T when the inputs are P. The value for spread constant should be larger than the distance between adjacent input vectors, so as to get good generalization, but smaller than the distance across the whole input space. To fit data closely, a smaller spread is suggested, i.e., smaller than the typical distance between input vectors. To fit the data more smoothly, a larger spread is to be chosen. A larger spread leads to a large area around the input vector where layer neurons will respond with significant outputs. Therefore, if the spread is small the radial basis function is very steep, so that the neuron with weight vector closest to the input will have a much larger output than other neurons. The network tends to respond with the target vector associated with the nearest design input vector. As the spread becomes larger the radial basis function s slope becomes smoother and several neurons can respond to an input vector. The network then acts as if it is taking a weighted average between target vectors whose design input vectors are closest to the new input vector. As the spread becomes larger, more and more neurons contribute to the average, with the result that the network function becomes smoother. D...3 Creating PRNN Network Architecture A PNN network can be created by calling the function in the following way: net = newpnn(p, T, Spread); The function newpnn() takes matrices of input vectors P and target vectors T, and a spread for the radial basis layer, and returns a network with weights and biases such that the outputs

4 464 Computational Intelligence are exactly T when the inputs are P. If the spread is near zero, the network will act as a nearest-neighbour classifier. As the spread becomes larger, the designed network will take into account several nearby design vectors. Although the PNN was derived from the same mathematical merits and similarities to those of RBF and GRNN networks, after defining the architecture it is found to be more appropriate for classification problems rather than prediction or approximation problems. Probabilistic neural networks can be used for classification problems. When an input is presented, the first layer computes distances from the input vector to the training input vectors and produces a vector whose elements indicate how close the input is to a training input. The second layer sums these contributions for each class of inputs to produce as its net output a vector of probabilities. Finally, a complete transfer function on the output of the second layer picks the maximum of these probabilities, and produces a for that class and a for the other classes. D..2 Training Networks Different backpropagation training algorithms are available as functions in MATLAB R. They have their own features and advantages. Some of the most widely used functions are discussed briefly. traingd basic gradient descent learning algorithm. It has slow response but can be used in incremental mode training. traingdm gradient descent with momentum. It is generally faster than traingd and can be used in incremental mode training. traingdx adaptive learning rate. It has faster training time than traingd but can only be used in batch mode training. trainrp resilient backpropagation. It is a simple batch mode training algorithm with fast convergence and minimal storage requirements. trainlm Levenberg Marquart algorithm. It is a faster training algorithm for networks of moderate size. It has a memory reduction feature for use when the training set is large. There are several parameters associated with training algorithms. The parameters are learning rate, error goal, epochs and show. These parameters are defined as: net.trainparam.lr - specifies learning rate net.trainparam.goal - specifies error goal net.trainparam.epochs - specifies the number of iterations net.trainparam.show - displays status for every show. Once the network has been defined and the parameters are set, the network can be trained using the function train() as [net, tr] = train(net, P, T) where net is the network object, tr contains information about the progress of training, P and T are the input and output vectors, respectively. Typically, one epoch of training is defined as a

5 Appendix D: MATLAB Programs for Neural Systems 465 single presentation of all input vectors to the network. The network is then updated according to the results of all those presentations. Training occurs until a maximum number of epochs occur, the performance goal is met or any other stopping condition of the function is met. For example: net.trainparam.lr =.5; net.trainparam.goal =.; net.trainparam.epochs =; net.trainparam.show = 25; [net,tr]=train(net, P, T) The network will be trained using the input and output data P and T, respectively for up to epochs or when an error goal of. is reached. D..3 Simulating Networks The function sim() simulates a network. It takes the network input P and the network objects net and returns the network output ŷ. A single matrix of concurrent vector is presented to the network and the network produces a single matrix of concurrent vector as output: ŷ= sim(net, P) D..4 Creating Neural Network Subsystem Once the network has been trained and tested with training and checking data, a Simulink R model can be created using the MATLAB R function gensim(). The function gensim() generates block descriptions of networks so that it can simulate the neural network in Simulink R.The function is called in the following way: gensim(net, st) gensim() takes these inputs net ˆ= neural network defined either in an M-file or NN Toolbox and st ˆ= sample time and creates a Simulink R system containing a block which simulates a neural network with a specified sampling time: gensim(net,st) The second argument to gensim() determines the sample time, which is normally chosen to be some positive real value. If the network has no delays associated with its input weights or layer weights, this value is set to. For example: gensim(net,-) The value of the parameter st is, i.e., tells gensim to generate a network with continuous sampling. Example D..: Define a feedforward network, train and simulate with input data In this example, a two-layer feedforward network is created. The network s input ranges from

6 466 Computational Intelligence to. The first layer has five neurons with tansig function; the second layer has one neuron with linear function. The training function traingd is used to train the network. %Chapter 4 Example D.. %Create a feedforward NN and train with data set [P T] %Training set P = [ ]; T = [ ]; net = newff([ ],[5 ],{'tansig' 'purelin'},'traingd'); %Set network parameters as follows net.trainparam.lr =. %Learning rate net.trainparam.goal =. %Performance goal net.trainparam.epochs = 5 %This sets maximum number of epochs in a training net.trainparam.show = 25 %This displays training status after each 25 epoch net.trainparam.time = inf %Maximum time to train in seconds net.trainparam.min_grad=e- %Minimum performance gradient %Here the network is simulated and its output plotted against the targets. net = train(net,p,t); Y = sim(net,p); plot(p,t, '-k', P,Y, 'ok') Example D..2: Updating weights of a two-layer neural net A two-layer neural network with two inputs x = [x, x 2 ] and one output y is given by y = W2 T [ f ( [ ] [ ] W T x + b ) ] + b2, where W T =, b =, W T = [ ] and b 2 = [ 2.8]. Update the weights and biases of the network, simulate the network and plot the output surface over the grid [ 2, 2] [ 2, 2]: %Chapter 4 Example D..2 %A two layer NN is given by y=w2'[f(w'x+b)]+b2 %with W=[ ; ]; b=[-2.29; 3.67]; W2=[ ]; b2=[-2.8]; %Update NN with W, W2, b and b2 %Plot the NN output surface y as a function of x over grid [-2,2] x[-2,2] %Weights and bias W=[ ; ]; b=[-2.29; 3.67];

7 Appendix D: MATLAB Programs for Neural Systems 467 W2=[ ]; b2=[-2.8]; %Output surface [x, x2]=meshgrid(-2:.:2); %Compute NN output p=x(:); p2=x2(:); p=[p';p2']; %NN weights and bias %nnt2ff () updates NN with specified weights and biases net=nnt2ff(minmax(p),{w,w2},{b,b2},{'tansig', 'purelin'}); %Simulate a=sim(net,p); %Arrange results for mesh plot a=eye(4); a(:)=a'; mesh(x,x2,a); AZ=6, EL=3; view(az,el); xlabel('x'); ylabel('x2'); title('nn output surface for tansigmoid function') See Figure D.. for the result. Example D..3: Approximation of output surface In this example, an output surface of a nonlinear function is approximated. The nonlinear function is defined by f (x, y) = NN output surface for tansigmoid function 5 5 x 2 x2 2 Figure D.. Output surface of a two-layer network

8 468 Computational Intelligence sin (π x) cos (πy) with x [ 2, 2] and y [ 2, 2]. An MLP with 2 tansigmoidal neurons and linear neuron can approximate the function after training the network for 5 epochs: %Chapter 4 - Example D..3 %NN Function approximation [x, y]=meshgrid(-2:.:2); %Nonlinear function defined by z=sin(pi*x).*cos(pi*y); %Generate Input & Target data for i=:2 p(:,i)=4*(rand(2,)-.5); T(:,i)=sin(pi*p(2*i-))*cos(pi*p(2*i)); end % %Two-layer NN created with 2 tansig %and one purelin neuron net=newff(minmax(p), [2,], {'tansig', 'purelin'}, 'trainlm'); net.trainparam.show=5; net.trainparam.epochs=5; net.trainparam.goal=e-6; [net,tr]=train(net,p,t) %Simulate the net a=zeros(4,4); [x, y]=meshgrid(-2:.:2); for i=:68 a(i)=sim(net,[x(i);y(i)]); end figure() %Original nonlinear function subplot(,2,) mesh(x,y,z); title('original function graphics'); xlabel('<--x-->') ylabel('<--y-->') zlabel('<--z-->') AZ=5; EL=59; view(az,el) subplot(,2,2) mesh(x,y,a); title('nn approximated graphics') xlabel('<--x-->') ylabel('<--y-->') zlabel('<--z-->') AZ=5; EL=59; view(az,el)

9 Appendix D: MATLAB Programs for Neural Systems 469 Original function graphics NN approximated graphics <--z--> <--z--> 2 <--y--> 2 2 <--x--> <--x--> 2 <--y--> 2 (a) (b) Figure D..2 function Approximation of a nonlinear function. (a) Nonlinear function; (b) NN approximated See Figure D..2. Example D..4: Approximation of a nonlinear function In this example, a nonlinear function defined by the input/output data is approximated using a two-layer feedforward network. The network s weights and biases are shown after training. The plot will also show how good the approximation is: %Chapter 4 Example D..4 %Function approximation clear all; close all; %Training data:examplar input pattern and target output vector x=-:.:; y=[-.96,-.577,-.73,.377,.64,.66,.46,.34, , -.434, -.5, -.393, -.65,.99,.37,.396, ,.82, -.3, -.29, -.32]; %Define a NN and initialise weights net=newff(minmax(x), [7 ], {'tansig', 'purelin'},'trainlm'); %Output of NN with initial weights ycapl=sim(net,x); %Train the NN net.trainparam.epochs = 5 net.trainparam.goal =. net.trainparam.lr =. %net.trainparam.min_grad=e- %Maximum number of epochs to train %Performance goal %Learning rate %Minimum performance gradient

10 47 Computational Intelligence net.trainparam.show = net.trainparam.time = inf [net,tr]=train(net,x,y); %Epochs between displays %Maximum time to train in seconds %Output of NN figure() %Generalisation: input vector is different %from the one used for training x2=-:.:; ycap2=sim(net,x2); plot(x,ycap,x2,ycap2,'-',x,y,'o'); plot(x,ycapl,'--k', x2,ycap2,'-.k',x,y,'ok'); title('function approximation'); xlabel('x-values'); ylabel('y-values'); legend('before Training', 'After Training','Function'); %Show weights and biases of NN w=net.iw{,} bw=net.b{} v=net.lw{2,} bv=net.b{2} See Figure D Function approximation Before Training After Training Function.2 y-values x-values Figure D..3 Function approximation

11 Appendix D: MATLAB Programs for Neural Systems 47 T double Target P Input double double p{} y{} Neural Network double (2) y{} Figure D..4 NN block description for Simulink R Example D..5: Creating an NN subsystem for simulation In this example, a neural network block description is created for simulation to be used under Simulink R.Todothis, an NN is trained with a set of input/output data and simulated. Once this is done, an NN block description is created using the gensim() function: %Chapter 4 Example D..5 %Training data set p=[-:.5:]; %noisy sine wave t=sin(2*pi*p)+.*randn(size(p)); net=newff([- ],[2,],{'tansig','purelin'},'traingdx'); net.trainparam.show=5; net.trainparam.epochs=3; pt=p*.979; y=sim(net,pt) plot(p,t,'-',p,y,'o') %generate NN block description gensim(net,-) The generated NN Simulink R block description is shown in Figure D..4. The Neural Network Toolbox provides three popular neural network Simulink R blocks for prediction and control that have been applied to many applications: Model Predictive Control, NARMA-L2 (or Feedback Linearization) Control and Model Reference Control. These Simulink R control blocks are discussed in detail in Chapter 5.