Multi-Layer Perceptron in MATLAB NN Toolbox

Transcription

1 Multi-Layer Perceptron in MATLAB NN Toolbox [Part 1] Yousof Koohmaskan, Behzad Bahrami, Seyyed Mahdi Akrami, Mahyar AbdeEtedal Department of Electrical Engineering Amirkabir University of Technology (Tehran Polytechnic) Advisor: Dr. F. Abdollahi Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

2 Introduction Rosenblatt in 1961 created many variations of the perceptron One of the simplest was a single-layer network whose weights Neuron and biases Model could be trained to produce a correct target vector when presented with the corresponding input vector Neuron Mod A perceptron neuron, which uses the hard-limit transfer function hardlim, is shown below. Input Perceptron Neuron Where p p 1 2 p3 p R w 1,1 n f b w 1, R a R = number of elements in input vector 1 a = hardlim (Wp + b) Each external input is weighted with an appropriate weight w 1j, and the sum Figure: of the weighted Perceptron inputs is Representation-[1] to the hard-limit transfer function, which also has an input of 1 transmitted to it through the bias. The hard-limit transfer function, which returns a 0 or a 1, is shown below. a Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

3 Input Vector Input function f to generate their output. Perceptron Preliminary Training Where Network Use Functions Solve Problem p p 1 w 1,1 Input General Neuron 2 Introduction p3 n a R = number of Where... pelements in f p 1 w input vector 1,1 2 R = number of Where p w 1, R b p3 n a p R f p 1 elements w 1,1 in 2 1 p w b p3 input vectorn a R = num 1, R R f elemen input ve a = hardlim (Wp + b) 1 p w b 1, R R a = purelin (Wp + b) 1 Each external input is weighted with an appropriate weight w There are many transfer function that can 1j, and the sum of the weighted inputs is sent to the This hard-limit network transfer has the function, same basic be which structure useda = also as inf(wp the the +b) perceptron. The only has an input of 1 transmitted to it through the bias. The hard-limit transfer perceptron function, which structure, returns a 0 or e.g. difference is that the linear neuron Multilayer uses a networks linear transfer often use function the log-sigm pure a 1, is shown below. a a a n 0-1 a = hardlim(n) Hard-Limit Transfer Function n 0-1 a = purelin(n) Linear Transfer Function The perceptron neuron produces a 1 if the net input into the transfer The function logsig generates outputs betw The linear transfer function calculates is equal to or greater than 0; otherwise it produces a 0. input goes the from neuron s negative output to positive by simply infinity retu the value passed to it. The hard-limit transfer function gives a perceptron the ability to classify input vectors by dividing the input space into a two = purelin regions. ( n Specifically, ) = purelin( outputs Wp + b) will = Wp + b be 0 if the net input n is less than 0, This or 1 neuron if the net can input be trained n is 0 to or learn greater. an affine The function of its inputs, or to following figure show the input space linear of a two-input approximation hard to limit a nonlinear neuron with function. the A linear network cannot, weights w 11, = 1, w 12, = 1 and course, a bias 5-8 bbe = made 1. to perform a nonlinear computation. Figure: hardlim, purelin, logsig Usually hardlim is selected as default 0 n -1 a = logsig(n) Log-Sigmoid Transfer Function Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

4 Perceptron Architecture The perceptron network consists of a single layer R input vector S output scaler Here, we consider just one-layer perceptron Perceptron Architecture The perceptron network consists of a sing connected to R inputs through a set of we forms. As before, the network indices i and the connection from the jth input to the it Input Perceptron Layer Inp p 1 p 2 p 3 p R w 1,1 w SR, S 1 S 1 S b 1 b 2 n 1 n 2 n S a 1 a 2 a S R W 1 b S a = hardlim(wp + b) R S Figure: Perceptron The perceptron learning rule described sh single layer. Thus only one-layer networks Architecture places limitations on the computation a p problems that perceptrons are capable of s and Cautions on page Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

5 Layer Notation Perceptron Preliminary Training Network Use Functions Solve Problem A single superscript is used to identify elements of a layer. For instance, the net input of layer 3 would be shown as n 3. Superscripts k, l are used to identify the source (l) connection and the Mathematical Notation destination (k) connection of layer weight matrices and input weight matrices. For instance, the layer weight matrix from layer 2 to layer 4 would be shown as LW 4,2. A single superscript Input weight ismatrix usediwto identify elements of a layer, e.g. k, l n 3 expresses input Layer weight to layer matrix LW 3 k, l Input weight matrix from layer k to layer l can be shown as IW l,k Figure and Equation Examples, The following figure, taken from Chapter also for layer weight matrix, i.e.lw l,k 12, Advanced Topics, illustrates notation used in such advanced figures. Inputs Layers 1 and 2 Layer 3 Outputs p1(k) 2 x 1 n1(k) 4 x 2 4 x 1 1 b1 2 4 x TDL 0,1 a1(k) = tansig (IW1,1p1(k) +b1) IW2,1 3 x (2*2) p2(k) TDL IW2,2 5 x x (1*5) n2(k) 3 x 1 3 a2(k) = logsig (IW2,1 [p1(k);p1(k-1) ]+ IW2,2p2(k-1)) a1(k) 4 x 1 a2(k) 3 x 1 TDL 1 LW3,3 1 x (1*1) LW3,1 1 x 4 1 b3 1 x 1 LW3,2 1 x 3 n3(k) a3(k) 1 x 1 1 x 1 a3(k)=purelin(lw3,3a3(k-1)+lw3,1 a1 (k)+b3+lw3,2a2 (k)) Figure: Example of a perceptron Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21 1 y2(k) 1 x 1 y1(k) 3 x 1

6 Matlab Notation Considerations superscripts cell array indices, e.g. p 1 p{1} subscripts indices within parentheses, e.g. p 2 p(2) superscripts + subscripts,e.g. p 1 2 p{1}(2) indices within parentheses a second cell array index 1 e.g. p 1 (k 1) p{1, k 1} 1 also in the Matlab help, it should be corrected; p{1, k 1] p{1, k 1} Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

7 Learning Rules It breaks into two parts, supervised and unsupervised learning, we consider the first topic The objective is to reduce the error e, which is the difference t-a between the neuron response a and the target vector t There are many algorithms that do this and we will introduce them All the algorithms compute w utilizing some methods Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

8 Learning Alghorithms Hebb weight learning rule 2 w k = ca k p k MATLAB command learnh Perceptron weight and bias learning function w k = e k (p k ) T MATLAB command learnp Normalized perceptron weight and bias learning function pn = p 1 + p 2, w k = e k (pn k ) T MATLAB command learnpn 2 in all, e: error, p: input and a: output Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

9 Learning Alghorithms Widrow-Hoff weight/bias learning function learning rate c, MATLAB command learnwh w k = ce k (pn k ) T Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

10 Training Training is using the learning rule to update the weights of network repeatedly MATLAB command train w k new = w k old + w k Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

11 Utility Functions MATLAB command init, Initialize neural network MATLAB command sim, Simulate neural networks MATLAB command adapt Allow neural network to change weights and biases on inputs MATLAB command newp, create perceptron Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

12 Perceptron problem, step by step Creating a Perceptron, >> net=newp(p,t); Assigning initial weight and bias, >> net.iw{1,1}= [ w1 w2 ] ; >> net.b{1} = [b]; Determining number of pass, >> net.trainparam.epochs = pass; Training Network, >> net = train(net,p,t); Test and Verifying, >> a = sim(net,p); Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

13 Perceptron problem, step by step Example 1: Classifying displayed data Vectors to be Classified 10 5 P(2) P(1) Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

14 Perceptron problem, step by step Example 1; Solution % number of samples of each class N = 20; % define inputs and outputs offset = 5; % offset for second class x = [randn(2,n) randn(2,n)+offset]; % inputs y = [zeros(1,n) ones(1,n)]; % outputs % Plot input samples with PLOTPV (Plot perceptron input/target vectors) plotpv(x,y); net=newp(x,y); net = train(net,x,y); % train network figure(1) plotpc(net.iw{1},net.b{1}); Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

15 Perceptron problem, step by step Example 1; Solution Vectors to be Classified P(2) P(1) Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

16 Note net.trainparam.epochs net.trainparam.goal Random Weights Initialization net.inputweights{1,1}.initfcn = rands ; net.biases{1}.initfcn = rands ; net = init(net); Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

17 Perceptron problem Example 2: Classifying displayed data Class A Class B Class D Class C Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

18 Perceptron problem Example 2; Solution K = 30; % define classes q =.6; % offset of classes A = [rand(1,k)-q; rand(1,k)+q]; B = [rand(1,k)+q; rand(1,k)+q]; C = [rand(1,k)+q; rand(1,k)-q]; D = [rand(1,k)-q; rand(1,k)-q]; % plot classes plot(a(1,:),a(2,:), bs ) hold on; grid on plot(b(1,:),b(2,:), r+ ); plot(c(1,:),c(2,:), go ); plot(d(1,:),d(2,:), m* ); % text labels for classes text(.5-q,.5+2*q, Class A ); text(.5+q,.5+2*q, Class B ) Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

19 Perceptron problem Example 2; Solution-Cont d text(.5+q,.5-2*q, Class C ); text(.5-q,.5-2*q, Class D ) a = [0 1] ; b = [1 1] ; c = [1 0] ; d = [0 0] ; P = [A B C D]; % define targets T = [repmat(a,1,length(a)) repmat(b,1,length(b))... repmat(c,1,length(c)) repmat(d,1,length(d)) ]; net=newp(p,t); E = 1; net.adaptparam.passes = 1; linehandle = plotpc(net.iw{1},net.b{1}); n = 0; while (sse(e) & n<1000) n = n+1; [net,y,e] = adapt(net,p,t); linehandle = plotpc(net.iw{1},net.b{1},linehandle); drawnow; end Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

20 Perceptron problem Example 2; Solution- Cont d Class A Class B Class D Class C Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

21 N N N N N N N N N N Thank You Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21

22 Matlab 7.9, Neural Network Toolbox Help MathWorks Inc. R2009b Click: LASIN - Laboratory of Synergetics Website, Slovenia Dr. F. Abdollahi, Lecture Notes on Neural Networks Winter 2011 Koohmaskan, Bahrami, Akrami, AbdeEtedal (AUT) Multi-Layer Perceptron - part 1 February / 21