NEURAL NETWORKS. Neural networks

Size: px

Start display at page:

Download "NEURAL NETWORKS. Neural networks"

Iris Rose
6 years ago
Views:

1 NEURAL NETWORKS Neural netwrks Mtivatin Humans are able t prcess cmplex tasks efficiently (perceptin, pattern recgnitin, reasning, etc.) Ability t learn frm examples Adaptability and fault tlerance Engineering applicatins Nnlinear apprximatin and classificatin Learning (adaptatin) frm data: black-bx mdeling Very-Large-Scale Integratin (VLSI) implementatin 272

273 Neural netwrks Bilgical neural netwrks Neurn switching time: 0.

2 Bilgical neurn Sma: bdy f the neurn. Dendrites: receptrs (inputs) f the neurn. Axn: utput f neurn; cnnected t dendrites f ther neurns via synapses. Synapses: transfer f infrmatin between neurns (electrchemical signals). 273 Neural netwrks Bilgical neural netwrks Neurn switching time: 0.00 secnd Number f neurns: 0 (00 bilin) Cnnectins per neurn (synapses): 0 4 (00 trilin) Recgnitin time: 0-3 s (milisecnds) parallel cmputatin Artificial neural netwrks Weighted cnnectins amngst units Highly parallel, distributed prcess Emphasis n tuning weights autmatically 274 2

Use f neural netwrks Input is high-dimensinal Output is multidimensinal Mathematical frm f system is unknwn Interpretability f identified mdel is unimprtant Applicatins

Warren McCullch & Walter Pitts Definitin f a neurn: The activity f a neurn is an all r nne prcess The structure f the net des nt change with time T simple structure,

3 Use f neural netwrks Input is high-dimensinal Output is multidimensinal Mathematical frm f system is unknwn Interpretability f identified mdel is unimprtant Applicatins Pattern recgnitin Classificatin Predictin Mdeling Bilgical neural netwrk Sma Dendrite Axn Synapse Artificial neural netwrk Neurn Input Output Weight 275 ANN: histry 943 Warren McCullch & Walter Pitts Definitin f a neurn: The activity f a neurn is an all r nne prcess The structure f the net des nt change with time T simple structure, hwever: Prved that netwrks f their neurns culd represent any finite lgical expressin Used a massively parallel architecture Prvided imprtant fundatin fr further develpment 276 3

ANN: histry 948 Dnald Hebb Majr cntributins: Recgnized that infrmatin is stred in the weight f the synapses Pstulated a learning rate that is prprtinal t the

grup. 277 ANN: histry 957 Frank Rsenblatt Defined first cmputer implementatin: the perceptrn Attracted attentin f engineers and physicists, using mdel f bilgical

4 ANN: histry 948 Dnald Hebb Majr cntributins: Recgnized that infrmatin is stred in the weight f the synapses Pstulated a learning rate that is prprtinal t the prduct f neurn activatin values Pstulated a cell assembly thery: repeated simultaneus activatin f weakly-cnnected cell grup results in a mre strngly cnnected grup. 277 ANN: histry 957 Frank Rsenblatt Defined first cmputer implementatin: the perceptrn Attracted attentin f engineers and physicists, using mdel f bilgical visin Defined infrmatin strage as being in cnnectins r assciatins rather than tpgraphic representatins Defined bth self-rganizing and supervised learning mechanisms 278 4

ANN: histry 959 Bernard Widrw & Marcian Hff Engineers wh simulated netwrks n cmputers and

Frmulated Least Mean Squares (LMS) algrithm that minimizes sum-squared errr.

279 ANN: histry 977 David Rummelhart Intrduced cmputer implementatin f backprpagatin learning

5 ANN: histry 959 Bernard Widrw & Marcian Hff Engineers wh simulated netwrks n cmputers and implemented designs in hardware (Adaline and Madaline). Frmulated Least Mean Squares (LMS) algrithm that minimizes sum-squared errr. LMS adapts weights even when classifier utput is crrect. 279 ANN: histry 977 David Rummelhart Intrduced cmputer implementatin f backprpagatin learning and delta rule 982 Jhn Hpfield Implemented recurrent netwrk Develped way t minimize energy f netwrk, defined stable states First NNs n silicn chips built by AT&T using Hpfield net 280 5

6 ANN: histry 989 Cybenk (apprximatin thery) 990 Jang et al. (neur-fuzzy systems) 993 Barrn (cmplexity vs. accuracy) 28 ADAPTIVE NETWORKS 6

7 Adaptive (neural) netwrks Massively cnnected cmputatinal units inspired by the wrking f the human brain Prvide a mathematical mdel fr bilgical neural netwrks (brains) Characteristics: learning frm examples adaptive and fault tlerant rbust fr fulfilling cmplex tasks 283 Netwrk classificatin Learning methds (supervised, unsupervised) Architectures (feedfrward, recurrent) Output types (binary, cntinuus) Nde types (unifrm, hybrid) Implementatins (sftware, hardware) Cnnectin weights (adjustable, hard-wired) Inspiratins (bilgical, psychlgical) 284 7

8 Adaptive netwrk architecture Ndes are static (n dynamics) and parametric Netwrk can cnsist f hetergeneus ndes Links d nt have weights r parameters assciated Nde functins are differentiable except at a finite number f pints x 3 x adaptive ndes Input layer Layer Layer 2 Layer3 (Output layer) 8 fixed ndes x 8 x Adaptive netwrks categries Feedfrward x 3 4 x x x 9 Recurrent x 3 4 x x x

9 Adap. netwrk representatins Layered x 3 4 x x x 9 Tplgical rdering x 8 x x x Feedfrward adaptive netwrk Static mapping between input and utput spaces Aim: cnstruct netwrk t btain nnlinear mapping regulated by a data set (training data set) f desired input-utput pairs f a target system t be mdeled Prcedures: learning rules r adaptatin algrithms (parameter adjustment t imprve netwrk perfrmance) Netwrk perfrmance: measured as the discrepancy between desired and netwrk s utput fr same input (errr measure) 288 9

10 Examples f adaptive netwrks Adaptive netwrk with single linear nde x f 3 x 2 x 3 Perceptrn netwrk (linear classifier) x f ( x, x ; a, a, a ) a x a x a x f 3 f 4 x 4 x3 x 2 x f ( x, x ; a, a, a ) a x a x a if x3 0 x4 f4( x3) 0 if x Examples f adaptive netwrks Multilayer perceptrn (3-3-2 neural netwrk) x 7 x x 2 x 3 Layer 0 (Input layer) exp[ ( w x w x w x t )] Parameter set f nde 7: Layer (Hidden Layer) 7 x 7 4,7 4 5,7 5 6, Layer 2 (Output layer) x 8 { w4,7, w5,7, w6,7, t7} weight threshld 290 0

11 SUPERVISED LEARNING NEURAL NETWORKS Perceptrn Early (and ppular) attempt t build intelligent and self-learning systems by using simple cmpnents Derived frm McCullch-Pitts (943) mdel f the bilgical neurn Mdels utput by weighted cmbinatins f selected features (feature classifier) Essentially a linear classifier Incremental learning rughly based n gradient descent 292

12 Perceptrn (fixed) g x g 2 x 2 w w 2 (adaptive) g 3 x 3 w 3 w 4 q Output g 4 x 4 Input Pattern Feature Detectin Layer 293 Training algrithm Perceptrn (Rsenblatt, 958). Can nly learn linearly separable functins. Training algrithm:. Select an input vectr x frm the training data set 2. If the perceptrn gives an incrrect respnse, mdify all cnnectin weights w i 294 2

13 Training algrithm Weight training: wi( l) wi() l wi() l Weight crrectin is given by the delta rule: wi() l xi() l el () learning rate e(l) = y d (l) y(l) Questin: Can we represent a simple exclusive-or (XOR) functin with a single-layer perceptrn? 295 XOR prblem input : class, x: class 2 X Y Class Hw t classify the patterns crrectly? input Linear classificatin is nt pssible! 296 3

Fr example, cnsider classifying furniture accrding t height and width: 297 Example Each categry can be

14 Example Linearly separable classificatins If classificatin is linearly separable, we can have any number f classes with a perceptrn. Fr example, cnsider classifying furniture accrding t height and width: 297 Example Each categry can be separated frm the ther 2 by a straight line: 3 straight lines each utput nde fires if pint is n right side f straight line: Mre than ne utput nde culd fire at same time! 298 4

15 Artificial neurn x x 2... w 2 Neurn y x n w n x i : i-th input f the neurn w i : synaptic strengh (weight) fr x i y = (w i x i ): utput signal 299 Types f neurns Threshld (McCullch and Pits, 943): n ysignwx i i i Other types f activatin functins (net = w i x i ): 0 step, if net 0 y 0, if net 0 y y linear net sigmid net e 300 5

16 Activatin functins Lgistic * f( x) e Hyperblic tangent * x x e f( x) tanh 2 x e Identity (linear) f ( x) x *sigmidal r squashing functins x Lgistic Functin (a) Hyperblic Tangent Functin (b) Identity Functin (c) 30 Single-layer perceptrn (SLP) Single-layer perceptrn can nly classify linearly separable patterns, regardless f the activatin functin used. Hw t cpe with prblems which are nt linearly separable? Using multilayer neural netwrks! 302 6

17 Multi-Layer Perceptrn fr XOR x w w 2 = -w + x 3 0 x 2 x 3.5 x + + x (a) x3-0.5 x 5 + x 4 (b) x x (c) x x x (d) x x x x x x2 303 Backprpagatin MLP Mst cmmnly used NN structures fr applicatins in wide range f areas: Pattern recgnitin, signal prcessing, data cmpressin and autmatic cntrl Well-knwn applicatins: NETtalk: trained an MLP t prnunce English text; Carnegie Melln University s ALVINN (Autnmus Land Vehicle in a Neural Netwrk) used an NN fr steering an autnmus vehicle; Optical Character Recgnitin (OCR)

18 Multi-Layer Perceptrn Can learn functins that are nt linearly separable. Output signals Input layer st hidden layer 2 nd hidden layer Output layer 305 Mst cmmn MLP Hidden layer w h b h w b Output layer x y x i... w ij h... j w jk k... y k x n w nm h b m h... m w ml b l l y l 306 8

19 Mst cmmn MLP Output f neurns in the hidden-layer h j : tanh n n 0 h h h i i n h wx i0 ij i sigmid h wxb wx j ij i j ij i Output f neurns in the utput-layer y k : m m 0 j j y w h b w h k jk j j jk k m j0 w h jk j linear 307 Learning in NN Bilgical neural netwrks: Synaptic cnnectins amngst neurns which simultaneusly exhibit high activity are strengthned. Artificial neural netwrks: Mathematical apprximatin f bilgical learning. Errr minimizatin (nnlinear ptimizatin prblem). Errr backprpagatin (first-rder gradient) Newtn methds (secnd-rder gradient) Levenberg-Marquardt (secnd-rder gradient) Cnjugate gradients

20 Supervised learning e x Training data: y X x x x T T T 2 N Y y y y T T T 2 N T T 309 Errr backprpagatin Initialize all weights and threshlds t small randm numbers Repeat. Input training examples and cmpute netwrk and hidden layer utputs 2. Adjust utput weights using utput errr 3. Prpagating utput errr backwards, adjust hiddenlayer weights Until satisfied with apprximatin 30 20

21 Backprpagatin in MLP Cmpute the utput f the utput-layer, and cmpute errr: e y, y, k,, l k d k k The cst functin t be minimized is the fllwing: J( w) 2 l N k q e 2 kq N number f data pints 3 Learning using gradient Ouput weight learning fr utput y k : w ( p) w ( p) J( w ) jk jk jk J J J J( wjk ),,, wk w2k w mk T 32 2

22 Output-layer weights h w k w 0k... w 2k w mk Neurn y k h m y w h e y y J w w e m l h 2 k jk j, k d, k k, ( jk, ij ) k j0 2 k 33 Output-layer weights Applying the chain rule with then J ek yk ek,, h e y w J w k k jk jk he j k J J e y w e y w k k jk k k jk j Thus: w ( p) w ( p) J( w ) w ( p) h e jk jk jk jk j k Recall that fr SLP: w xe i i 34 22

23 Hidden-layer weights x w j h h w 0j x 2 w 2j h w nj h Neurn h j x n n h j ij i j j i0 net w x, h tanh( net ) h h h w ( p) w ( p) J( w ) ij ij ij J J h net w h net w j j h h ij j j ij 35 Hidden-layer weights Partial derivatives: J h net h net w k j j ekwjk, j( hj), x h i j k j ij then J w h ij l i j j k jk k x ( h) ( ew ) and l h ij i j j k jk k w( p) x ( h) ( ew ) 36 23

24 Errr backprpagatin algrithm Initialize all weights t small randm numbers Repeat:. Input training example and cmpute netwrk utputs. 2. Adjust utput weights using gradients: w ( p) w ( p) h e jk jk j k 3. Adjust hidden-layer weights: l h h ij ij i j j k jk k w( p ) w( p) x ( h) ( ew ) Until satisfied r fixed number f epchs p 37 First-rder gradient methds Jw ( ) n Jw ( n ) w n w n

25 Secnd-rder gradient methds Update rule fr the weights: w( p) w( p) H( w( p)) J( w( p)) h w( p) w, w, H(w) is the Hessian matrix f w ij jk Learning des nt depend n a learning cefficient Much mre efficient in general 39 Secnd-rder gradient methds Jw ( ) w n w n

26 Apprximatin pwer General functin apprximatrs Feedfrward neural netwrk with ne hidden layer and sigmidal activatin functins can apprximate any cntinuus functin arbitrarily well n a cmpact set (Cybenk) Intuitive relatin t lcalized receptive fields Little cnstructive results 32 Functin apprximatin y w tanh( w xb ) w tanh( w xb ) h h h h h w x+b h z h w x+b h 0 2 x Activatin (weighted summatin) z z

27 Functin apprximatin Transfrmatin thrugh tanh tanh( z 2 ) 2 v 0 2 tanh( z ) x v v 2 2 y wv+wv x Summatin f neurn utputs wv wv RADIAL BASIS FUNCTION NETWORKS 27

28 Radial Basis Functin Netwrks (RBFN) Feedfrward neural netwrks where hidden units d nt implement an activatin functin; they represent a radial basis functin. Develped as an apprach t imprve accuracy and decrease training time cmplexity. 325 Radial Basis Functin Netwrks Activatin functins are radial basis functins Activatin level f i th receptive field (hidden unit): xui Ri( x) Ri i u i center f basis functin i spread f basis functin j =, 2,...,n N cnnectin weights between input and hidden layers... x n... c c m c ml... y y l

29 Radial Basis Functin Netwrks Lcalized activatin functins. Gaussian and lgistic: Ri ( x) exp x u 2 i 2 i Weighted sum r average utput: 2 Ri ( x) exp x u H H cr i i( x) y( x ) cw i i cr i i( x i ) y( x) H i i R( x) H i i i 2 2 i c i can be cnstants r functins f inputs: c i = a it x + b i 327 RBFN architecture Weighted sum Weighted average Hidden layer Hidden layer Lcalized activatin functins in the hidden layer

30 RBFN learning Supervised learning t update all parameters (e.g. with Genetic Algrithms) Sequential training: fix basis functins and then adjust utput weights by: rthgnal least squares data clustering sft cmpetitin based n maximum likelihd estimate i smetimes estimated based n standard deviatins Many ther schemes als exist 329 Least-squares estimate f weights Given basis functins R and a set f input-utput data: [x k, y k ], k =,...,N, estimate ptimal weights c ij. Cmpute the utput f the neurns: 2i Ri( xk) e The utput is linear in the weights: y = Rc. 2. Least squares estimate: 2 i 2 xku T T c[ RR] Ry

RBFN and Sugen systems Equivalent if the fllwing hld: Bth RBFN and TS use same aggregatin methd fr utput (weighted sum r weighted average). Number f basis functins in RBFN equals number f rules in TS.

31 RBFN and Sugen systems Equivalent if the fllwing hld: Bth RBFN and TS use same aggregatin methd fr utput (weighted sum r weighted average). Number f basis functins in RBFN equals number f rules in TS. TS uses Gaussian membership functins with same (variance) as basis functins and rule firing is determined by prduct. RBFN respnse functin (c i ) and TS rule cnsequents are equal. 33 General functin apprximatr 332 3

32 Apprximatin prperties f NN [Cybenk, 989]: A feedfrward NN with at least ne hidden layer can apprximate any cntinuus functin R p R n n a cmpact interval, if sufficient hidden neurns are available. [Barrn, 993]: A feedfrward NN with ne hidden layer and sigmidal activatin functins can achieve an integrated squared errr f the rder J = O( / h). independently f the dimensin f the input space p h: number f hidden neurns (fr smth functins) 333 Apprximatin prperties Fr a basis functin expansin (plynmial, trignmetric, singletn fuzzy mdel, etc.) with h terms, J = O( / h 2/p ), where p is the dimensin f the input. Examples:. p = 2: plynmial J = O( / h 2/2 )=O( / h) neural net J = O( / h) 2. p = 0, h = 2: plynmial J = O(/2 2/0 ) = 0.54 neural net J = O(/2) =

33 Example f aprximatin T achieve the same accuracy: J = O( / h n )=O( / h b ), h n = h 2/p b, h b h n p Hpfield netwrk Recurrent ANN. Example (single-layer): Learning capability is much higher. Successive iteratins may nt necessarily cnverge; may lead t chatic behavir (unstable netwrk)

34 NEURAL NETWORKS MATLAB EXAMPLE (R2007b) Feedfrward backprpagatin netwrk. Input and target P = [ ]; %input T = [ ]; %target 2. Create net help newff net = newff(p,t,5); 3. Simulate and plt net Yi = sim(net,p); plt(p,t,'rs-',p,yi, -') legend('t','yi',0),xlabel('p')

35 Feedfrward backprpagatin netwrk T Yi P 339 Feedfrward backprpagatin netwrk 4. Train the netwrk fr 50 epchs net.trainparam.epchs = 50; net = train(net,p,t); T = [ ]; %target 5. Simulate net and plt the results Y = sim(net,p); figure, plt(p,t,'rs-',p,yi,'b',p,y,'g^') legend('t','yi','y',0),xlabel('p')

36 Feedfrward backprpagatin netwrk T Yi Y P 34 Feedfrward backprpagatin netwrk Cmpute the mean abslute and squared errrs ma_errr = mae(t-y) ma_errr = 0.20 ms_errr = mse(t-y) ms_errr = Plt the netwrk errr figure,plt(p,t-y,''),grid ylabel('errr'),xlabel('p')

37 Feedfrward backprpagatin netwrk errr P 343 Feedfrward backprpagatin netwrk Check the parameters f the netwrk net Sme imprtant parameters inputs: {x cell} f inputs layers: {2x cell} f layers utputs: {x2 cell} cntaining utput targets: {x2 cell} cntaining target biases: {2x cell} cntaining 2 biases inputweights: {2x cell} cntaining input weight layerweights: {2x2 cell} cntaining layer weight

38 Feedfrward backprpagatin netwrk adaptfcn: 'trains' initfcn: 'initlay' perfrmfcn: 'mse' trainfcn: 'trainlm' adaptparam:.passes trainparam:.epchs,.gal,.shw,.time IW: {2x cell} cntaining input weight matrix LW: {2x2 cell} cntaining layer weight matrix b: {2x cell} cntaining 2 bias vectrs 345 Feedfrward backprpagatin netwrk Nte that every time that a netwrk is initialized, different randm numbers are used fr the weights. Example in the fllwing: Initializatin and training f 0 netwrks Cmputatin f mean abslute errr Cmputatin f mean squared errr

39 Feedfrward backprpagatin netwrk MA_errr = []; MS_errr = []; fr i = :0 net = newff(p,t,5); net.trainparam.epchs = 50; net = train(net,p,t); Y = sim(net,p); MA_errr = [MA_errr mae(t-y)]; MS_errr = [MS_errr mse(t-y)]; end 347 Feedfrward backprpagatin netwrk figure, subplt(2,,),plt(ma_errr,''),grid, title('mean Abslute Errr') subplt(2,,2),plt(ms_errr,''),grid title('mean Squared Errr')

40 Feedfrward backprpagatin netwrk 0.25 Mean Abslute Errr Mean Squared Errr

A Scalable Recurrent Neural Network Framework for Model-free

A Scalable Recurrent Neural Netwrk Framewrk fr Mdel-free POMDPs April 3, 2007 Zhenzhen Liu, Itamar Elhanany Machine Intelligence Lab Department f Electrical and Cmputer Engineering The University f Tennessee