Linear and Nonlinear Identification of Dryer System Using Artificial Intelligence and Neural Networks

Transcription

1 Linear and Nonlinear Idenificaion of Dryer Sysem Using Arificial Inelligence and Neural Neworks By Mosafa Darvishi A Pracical Approach Toward Sysem Idenificaion The MATLAB codes relaed o his work can be freely downloaded from my GiHub reposiory July 2006

2 Chaper : Inroducion Chaper Inroducion. Objecives As you read hese words you are using a complex biological neural nework. You have a highly inerconneced se of some 0 neurons o faciliae your reading, breahing, moion and hinking. Each of your biological neurons, a rich assembly of issue and chemisry, has he complexiy, if no he speed, of a microprocessor. Some of your neural srucure was wih you a birh. Oher pars have been esablished by experience. Scieniss have only jus begun o undersand how biological neural neworks operae. I is generally undersood ha all biological neural funcions, including memory, are sored in he neurons and in he connecions beween hem. Learning is viewed as he esablishmen of new connecions beween he neurons or he modificaion of exising connecions. This leads o he following quesions: Alhough we have only rudimenary undersanding of biological neural neworks, is i possible o consruc a small se of simple arificial neurons and perhaps rain hem o serve a useful funcion? The answer is yes. The neurons ha we consider here are no biological. They are exremely simple absracions of biological neurons, realized as elemens in a program or perhaps as circuis made of silicon. Neworks of hese arificial neurons do no have a fracion of he power of he human brain, bu hey can be rained o perform useful funcions.

3 Chaper : Inroducion 2.2 Hisory The hisory of arificial neural neworks is filed wih colorful, creaive individuals from many differen fields, many of whom sruggled for decades o develop conceps ha we now ake for graned. This hisory has been documened by various auhors. Hisories of some of he main neural neworks conribuors are included a he beginning of various chapers hroughou his ex and will no be repeaed here. However, i seems appropriae o give a brief overview, a sample of he major developmens. Some of he background work for he field of neural neworks occurred in he lae 9 h and 20 h cenuries. This consised primarily of inerdisciplinary work in physics, psychology and neurophysiology by such scieniss as Herman Von, Helmholz, Erns Mach and Evan Pavlov. This early work emphasized general heories of learning, vision, condiioning, ec, and did no include specific mahemaical models of neuron operaion. The modern view of neural neworks began in he 940s wih he work of Warren McCulloch and Waler Pis, who showed ha neworks of arificial neurons could, in principle, compue any arihmeic or logical funcion. Their work is ofen acknowledged as he origin of he neural nework field. The firs pracical of arificial neural neworks came in he lae 950s, wih he invenion of he percepron nework and associaed learning rule by Frank Rosenbla. Rosenbla and his colleagues buil a percepron nework and demonsraed is abiliy o perform paern recogniion. This early success generaed a grea deal of ineres in neural nework research. Unforunaely, i was laer shown ha he basic percepron nework could solve only a limied class of problems. A abou he same ime, Bernard Widrow and Ted Hoff inroduced a new learning algorihm and used i o rain adapive linear neural neworks, which were similar in srucure and capabiliy o Rosenbla s percepron. The Widrow-Hoff learning rule is sill in use oday. Unforunaely, boh Rosenbla s and Widrow s neworks suffered from he same inheren limiaions. Rosenbla and Widrow were aware of

4 Chaper : Inroducion 3 hese limiaions and proposed new neworks ha would overcome hem. However, hey were no able o successfully modify heir learning algorihms o rain he more complex neworks. Ineres in neural neworks had falered during he lae 960s because of he lack of new idea and powerful compuers wih which o experimen. During he 980s boh of hese impedimens were overcome, and research in neural neworks increase dramaically. The discipline of neural neworks is presenly living hrough he second of a pair of revoluions, he firs having sared in 943 wih he publicaion of a sarling resul by he American scieniss Warren McCulloch and Waler Pis. They considered he case of a nework made up of binary decision unis BDNs and showed ha such a nework could perform any logical funcion on is inpus. This was aken o mean ha one could mechanize hough, and i helped o suppor he developmen of he digial compuer and is use as a paradigm for human hough. The resul was made even more inriguing due o he fac ha he BDN is a beauifully simple model of he sor of nerve cell used in he human brain o suppor hinking. This led o he suggesion ha here was a good model of human hough. Before he logical paradigm won he day, anoher American, Frank Rosenbla, and several of his colleagues showed how i was possible o rain a nework of BDNs, called a percepron appropriae for a device which could apparenly perceive, so as o be able o recognize a se of paerns chosen beforehand Rosenbla 962. This raining used wha are called he connecion weighs. Each of hese weighs is a number by which one mus muliply he aciviy on a paricular inpu in order o obain he effec of ha inpu on he BDN. The oal aciviy on he BDN is he sum of such erms over all he inpus. The connecion weighs are he mos imporan objecs in a neural nework, and heir modificaion so-called raining is presenly under close sudy. The las word has clearly no ye been said on wha is he mos effecive raining algorihm, and here are many proposals for new learning algorihms each year. The essence of he raining rules was very simple: one would presen he nework wih examples and change hose connecion weighs which led o an improvemen of he resuls, so as o be closer o he desired values. This rule worked miracles, a leas on a se of raher oy example

5 Chaper : Inroducion 4 paerns. This caused a wave of euphoria o sweep hrough he research communiy, and Rosenbla spoke o packed houses when he wen o campuses o describe his resuls. One of he facors in his success was ha he appeared o be building a model duplicaing, o some exen, he aciviy of he human brain. The early resul of McCulloch and Pis indicaed ha a nework of BDNs could solve any logical ask; now Rosenbla had demonsraed ha such a nework could also be rained o classify any paern se. Moreover, he nework of BDNs used by Rosenbla, which possessed a more deailed descripion of he sae of he sysem in erms of he connecion weighs beween he model neurons han did he McCulloch Pis nework, seemed o be a more convincing model of he brain. The brief hisorical accoun given above is no inended o idenify all of he major conribuors, bu i simply o give he reader some feel for how knowledge in he neural neworks has progressed.

6 Chaper : Inroducion 5.3 Applicaions The neural nework applicaions are expanding because neural neworks are good a solving problems, no jus in engineering, science and mahemaics, bu in medicine, business, finance and lieraure as well. Their applicaion o a wide variey of problems in many fields makes hem very aracive. Also faser compuers and faser algorihms have made i possible o use neural neworks o solve complex indusrial problems ha formerly required oo much compuaion. A lis of some applicaions menioned in he lieraure follows. Aerospace High performance aircraf auopilos, fligh pass simulaions, aircraf conrol sysems, auopilo enhancemens, aircraf componen simulaions, aircraf componen faul deecors. Auomoive Auomobile auomaic guidance sysems, warrany aciviy analyzers. Banking Defense Check and oher documen readers, credi applicaion evaluaors. Weapon seering, arge racking, objec discriminaion, facial recogniion, new kinds of sensors, sonar, radar and image signal processing including daa compression, feaure exracion and noise suppression, signal / image idenificaion. Elecronics Code sequence predicion, inegraed circui chip layou, process conrol, chip failure analysis, machine vision, voice synhesis, nonlinear modeling. Medical Breas cancer analysis, EEG and ECG analysis proshesis design, opimizaion of ransplan imes, hospial expense reducion, hospial qualiy improvemen, emergency room es advisemen.

7 Chaper : Inroducion 6 Roboics Speech Trajecory conrol, forklif robo, manipulaor conrollers, vision sysems. Speech recogniion, speech compression, ex o speech synhesis. Telecommunicaion Image and daa compression auomaed informaion services, real-ime ranslaion of spoken language, cusomer paymen processing sysem. Signal processing Neural neworks abiliy o adap coninuously o new daa allows hem o rack changes in a sysem over ime, and heir abiliy o learn arbirary, nonlinear ransfer funcions permis hem o solve problems ha canno be handled adequaely wih more convenional adapive linear echniques. Conrol Neuroconrol is a subse of he larger field of conrol heory, which designs sysems for a broad specrum of applicaions, ranging from simple regulaors like hermosas or muscle neurons o opimal decision-making in complex environmens as in he brain as a whole sysem. Neuroconrol, like classical conrol and arificial inelligence, includes general designs for hree basic ypes of ask: cloning, racking and opimizaion. Neural cloning sysems copy he inpu oupu behavior of human expers or auomaic conrollers. Tracking sysems may be regulaors, or sysems o make a robo arm follow rack a desired pah in space, ec. Opimizaion over ime may be used o solve racking problems, wih improved sabiliy, or o solve planning problems which require real inelligence. This secion compares he pracical advanages and disadvanages of a wide variey of conrol designs, neural and oherwise, ranging from simple regulaors hrough o designs which begin o provide an explanaion of inelligence in brain circuis.

8 Chaper : Inroducion 7.4 Biological inspiraion Abou 00 years ago, a Spanish hisologis, Saniago Ramon y Cajal, he faher of modern brain science, realized ha he brain was made up of discree unis he called neurons, he Greek word for nerves. He described neurons as polarized cells ha receive signals via highly branched exensions, called dendries, and send informaion along unbranched exensions, called axons Fig... Figure.: A biological neuron. The brain is a very complex machine. The human brain conains beween and 8 0 neurons, which is on he same order of magniude as 3 he number of sars in our galaxy, and each neuron is conneced up o 0 4 o 0 oher neurons. In oal, he human brain conains approximaely o 0 inerconnecions. Alhough he brain exhibis a grea diversiy of neuron shapes, dendrie rees, axon lenghs, ec., all neurons seem o process informaion in much he same way. Informaion is ransmied in he form of elecrical impulses called acion poenials via he axons from oher neuron cells. These acion poenials have ampliude of abou 00 millivols, and a frequency of approximaely KHz. When he acion poenial arrives a he axon erminal, he neuron releases chemical neuroransmiers from

9 Chaper : Inroducion 8 he synapic vesicle which mediae he iner-neuron communicaion a specialized connecions called synapses Fig..2 These neuroransmiers bind o recepors in he membrane of he pos synapic neuron o excie or inhibi i. A neuron may have housands of synapses connecing i wih housands of oher neurons. The resuling effec of all exciaions and inhibiions reduces he exernal membrane's elecrical poenial difference by abou 70 millivols he inner surface becomes negaive relaive o he ouer surface. Therefore, he permeabiliy o sodium Na+ increases, leading o he movemen of posiively charged sodium from he exra cellular fluid ino he cell's inerior, he cyoplasm, which in urn may generae an acion poenial in he pos synapic neuron i.e., he neuron fires. This very brief descripion of he neuron's physiology has inspired engineers and scienis o develop adapive sysems wih learning capabiliies. In he following secion we will describe he main compuaional models ha have been developed so far, as a resul of such biological inspiraion. Figure.2: Synapses are specialized connecions beween neurons where chemical neuroransmiers mediae he iner-neuron communicaion.

10 Chaper : Inroducion 9.5 References [] I. Alexander and H. Moron. An Inroducion o Neural Compuing. Inernaional Thomson Compuer Press, second ediion, Ocober 995. [2] T. Ash and G. Corell. Topology-Modifying Neural Nework Algorihms. In M.A. Arbib, edior, Handbook of Brain Theory and Neural Neworks. MIT Press, 995. [3] Marin T.Hagan, Howard B.Demuh, Mark Beale. Neural Nework Design. PWS Publishing Co, 996. [4] MA: MIT Lincoln Laboraory. DARPA Neural Nework Sudy. Lexingon, 988. [5] R.A. Brooks. Inelligence Wihou Reason. Technical Repor A.I., Massachuses Insiue of Technology, April 99.

11 Chaper 2: Neuron Model and Nework Archiecure 0 Chaper 2 Neuron Model and Nework Archiecures 2. Objecives In chaper we presened a simplified descripion of biological neurons and neural neworks. Now we will inroduce our simplified mahemaical model of neuron and will explain how hese arificial neurons can be inerconneced o form a variey of nework archiecures. We will also illusrae he basic operaion of hese neworks hrough some simple examples. 2.2 Neuron Model 2.2. Simple Neuron A neuron wih a single scalar inpu and no bias appears on he lef below. The scalar inpu p is ransmied hrough a connecion ha muliplies is

12 Chaper 2: Neuron Model and Nework Archiecure srengh by he scalar weigh w, o form he produc wp, again a scalar. Here he weighed inpu wp is he only argumen of he ransfer funcion f, which produces he scalar oupu a. The neuron on he righ has a scalar bias, b. You may view he bias as simply being added o he produc wp as shown by he summing juncion or as shifing he funcion f o he lef by an amoun b. The bias is much like a weigh, excep ha i has a consan inpu of. The ransfer funcion ne inpu n, again a scalar, is he sum of he weighed inpu wp and he bias b. This sum is he argumen of he ransfer funcion f. Here f is a ransfer funcion, ypically a sep funcion or a sigmoid funcion, which akes he argumen n and produces he oupu a. Examples of various ransfer funcions are given in he nex secion. Noe ha w and b are boh adjusable scalar parameers of he neuron. The cenral idea of neural neworks is ha such parameers can be adjused so ha he nework exhibis some desired or ineresing behavior. Thus, we can rain he nework o do a paricular job by adjusing he weigh or bias parameers, or perhaps he nework iself will adjus hese parameers o achieve some desired end. All of he neurons have provision for a bias. However, you may omi a bias in a neuron if you wan. As previously noed, he bias b is an adjusable scalar parameer of he neuron. I is no an inpu. However, he consan ha drives he bias is an inpu and mus be reaed as such when considering he linear dependence of inpu vecors. 2.3 Transfer Funcions Many ransfer funcions are included in neural nework design. A complee lis of hem can be found in Transfer Funcion Graphs in he following. Three of he mos commonly used funcions are shown below.

13 Chaper 2: Neuron Model and Nework Archiecure 2 Hard-Limi Transfer Funcion The hard-limi ransfer funcion shown above limis he oupu of he neuron o eiher 0, if he ne inpu argumen n is less han 0 ; or, if n is greaer han or equal o 0. The linear ransfer funcion is shown below. Linear Transfer Funcion Neurons wih his ransfer funcion are used in he ADALINE neworks. The sigmoid ransfer funcion shown below akes he inpu, which may have any value beween plus and minus infiniy, and squashes he oupu ino he range 0 o. Log-Sigmoid Transfer Funcion

14 Chaper 2: Neuron Model and Nework Archiecure 3 This ransfer funcion is commonly used in backpropagaion neworks, in par because i is differeniable. The symbol in he square o he righ of each ransfer funcion graph shown above represens he associaed ransfer funcion. These icons will replace he general f in he boxes of nework diagrams o show he paricular ransfer funcion being used. Mos of he ransfer funcions used in neural nework design are summarized in able 2.. Of course you can define oher ransfer funcion in addiion o hose shown in able 2. if you wish. compe hardlim hardlims logsig poslin purelin radbas salin salins sofmax ansig ribas Compeiive ransfer funcion. Hard limi ransfer funcion. Symmeric hard limi ransfer funcion Log sigmoid ransfer funcion. Posiive linear ransfer funcion Linear ransfer funcion. Radial basis ransfer funcion. Sauraing linear ransfer funcion. Symmeric sauraing linear ransfer funcion Sofmax ransfer funcion. Hyperbolic angen sigmoid ransfer funcion. Triangular basis ransfer funcion. Table 2. Lis of ransfer funcions

15 Chaper 2: Neuron Model and Nework Archiecure Muliple Inpu Neuron A neuron wih a single R-elemen inpu vecor is shown below. Here he individual elemen inpus p, p 2,..., p R are muliplied by weighs w,, w,2,..., w, R and he weighed values are fed o he summing juncion. Their sum is simply wp, he do produc of he single row marix W and he vecor p. The neuron has a bias b, which is summed wih he weighed inpus o form he ne inpu n. This sum n is he argumen of he ransfer funcion f. n w p + w p +...+,,2 2 w, R p R The figure of a single neuron shown above conains a lo of deail. When we consider neworks wih many neurons and perhaps layers of many neurons, here is so much deail ha he main houghs end o be los. Thus, he auhors have devised an abbreviaed noaion for an individual

16 Chaper 2: Neuron Model and Nework Archiecure 5 neuron. This noaion, which will be used laer in circuis of muliple neurons, is illusraed in he diagram shown below. Here he inpu vecor p is represened by he solid dark verical bar a he lef. The dimensions of p are shown below he symbol p in he figure as R. Noe ha we will use a capial leer, such as R in he previous senence, when referring o he size of a vecor. Thus, p is a vecor of R inpu elemens. These inpus pos muliply he single row, R column marixw. As before, a consan eners he neuron as an inpu and is muliplied by a scalar bias b. The ne inpu o he ransfer funcion f is n, he sum of he bias b and he produc wp. This sum is passed o he ransfer funcion f o ge he neuron s oupu a, which in his case is a scalar. Noe ha if we had more han one neuron, he nework oupu would be a vecor. A layer of a nework is defined in he figure shown above. A layer includes he combinaion of he weighs, he muliplicaion and summing operaion here realized as a vecor produc wp, he bias b, and he ransfer funcion f. The array of inpus, vecor p, is no included in or called a layer. Each ime his abbreviaed nework noaion is used, he size of he marices will be shown jus below heir marix variable names. We hope ha his noaion will allow you o undersand he archiecures and follow he marix mahemaics associaed wih hem.

17 Chaper 2: Neuron Model and Nework Archiecure 6 As discussed previously, when a specific ransfer funcion is o be used in a figure, he symbol for ha ransfer funcion will replace he f shown above. Here are some examples. 2.5 Nework Archiecure Two or more of he neurons shown earlier can be combined in a layer, and a paricular nework could conain one or more such layers. Firs consider a single layer of neurons A Layer of Neuron A one-layer nework wih R inpu elemens and S neurons follow.

18 Chaper 2: Neuron Model and Nework Archiecure 7 In his nework, each elemen of he inpu vecor p is conneced o each neuron inpu hrough he weigh marixw. The ih neuron has a summer ha gahers is weighed inpus and bias o form is own scalar oupu n i. The various n i aken ogeher form an S -elemen ne inpu vecor n. Finally, he neuron layer oupus form a column vecor a. We show he expression for a a he boom of he figure. Noe ha i is common for he number of inpus o a layer o be differen from he number of neurons i.e., R S. A layer is no consrained o have he number of is inpus equal o he number of is neurons. You can creae a single composie layer of neurons having differen ransfer funcions simply by puing wo of he neworks shown earlier in parallel. Boh neworks would have he same inpus, and each nework would creae some of he oupus. The inpu vecor elemens ener he nework hrough he weigh marix W. w, w w 2, w S, w,2 w 2,2 w S, w, R w 2, R w S, R Noe ha he row indices on he elemens of marix W indicae he desinaion neuron of he weigh, and he column indices indicae which source is he inpu for ha weigh. Thus, he indices in W, 2 say ha he srengh of he signal from he second inpu elemen o he firs and only neuron isw, 2.The S neuron R inpu one-layer nework also can be drawn in abbreviaed noaion.

19 Chaper 2: Neuron Model and Nework Archiecure 8 Here P is an R lengh inpu vecor, W is an S * R marix, and a and b are S lengh vecors. As defined previously, he neuron layer includes he weigh marix, he muliplicaion operaions, he bias vecor b, he summer, and he ransfer funcion boxes Inpus and Layers We are abou o discuss neworks having muliple layers so we will need o exend our noaion o alk abou such neworks. Specifically, we need o make a disincion beween weigh marices ha are conneced o inpus and weigh marices ha are conneced beween layers. We also need o idenify he source and desinaion for he weigh marices. We will call weigh marices conneced o inpus, inpu weighs; and we will call weigh marices coming from layer oupus, layer weighs. Furher, we will use superscrips o idenify he source second index and he desinaion firs index for he various weighs and oher elemens of he nework. To illusrae, we have aken he one-layer muliple inpu nework shown earlier and redrawn i in abbreviaed form below. As you can see, we have labeled he weigh marix conneced o he inpu, vecor p as an inpu weigh marix IW having a source second index and a desinaion firs index. Also, elemens of layer one, such as is bias, ne inpu, and oupu have a superscrip o say ha hey are associaed wih he firs layer.

20 Chaper 2: Neuron Model and Nework Archiecure 9 In he nex secion, we will use Layer Weigh LW marices as well as Inpu Weigh IW marices. You migh recall from he noaion secion of he Preface ha conversion of he layer weigh marix from mah o code for a paricular nework, called ne is: IW ne. IW{, } 2.6 Muliple Layers of Neurons A nework can have several layers. Each layer has a weigh marix W, a bias vecor b, and an oupu vecor a. To disinguish beween he weigh marices, oupu vecors, ec., for each of hese layers in our figures, we append he number of he layer as a superscrip o he variable of ineres. You can see he use of his layer noaion in he hree-layer nework shown below, and in he equaions a he boom of he figure. 2 The nework shown above has R inpus, S neurons in he firs layer, S neurons in he second layer, ec. I is common for differen layers o have differen numbers of neurons. A consan inpu is fed o he biases for each neuron. Noe ha he oupus of each inermediae layer are he inpus o he following layer. Thus layer 2 can be analyzed as a one-layer nework wih

21 Chaper 2: Neuron Model and Nework Archiecure 20 2 S inpus, S neurons, and an S 2 * S weigh marixw 2. The inpu o layer 2 is a ; he oupu is a 2. Now ha we have idenified all he vecors and marices of layer 2, we can rea i as a single-layer nework on is own. This approach can be aken wih any layer of he nework. The layers of a mulilayer nework play differen roles. A layer ha produces he nework oupu is called an oupu layer. All oher layers are called hidden layers. The hree-layer nework shown earlier has one oupu layer layer 3 and wo hidden layers layer and layer 2. Some auhors refer o he inpus as a fourh layer. We will no use ha designaion. The same hree-layer nework discussed previously also can be drawn using our abbreviaed noaion. Muliple-layer neworks are quie powerful. For insance, a nework of wo layers, where he firs layer is sigmoid and he second layer is linear, can be rained o approximae any funcion wih a finie number of disconinuiies arbirarily well. 3 Here we assume ha he oupu of he hird layer, a is he nework oupu of ineres, and we have labeled his oupu as y. We will use his noaion o specify he oupu of mulilayer neworks. 2.7 Training Syles In his secion, we describe wo differen syles of raining. In incremenal raining he weighs and biases of he nework are updaed each ime an inpu is presened o he nework. In bach raining he weighs and biases are only updaed afer all of he inpus are presened.

22 Chaper 2: Neuron Model and Nework Archiecure Incremenal Training Incremenal raining can be applied o boh saic and dynamic neworks, alhough i is more commonly used wih dynamic neworks, such as adapive filers. In his secion, we demonsrae how incremenal raining is performed on boh saic and dynamic neworks Bach Training Bach raining, in which weighs and biases are only updaed afer all of he inpus and arges are presened, can be applied o boh saic and dynamic neworks.

23 Chaper 2: Neuron Model and Nework Archiecure Summary The inpus o a neuron include is bias and he sum of is weighed inpus using he inner produc. The oupu of a neuron depends on he neuron s inpus and on is ransfer funcion. There are many useful ransfer funcions. A single neuron canno do very much. However, several neurons can be combined ino a layer or muliple layers ha have grea power. Hopefully his oolbox makes i easy o creae and undersand such large neworks. The archiecure of a nework consiss of a descripion of how many layers a nework has, he number of neurons in each layer, each layer s ransfer funcion, and how he layers connec o each oher. The bes archiecure o use depends on he ype of problem o be represened by he nework. A nework effecs a compuaion by mapping inpu values o oupu values. The paricular mapping problem o be performed fixes he number of inpus, as well as he number of oupus for he nework. Aside from he number of neurons in a nework s oupu layer, he number of neurons in each layer is up o he designer. Excep for purely linear neworks, he more neurons in a hidden layer, he more powerful he nework. If a linear mapping needs o be represened linear neurons should be used. However, linear neworks canno perform any nonlinear compuaion. Use of a nonlinear ransfer funcion makes a nework capable of soring nonlinear relaionships beween inpu and oupu. A very simple problem can be represened by a single layer of neurons. However, single-layer neworks canno solve cerain problems. Muliple feed-forward layers give a nework greaer freedom. For example, any reasonable funcion can be represened wih a wo-layer nework: a sigmoid layer feeding a linear oupu layer. Neworks wih biases can represen relaionships beween inpus and oupus more easily han neworks wihou biases. For example, a neuron wihou a bias will always have a ne inpu o he ransfer funcion of zero when all of is inpus are zero. However, a neuron wih a bias can learn o

24 Chaper 2: Neuron Model and Nework Archiecure 23 have any ne ransfer funcion inpu under he same condiions by learning an appropriae value for he bias. Feed-forward neworks canno perform emporal compuaion. More complex neworks wih inernal feedback pahs are required for emporal behavior. If several inpu vecors are o be presened o a nework, hey may be presened sequenially or concurrenly. Baching of concurren inpus is compuaionally more efficien and may be wha is desired in any case. The marix noaion used in MATLAB makes baching simple.

25 Chaper 2: Neuron Model and Nework Archiecure References [] I. Alexander and H. Moron. An Inroducion o Neural Compuing. Inernaional Thomson Compuer Press, second ediion, Ocober 995. [2] T. Ash and G. Corell. Topology-Modifying Neural Nework Algorihms. In M.A. Arbib, edior, Handbook of Brain Theory and Neural Neworks. MIT Press, 995. [3] Marin T.Hagan, Howard B.Demuh, Mark Beale. Neural Nework Design. PWS Publishing Co, 996. [4] MA: MIT Lincoln Laboraory. DARPA Neural Nework Sudy. Lexingon, 988. [5] R.A. Brooks. Inelligence Wihou Reason. Technical Repor A.I., Massachuses Insiue of Technology, April 99.

26 Chaper 3: Fundamenal of Various Neworks and Learning Rules 25 Chaper 3 Fundamenals of Various Neworks and Learning Rules 3. Objecives This chaper has a number of objecives. Firs we wan o inroduce you he learning rules, mehods of deriving he nex changes ha migh be made in a nework, and raining, a procedure whereby a nework is acually adjused o do a paricular job. In his secion we discuss abou various neworks and heir capabiliies. There is a choice of using differen neworks according o he sysem srucure. 3.2 Percepron 3.2. Inroducion Rosenbla creaed many variaions of he percepron. One of he simples was a single-layer nework whose weighs and biases could be rained o produce a correc arge vecor when presened wih he corresponding inpu vecor. The raining echnique used is called he percepron learning rule. The percepron generaed grea ineres due o is abiliy o generalize from is raining vecors and learn from iniially randomly disribued connecions. Perceprons are especially suied for simple problems in paern classificaion. They are fas and reliable neworks for he problems hey can solve. In addiion, an undersanding of he operaions of he percepron provides a good basis for undersanding more complex neworks. In his chaper we define wha we mean by a learning rule, explain he percepron nework and is learning rule, and ell you how o iniialize and simulae percepron neworks.

27 Chaper 3: Fundamenal of Various Neworks and Learning Rules Neuron Model A percepron neuron, which uses he hard-limi ransfer funcion hardlim, is shown below. Each exernal inpu is weighed wih an appropriae weigh W j, and he sum of he weighed inpus is sen o he hard-limi ransfer funcion, which also has an inpu of ransmied o i hrough he bias. The hardlimi ransfer funcion, which reurns a 0 or a, is shown below. Hard-Limi Transfer Funcion The percepron neuron produces a if he ne inpu ino he ransfer funcion is equal o or greaer han 0; oherwise i produces a 0. The hard-limi ransfer funcion gives a percepron he abiliy o classify inpu vecors by dividing he inpu space ino wo regions. Specifically, oupus will be 0 if he ne inpu n is less han 0, or if he ne inpu n is

28 Chaper 3: Fundamenal of Various Neworks and Learning Rules 27 0 or greaer. The inpu space of a wo-inpu hard limi neuron wih he weighsw, W and a bias b, is shown below.,,2 Two classificaion regions are formed by he decision boundary line L aw P + b 0. This line is perpendicular o he weigh marix W and shifed according o he bias b. Inpu vecors above and o he lef of he line L will resul in a ne inpu greaer han 0; and herefore, cause he hard-limi neuron o oupu a. Inpu vecors below and o he righ of he line L cause he neuron o oupu 0. The dividing line can be oriened and moved anywhere o classify he inpu space as desired by picking he weigh and bias values. Hard-limi neurons wihou a bias will always have a classificaion line going hrough he origin. Adding a bias allows he neuron o solve problems where he wo ses of inpu vecors are no locaed on differen sides of he origin. The bias allows he decision boundary o be shifed away from he origin as shown in he plo above.

29 Chaper 3: Fundamenal of Various Neworks and Learning Rules Percepron Archiecure The percepron nework consiss of a single layer of S percepron neurons conneced o R inpus hrough a se of weighs W i, j as shown below in wo forms. As before, he nework indices i and j indicae ha W i, j is he srengh of he connecion from he j h inpu o he i h neuron. The percepron learning rule ha we will describe shorly is capable of raining only a single layer. Thus, here we will consider only one-layer neworks. This resricion places limiaions on he compuaion a percepron can perform Learning Rules We define a learning rule as a procedure for modifying he weighs and biases of a nework. This procedure may also be referred o as a raining algorihm.. The learning rule is applied o rain he nework o perform some paricular ask. Learning rules in his oolbox fall ino wo broad caegories: supervised learning, and unsupervised learning. In supervised learning, he learning rule is provided wih a se of examples he raining se of proper nework behavior

30 Chaper 3: Fundamenal of Various Neworks and Learning Rules 29 {p, },{p 2, 2},...,{p Q, Q} Where is pq an inpu o he nework, and Q is he corresponding correc arge oupu. As he inpus are applied o he nework, he nework oupus are compared o he arges. The learning rule is hen used o adjus he weighs and biases of he nework in order o move he nework oupus closer o he arges. The percepron learning rule falls in his supervised learning caegory. In unsupervised learning, he weighs and biases are modified in response o nework inpus only. There are no arge oupus available. Mos of hese algorihms perform clusering operaions. They caegorize he inpu paerns ino a finie number of classes. This is especially useful in such applicaions as vecor quanizaion. As noed, he percepron discussed in his chaper is rained wih supervised learning. Hopefully, a nework ha produces he righ oupu for a paricular inpu will be obained Percepron Learning Rules learnp Perceprons are rained on examples of desired behavior. The desired behavior can be summarized by a se of inpu, oupu pairs p, p 2,..., p Q Q where p is an inpu o he nework and is he corresponding correc arge oupu. The objecive is o reduce he error e, which is he difference a beween he neuron response a, and he arge vecor. The percepron learning rule learnp calculaes desired changes o he percepron s weighs and biases given an inpu vecor p, and he associaed error e. The arge vecor mus conain values of eiher 0 or, as perceprons wih hardlim ransfer funcions can only oupu such values. Each ime learnp is execued, he percepron has a beer chance of producing he correc oupus. The percepron rule is proven o converge on a soluion in a finie number of ieraions if a soluion exiss. If a bias is no used, learnp works o find a soluion by alering only he weigh vecor w o poin oward inpu vecors o be classified as, and

31 Chaper 3: Fundamenal of Various Neworks and Learning Rules 30 away from vecors o be classified as 0. This resuls in a decision boundary ha is perpendicular o w, and which properly classifies he inpu vecors. There are hree condiions ha can occur for a single neuron once an inpu vecor p is presened and he nework s response a is calculaed: CASE. If an inpu vecor is presened and he oupu of he neuron is correc a, and e a 0, hen he weigh vecor w is no alered. CASE 2. If he neuron oupu is 0 and should have been a 0 and, and e a, he inpu vecor p is added o he weigh vecor w. This makes he weigh vecor poin closer o he inpu vecor, increasing he chance ha he inpu vecor will be classified as a in he fuure. CASE 3. If he neuron oupu is and should have been 0 a and 0, and e a, he inpu vecor p is subraced from he weigh vecor w. This makes he weigh vecor poin farher away from he inpu vecor, increasing he chance ha he inpu vecor is classified as a 0 in he fuure. The percepron learning rule can be wrien more succincly in erms of he error e a, and he change o be made o he weigh vecor w : CASE. If e 0, hen make a change equal w o 0. T CASE 2. If e, hen make a change wequal o p T CASE 3. If e, hen make a change w equal o p. All hree cases can hen be wrien wih a single expression: T w a p e p T We can ge he expression for changes in a neuron s bias by noing ha he bias is simply a weigh ha always has an inpu of : b a e For he case of a layer of neurons we have: T T w a P e P and b a E

32 Chaper 3: Fundamenal of Various Neworks and Learning Rules 3 The Percepron Learning Rule can be summarized as follows w new old T w e p and b new b old + e where e a Limiaions and cauions Percepron neworks have several limiaions. Firs, he oupu values of a percepron can ake on only one of wo values 0 or due o he hardlimi ransfer funcion. Second, perceprons can only classify linearly separable ses of vecors. If a sraigh line or a plane can be drawn o separae he inpu vecors ino heir correc caegories, he inpu vecors are linearly separable. If he vecors are no linearly separable, learning will never reach a poin where all vecors are classified properly. Noe, however, ha i has been proven ha if he vecors are linearly separable, perceprons rained adapively will always find a soluion in finie ime Ouliers and he Normalized percepron Rule Long raining imes can be caused by he presence of an oulier inpu vecor whose lengh is much larger or smaller han he oher inpu vecors. Applying he percepron learning rule involves adding and subracing inpu vecors from he curren weighs and biases in response o error. Thus, an inpu vecor wih large elemens can lead o changes in he weighs and biases ha ake a long ime for a much smaller inpu vecor o overcome. By changing he percepron learning rule slighly, raining imes can be made insensiive o exremely large or small oulier inpu vecors. Here is he original rule for updaing weighs: T w a p e p T

33 Chaper 3: Fundamenal of Various Neworks and Learning Rules 32 As shown above, he larger an inpu vecor p, he larger is effec on he weigh vecor w. Thus, if an inpu vecor is much larger han oher inpu vecors, he smaller inpu vecors mus be presened many imes o have an effec. The soluion is o normalize he rule so ha effec of each inpu vecor on he weighs is of he same magniude: p T p T w a e p p 3.3 Muli Layer Percepron MLP Nework The muli layer percepron MLP is he mos widely known and used neural nework archiecure. An MLP nework is a universal approximaor. This means ha an MLP can approximae any smooh funcion o an arbirary degree of accuracy as he number of hidden layer neuron increases. The universal approximaion capabiliy is an imporan propery since i jusifies he applicaion of he MLP o any funcion approximaion problem. However, virually all oher widely used approximaors polynomials, fuzzy models, and mos oher neural nework archiecures are universal approximaors oo furhermore, he proof for he universal approximaion abiliy is no consrucive in he sense ha i ells he user how many hidden layer neurons would be required o achieve a given accuracy. The reason why he universal approximaion capabiliy of he MLP has araced such aenion is ha, in conras o many oher approximaors, he MLP oupu is compued by he combinaion of one-dimensional funcions. This is a direc consequence of he ridge consrucion mechanism. In fac, i is no inuiively undersandable ha a general funcion of many inpus can be approximaed wih arbirary accuracy by a se of one-dimensional funcions. Noe ha he approximaion abiliies of he MLP have no direc relaionship o Kolmogorov s heorem, wih saes ha a mulidimensional funcion can be exacly represened by onedimensional funcion. The famous heory by Kolmogorov from 957 disproved a conjecure by Hilber in conex wih his 3 h problem from he famous lis of Hilber s 23 unsolved mahemaical problems in 900.

34 Chaper 3: Fundamenal of Various Neworks and Learning Rules 33 The MLP nework as described so far represens only mos commonly applied MLP ype differen varians exis. Someimes he oupu neuron is no of he pure linear combinaion ype bu is chosen as a complee pecepron. Of course, his resrics he MLP oupu o he inerval [0,] or [-,]. Furhermore, he oupu layer weighs become nonlinear, and hus raining becomes harder. Anoher possible exension is he use of more han one hidden layer. Muliple hidden layers make he nework much more powerful and complex. An MLP nework consiss of wo ypes of parameers: Oupu layer weighs are linear parameers. They deermine he ampliudes of he basis funcions and he operaing poin. Hidden layer weighs are nonlinear parameers, and deermine he direcion, slopes, and posiions of he basis funcions MLP properies The mos imporan properies of MLP neworks can be summarized as follows: Inerpolaion behavior ends o be monoonic owing o he shape of he sigmoid funcions. Exrapolaion behavior is consan in he long range owing o he sauraion of he sigmoid funcions. However, in he shor range exrapolaion behavior can be linear if a sigmoid funcion has a very small slope. A difficuly is ha i is no clear o he user a which ampliude level he nework exrapolaes, and hus he nework s exrapolaion behavior is hard o predic. Localiy does no exis since a change in one oupu layer weigh significanly influences he model oupu for a large region of he inpu space. Neverheless, since he acivaion funcions are no sricly global, a MLP has approximaion flexibiliy only if he acivaion funcions of he neurons are no sauraed. So he approximaion mechanism possesses some localiy. Accuracy is ypically very high. Owing o he opimizaion of he hidden layer weigh he MLP is exremely powerful and usually requires fewer neurons and fewer parameers han oher model

35 Chaper 3: Fundamenal of Various Neworks and Learning Rules 34 archiecures o achieve comparable approximaion accuracy. This propery can be inerpreed as a high informaion compression capabiliy, which is paid for by long raining imes and he oher disadvanages caused by nonlinear parameers. Sensiiviy o noise is very low since, owing o he global characers, almos all raining daa samples are exploied o esimae all model parameers. Training speed is very slow since nonlinear opimizaion echniques have o be applied and perhaps repeaed for several nework weigh iniializaions if an unsaisfacory local opimum has been reached. Usage is very high. MLP nework are sill he sandard neural neworks. 3.4 Radial Basis Funcion RBF Nework Inroducion This documen is an inroducion o linear neural neworks, paricularly radial basis funcion RBF neworks. The approach described places an emphasis on reaining, as much as possible, he linear characer of RBF neworks5, despie he fac ha for good generalizaion here has o be some kind of nonlinear opimizaion. The wo main advanages of his approach are keeping he mahemaics simple i is jus linear algebra and he compuaions relaively cheap here is no opimizaion by general purpose gradien descen algorihms. Linear models have been sudied in saisics for abou 200 years and he heory is applicable o RBF neworks which are jus one paricular ype of linear model. However, he fashion for neural neworks, which sared in he mid-80's, has given rise o new names for conceps already familiar o saisicians. The documen is srucured as follows. We firs ouline an overview of RBF, Linear models, Radial funcions, Gaussian RBF Nework, and Leas squares.

36 Chaper 3: Fundamenal of Various Neworks and Learning Rules 35 In he following we describe Comparison beween RBF Neworks and MLPs, RBF properies Overview An RBF nework generally consiss of wo weigh layers: he hidden layer and he oupu layer. They can be described by he following equaion: n h y w + 0 w f x c i i i where f are he radial basis funcions, w i are he oupu layer weighs, w0 is he oupu offse, x are he inpus o he nework, ci are he ceners associaed wih he basis funcions, nh is he number of basis funcions in he nework, and denoes he Euclidean norm. Given he vecor x [ x,..., xn] On R n, he Euclidean norm on his space measures he size of he vecor in a general sense and is defined as 2 2 i x 2 n x x x i The basic operaion of an RBF nework can be hough of as follows. n Each inpu vecor x passed o he nework is shifed in R space according o some sored parameers he ceners in he nework. The Euclidean norm is compued for each of hese shifed vecors x ci fori,..., nh. Each cj is a vecor wih he same number of elemens as he inpu vecor x. Noe ha here is one comparison or shifing operaion for each c j sored in he nework, and one cener is defined for each radial basis funcion in he nework. Ceners which are closes o he inpu daa vecor will have he smalles oupu, he limiing case being where a cener exacly coincides wih an inpu vecor. In his case, he Euclidean disance is zero. Conversely, when he daa are furher away from a given cener, he oupu will become larger. In ha case, he Euclidean disance will also become large.

37 Chaper 3: Fundamenal of Various Neworks and Learning Rules 36 Now consider he acion of he Gaussian basis funcion on he resuling oupus from he Euclidean disance measures. For daa which are far away from he ceners, he oupu from he corresponding basis funcions will be small, approaching zero wih increased disance. On he oher hand, for daa which are close o he ceners, he oupu from he corresponding basis funcions will be larger, approaching one wih decreased disance. Hence, radial basis funcion neworks are able o model daa in a local sense. For each inpu daa vecor, one or more basis funcions provide an oupu. In he exreme case, one basis funcion is used for every inpu daa vecor, and he ceners hemselves are idenical o he daa vecors. Therefore, i is hen a simple maer o map he oupu from he basis funcion o any required oupu value by means of he oupu layer weighs Linear Models A linear model for a funcion y x akes he form m f x w jh j j x The model f is expressed as a linear combinaion of a se of m fixed funcions ofen called basis funcions by analogy wih he concep of a vecor being composed of a linear combinaion of basis vecors. The choice of he leer he leer `h' for he basis funcions reflecs our ineres in neural neworks which have weighs and hidden unis. ' w ' for he coefficiens of he linear combinaions and The flexibiliy of f, is abiliy o _ many differen funcions, derives only from he freedom o choose differen values for he weighs. The basis funcions and any parameers which hey migh conain are fixed. If his is no he case, if he basis funcions can change during he learning process, hen he model is nonlinear. Any se of funcions can be used as he basis se alhough i helps, of course, if hey are well behaved differeniable. However, models conaining only basis funcions drawn from one paricular class have a special ineres. Classical saisics abounds wih linear models whose

38 Chaper 3: Fundamenal of Various Neworks and Learning Rules 37 basis funcions are polynomials h j x x j or variaions on he heme. Combinaions of sinusoidal waves Fourier series 2πj x θ j h x j sin m are ofen used in signal processing applicaions. Logisic funcions, of he sor h x T + exp b x b 0 are popular in arificial neural neworks, paricularly in muli-layer perceprons Radial Funcions Radial funcions are a special class of funcion. Their characerisic feaure is ha heir response decreases or increases monoonically wih disance from a cenral poin. The cenre, he disance scale, and he precise shape of he radial funcion are parameers of he model, all fixed if i is linear. A ypical radial funcion is he Gaussian which, in he case of a scalar inpu, is: h x exp x c 2 r 2 Is parameers are is cenre c and is radius r. Figure 2 illusraes a Gaussian RBF wih cenre c 0 and radius r. A Gaussian RBF monoonically decreases wih disance from he cenre. In conras, a muliquadric RBF which, in he case of scalar inpu, is h x r 2 + x c r 2 monoonically increases wih disance from he cenre see Figure 3.. Gaussian-like RBFs are local give a significan response only in a neighborhood near he cenre and are more commonly used han

39 Chaper 3: Fundamenal of Various Neworks and Learning Rules 38 muliquadric-ype RBFs which have a global response. They are also more biologically plausible because heir response is finie. Figure 3. Gaussian lef and muliquadric RBFs Radial Basis funcion Neworks Radial funcions are simply a class of funcions. In principle, hey could be employed in any sor of model linear or nonlinear and any sor of nework single-layer or muli-layer. However, since Broomhead and Lowe's 988 seminal paper [3], radial basis funcion neworks RBF neworks have radiionally been associaed wih radial funcions in a single-layer nework such as shown in figure 3.2. Figure 3.2 The radiional radial basis funcion nework.

40 Chaper 3: Fundamenal of Various Neworks and Learning Rules 39 In he figure above each of n componens of he inpu vecor x feeds forward o m basis funcions whose oupus are linearly combined wih weighs { ino he nework oupu f x. w } m j j An RBF nework is nonlinear if he basis funcions can move or change size or if here is more han one hidden layer. Here we focus on singlelayer neworks wih funcions which are fixed in posiion and size. We do use nonlinear opimizaion bu only for he regularizaion parameers in ridge regression and he opimal subse of basis funcions in forward selecion. We also avoid he kind of expensive nonlinear gradien descen algorihms such as he conjugae gradien and variable meric mehods ha are employed in explicily nonlinear neworks. Keeping one foo firmly planed in he world of linear algebra makes analysis easier and compuaions quicker Gaussian RBF Nework The Gaussian RBF nework can be wrien using convenional ime series noaion as y z w where w is he oupu layer weigh vecor and z is he basis funcion oupu vecor a ime given, respecively, by w [w,w 0 z [, z,..., z,..., w nh nh ] ] z i exp x c 2 i r i 2 c ih basis funcion cener vecor i r ih basis funcion widh. i Noe ha z i corresponds o he oupu from he ih basis funcion uni due o he inpu vecor x presened o he nework a ime defined by x [x,..., x ] nh

41 Chaper 3: Fundamenal of Various Neworks and Learning Rules 40 I is also possible o wrie he radial basis in vecor noaion for sequences of inpu and oupu daa, as follows: where y Z w y [y,..., y Z [z z j [, z,..., z j m m ] ],..., z jn h ] j,..., m z ji exp x c 2 i r i 2 Here, z ji corresponds o he oupu from he ih basis funcion uni due o he jh inpu vecor x j, defined by x j [x j,..., x jn ] h Leas Squares When applied o supervised learning wih linear models secion 3 he leas squares principle leads o a paricularly easy opimizaion problem. If he model is m f x w j h j j x and he raining se is { ˆ } p x i, y i i minimize he sum-squared-error, hen he leas squares recipe is o p S i yˆ f x i i 2

42 Chaper 3: Fundamenal of Various Neworks and Learning Rules 4 wih respec o he weighs of he model. If a weigh penaly erm is added o he sum-squared-error, as is he case wih ridge regression secion 6, hen he following cos funcion is minimized C p yi f xi m 2 ˆ + λ w i j } m j j where he { λ are regularizaion parameers. j 2 j Comparison beween RBF Neworks and MLPs Some work has been performed o show he relaionship beween RBF neworks and MLPs. Essenially, if one considers ha an MLP is a universal approximaor, hen i may approximae an RBF nework and vice versa. Maruyama, Girosi, and Poggio also showed ha for normalized inpus, MLPs can be considered o be RBF neworks wih irregular basis funcions. In a similar vein, Jang and Sun showed he equivalence beween RBF neworks and fuzzy inference sysems. Alhough hese resuls are of pedagogical ineres, i should be kep in mind ha, since boh ypes of neworks are capable of universaion approximaion capabiliies, he main reason o consider one nework over anoher is is learning performance on paricular daa ses RBF Properies The mos imporan properies of RBF neworks are as follows: Inerpolaion behavior ends o possess dips when he sandard deviaions of some RBFs are chosen oo small, and i hus has a endency o be non-monoonic. Exrapolaion behavior ends o zeros because he acivaion funcions are ypically local. Localiy is guaraneed when local acivaion funcions are employed.

43 Chaper 3: Fundamenal of Various Neworks and Learning Rules 42 Accuracy is ypically medium. Because he hidden layer parameers of he RBF nework are usually no opimized, bu deermined heurisically, many neurons are required o achieve high accuracy. If he hidden layer parameers are opimized, accuracy is comparable wih ha of MLP neworks perhaps slighly worse owing o he local characer. Sensiiviy o noise is low since, basis funcions are usually placed only in a regions where enough daa is variable. Furhermore small variaions in he nework parameers have only local effec. Training speed is fas, medium, and slow for he combined unsupervised / supervised learning, he subse selecion, and he nonlinear opimizaion mehod respecively. Usage is medium. RBF nework became more popular in he lae 990s.

44 Chaper 3: Fundamenal of Various Neworks and Learning Rules Backpropagaion 3.5. Overview Backpropagaion was creaed by generalizing he Widrow-Hoff learning rule o muliple-layer neworks and nonlinear differeniable ransfer funcions. Inpu vecors and he corresponding arge vecors are used o rain a nework unil i can approximae a funcion, associae inpu vecors wih specific oupu vecors, or classify inpu vecors in an appropriae way as defined by you. Neworks wih biases, a sigmoid layer, and a linear oupu layer are capable of approximaing any funcion wih a finie number of disconinuiies. Sandard backpropagaion is a gradien descen algorihm, as is he Widrow-Hoff learning rule, in which he nework weighs are moved along he negaive of he gradien of he performance funcion. The erm backpropagaion refers o he manner in which he gradien is compued for nonlinear mulilayer neworks. There are a number of variaions on he basic algorihm ha are based on oher sandard opimizaion echniques, such as conjugae gradien and Newon mehods. The Neural Nework Toolbox implemens a number of hese variaions. This chaper explains how o use each of hese rouines and discusses he advanages and disadvanages of each Archiecure and Neuron Model ansig, logsig, purelin This secion presens he archiecure of he nework ha is mos commonly used wih he backpropagaion algorihm - he mulilayer feedforward nework. An elemenary neuron wih R inpus is shown below. Each inpu is weighed wih an appropriae w. The sum of he weighed inpus and he bias forms he inpu o he ransfer funcion f. Neurons may use any differeniable ransfer funcion f o generae heir oupu.

45 Chaper 3: Fundamenal of Various Neworks and Learning Rules 44 Mulilayer neworks ofen use he log-sigmoid ransfer funcion logsig. Log-Sigmoid Transfer Funcion The funcion logsig generaes oupus beween 0 and as he neuron s ne inpu goes from negaive o posiive infiniy. Alernaively, mulilayer neworks may use he an-sigmoid ransfer funcion ansig. Tan-Sigmoid Transfer Funcion Occasionally, he linear ransfer funcion purelin is used in backpropagaion neworks.

46 Chaper 3: Fundamenal of Various Neworks and Learning Rules 45 Linear Transfer Funcion If he las layer of a mulilayer nework has sigmoid neurons, hen he oupus of he nework are limied o a small range. If linear oupu neurons are used he nework oupus can ake on any value. In backpropagaion i is imporan o be able o calculae he derivaives of any ransfer funcions used. Each of he ransfer funcions above, ansig, logsig, and purelin, have a corresponding derivaive funcion: dansig, dlogsig, and dpurelin Feedforward Neworks A single-layer nework of S logsig neurons having R inpus is shown below in full deail on he lef and wih a layer diagram on he righ.

47 Chaper 3: Fundamenal of Various Neworks and Learning Rules 46 Feedforward neworks ofen have one or more hidden layers of sigmoid neurons followed by an oupu layer of linear neurons. Muliple layers of neurons wih nonlinear ransfer funcions allow he nework o learn nonlinear and linear relaionships beween inpu and oupu vecors. The linear oupu layer les he nework produce values ouside he range o +. On he oher hand, if you wan o consrain he oupus of a nework such as beween 0 and, hen he oupu layer should use a sigmoid ransfer funcion such as logsig. As noed previously, for muliple-layer neworks we use he number of he layers o deermine he superscrip on he weigh marices. The appropriae noaion is used in he wo-layer ansig/purelin nework shown nex. This nework can be used as a general funcion approximaor. I can approximae any funcion wih a finie number of disconinuiies, arbirarily well, given sufficien neurons in he hidden layer.

48 Chaper 3: Fundamenal of Various Neworks and Learning Rules Self-Organizing Neworks 3.6. Inroducion Self-organizing in neworks is one of he mos fascinaing opics in he neural nework field. Such neworks can learn o deec regulariies and correlaions in heir inpu and adap heir fuure responses o ha inpu accordingly. The neurons of compeiive neworks learn o recognize groups of similar inpu vecors. Self-organizing maps learn o recognize groups of similar inpu vecors in such a way ha neurons physically near each oher in he neuron layer respond o similar inpu vecors. Learning vecor quanizaion LVQ is a mehod for raining compeiive layers in a supervised manner. A compeiive layer auomaically learns o classify inpu vecors. However, he classes ha he compeiive layer finds are dependen only on he disance beween inpu vecors. If wo inpu vecors are very similar, he compeiive layer probably will pu hem in he same class. There is no mechanism in a sricly compeiive layer design o say wheher or no any wo inpu vecors are in he same class or differen classes Compeiive Learning The neurons in a compeiive layer disribue hemselves o recognize frequenly presened inpu vecors. The archiecure for a compeiive nework is shown below.

49 Chaper 3: Fundamenal of Various Neworks and Learning Rules 48 The dis box in his figure acceps he inpu vecor P and he inpu, weigh marix IW, and produces a vecor having S elemens. The elemens are he negaive of he disances beween he inpu vecor and, vecors IW formed from he rows of he inpu weigh marix. j The ne inpu n of a compeiive layer is compued by finding he negaive disance beween inpu vecor P and he weigh vecors and adding he biases b. If all biases are zero, he maximum ne inpu a neuron can have is 0. This occurs when he inpu vecor P equals ha neuron s weigh vecor.

50 Chaper 3: Fundamenal of Various Neworks and Learning Rules References [] Nelles, O., Nonlinear Sysem Idenificaion-From Classical Approaches o Neural Neworks and Fuzzy Model, springer-velag, Berlin Heidelberg, New York. [2] Rosenbla, F., Principles of Neurodynamics, Washingon D.C. Sparan Press, 96. This book presens all of Rosenbla s resuls on perceprons. In paricular, i presens his mos imporan resul, he percepron learning heorem. [3] Riedmiller, M., and H. Braun, A direc adapive mehod for faser backpropagaion learning: The RPROP algorihm, Proceedings of he IEEE Inernaional Conference on Neural Neworks, 993. [4] Vogl, T. P., J. K. Mangis, A. K. Rigler, W. T. Zink, and D. L. Alkon, Acceleraing he convergence of he backpropagaion mehod, Biological Cyberneics, vol. 59, pp , 988. [5] Hagan, M. T., and M. Menhaj, Training feedforward neworks wih he Marquard algorihm, IEEE Transacions on Neural Neworks, vol. 5, no. 6, pp , 994. [6] Kohonen, T., Self-Organizaion and Associaive Memory, 2 nd Ediion, Berlin: Springer-Verlag, 987. This book analyzes several learning rules. The Kohonen learning rule is hen inroduced and embedded in self-organizing feaure maps. Associaive neworks are also sudied. [7] Kohonen, T., Self-Organizing Maps, Second Ediion, Berlin: Springer-Verlag, 997. This book discusses he hisory, fundamenals, heory, applicaions and hardware of self-organizing maps. I also includes a comprehensive lieraure survey.

51 Chaper 4: Idenificaion Tes Design and Daa Prereamen 50 Chaper 4 Idenificaion Tes Design and Daa Prereamen 4. Inroducion In his chaper he problem of es design and relaed issues will de described. The basic assumpion is ha he idenified model will be used in conrol. Idenificaion is o absrac mahemaical models from daa. Daa are colleced by observing he given process or sysems. For some sysems, such as economical, ecological and biological sysems, he daa collecion is ofen a passive process, meaning ha we can only observe he sysem oucomes under a given circumsance and i is oo cosly or echnically impossible o inroduce exra exciaion on hese sysems. The daa from hese sysems may no be informaive enough. This can make he idenificaion of hese sysems difficul. On he oher hand, in conrol sysems, i is mos ofen possible o used exra exciaions in order o obain informaive daa for model idenificaion. Process exciaion and daa collecion are called idenificaion es or experimen. Idenificaion es design plays a dominae role in pracical idenificaion for conrol. In idenificaion lieraure, much more aenion has been paid o esimaion algorihms han o he design of idenificaion ess. If he ess are no performed properly hen no idenificaion algorihms, however sophisicaed, can arrive a a relevan process model for conrol. The belief ha idenificaion is jus a se of algorihms and i is simply daa-in-and-model-ou will ofen lead o disappoining resuls in idenificaion applicaions. The purpose of idenificaion ess is o excie and o collec relevan informaion abou he process dynamics and is environmen disurbances. Ofen several differen ypes of ess need o be performed, each of hem for collecing cerain kind of informaion abou he process. In his par he idenificaion signal waveform will be

52 Chaper 4: Idenificaion Tes Design and Daa Prereamen 5 sudied. We inroduce he desired signals for process idenificaion and meaning of persisen exciing. Finally he perfec sampling frequency will be under invesigaion. 4.2 Signal Waveform In order o have daa wih high signal-o-noise raio, one needs o have as much signal power as possible. In pracice, on he oher hand, he es signal ampliudes are consrained because hey should no cause upse in normal process operaion and 2hey should no excie oo much nonlineariy for linear model idenificaion. Therefore, for a given signal power, he small ampliude is desirable. This propery can be expressed in erms of cres facor [4]. Denoe a signal wih zero mean as u, is cres facor is defined as C r max u lim N N 2 N u A good signal shape needs o have a small cres facor. Normally used signals are binary signals, filered whie noised and muliple sinusoids. Binary signals have he smalles cres facor which is. 4.3 Tes Signals for Model Idenificaion Four ypes of es signals will be discussed in he sequel: Pseudo random binary sequence PRBS Generalized binary noise GBN Filered whie noise or auoregressive moving average ARMA process. Sum of sinusoids. In pracice, mos ofen es signals are added o he inpus of he openloop process and /or a he se poins of he closed-loop sysem in order o move he process up and down around a working poin. Therefore, he mean value of es signals should be zero or close o zero.

53 Chaper 4: Idenificaion Tes Design and Daa Prereamen Pseudo Random Binary Sequence PRBS A pseudo random binary sequence PRBS is a wo sae signal which can be generaed by using a feedback shif regiser. Where n denoes he number of regisers or of saes. The regisers variables are fed wih or 0, and he iniial sae vecor is non zero. This is called pseudo random binary sequence PRBS because i is a deerminisic signal and is auocorrelaion funcion can resemble he auocorrelaion funcion of a whie random noise. The maximum possible period of a PRBS is M 2 n. Le u be a maximum lengh PRBS wih period M 2 n where n is he number of regisers [2]. This signal has wo ampliudes a when he regiser oupu is 0 and a when he regiser oupu is. Assume ha he sampling ime equals he clock period T CLK, hen he following can be shown: The mean value of u is a M The auocorrelaion funcion of u is The power specrum of u is 2 a τ 0, ± M, ± 2M R u τ a else K M 2π a 2 M Φ ω δ ω 2kπ / M 0 ω 2π u M k I can be seen ha when he clock period T CLK equals he sampling ime, and when M a large number is, a PRBS signal can approximae a whie noise signal wih zero mean. Such signals have maximum power for given ampliude, in oher words, hey have minimum cres facor.

54 Chaper 4: Idenificaion Tes Design and Daa Prereamen Generalized Binary Noise GBN This signal was proposed by Tullenken [7]. The moivaion was o generae a es signal ha is suiable for conrol-relevan idenificaion of indusrial processes. A GBN signal u akes wo values a and a.a each candidae swiching ime, i swiches according o he following rule: P[ u P[ u u ] p sw u ] p sw where p sw is swiching probabiliy Filered Whie Noise Colored Noise Because of he limied flexibiliy of power specrum of GBN and PRBS signals, i is no always possible o realize a desired disribuion of frequency conens using hese signals. This problem can be solved using filered whie noise. From [3] we know ha when a signal is generaed by filering a zero mean whie noise sequence e wih variance λ u F q e where F q is a sable filer, is power specrum is deermined by Φ ω F e u iω 2 λ Therefore, any disribuion of frequency conens can be approximaed arbirarily well by a proper design of he filer F q. he ampliude disribuion of whie noise is no crucial for generaing colored noise. The ampliude disribuion of a filered whie noise is coninuous which is beer for approximaing a nonlinear process. The cres facors of his kind of signals are larger han ha are binary signals Muliple Sinusoids Consider he sum of m sinusoids

55 Chaper 4: Idenificaion Tes Design and Daa Prereamen 54 m u a j sin ω j + ϕ j ; 0 < ω < ω2 < L < ωm < π j Is specrum is given as m a j Φu ω 2π j ω j 4 j 2 [ δ ω ω + δ ω + ] Assume ha he frequencies j 0,π and ha he number m is large enough. Then any desired disribuion of frequency conens can be realized by seing he ampliudes a properly. ω are uniformly disribued in [ ] The cres facor of a muli-sine can be large. Assuming ha all ampliudes 2 a are equal, he power of he signal is ma / 2. If all sinusoids are in j phase, he squared ampliude will be ma 2, and he cres facor is hus 2 m which will be large when m is large. In [8] here is several opimum soluions for ϕ j selecion. A simple soluion has been developed by Schroeder. The so-called Schroeder phase signal is creaed when he ampliudes are equal and he phases are spread as follows: j ϕ ϕ j arbirary ϕ j j π m ; 2 j m Anoher simple way is o choose phases randomly. 4.4 Persisen Exciaion In order o guaranee ha he esimaion algorihms have unique soluion, some minimum requiremen should be imposed on he es signals. This is called a persisen exciaion condiion. A discree-ime signal u is said o be persisenly exciing of order n if he following limi exiss: N Ru τ lim u u τ N N

56 Chaper 4: Idenificaion Tes Design and Daa Prereamen 55 And he marix Ru 0 Ru M Ru n R R u u 0 L O L Ru n M Ru 0 is nonsingular. The concep can also be exended o oher model form. I can be shown ha he idenifiably of an n h order linear process bq + L+ bnq G q + a q + L+ a q n n n is ha he es signal is persisenly exciing of order 2 n. The frequency domain inerpreaion of persisen exciaion of order n is ha he specrum of he signal in nonzero in a leas n frequencies in he inerval π, π. Based on his propery we give some commens on commonly used es signals. Pseudo random binary sequence PRBS. Consider a PRBS of period M. I can be shown ha he order of persisen exciing of he PRBS is M. When he clock ime of he PRBS signal is small, which is he case when we wan o approximae a whie noise, M is quie large which is enough for mos idenificaion mehods. Generalized binary noise GBN. Because a GBN has a coninuous specrum over he whole frequency range. Hence a filered whie noise is persisenly exciing for any finie order. Sum of sinusoids. I is well kwon ha is specrum has exacly 2 m lines in π, π. Thus for he idenificaion of an n h order process, a leas n sinusoids need o be injeced o he process. For good noise reducion he number of sinusoids should be much greaer han he process order. Therefore, he four ypes of es signals discussed above are all persisen exciing signals.

57 Chaper 4: Idenificaion Tes Design and Daa Prereamen Inroducing Dryer Sysem In his secion we are abou o inroduce he acual sysem used in his projec for idenificaion. We chose Dryer, an indusrial sysem for his purpose Descripion Conribued by: Process/Descripion: Sampling ime: Favoreel K.U.Leuven Deparmen Elecroechniek ESAT - SISTA Kardinaal Mercierlaan 94 B-300 Leuven Belgium wouer.favoreel@esa.kuleuven.ac.be Daa of he dryer pracicum a ESAT-SISTA. 0. sec Number of samples: Inpus: Oupus: 000 samples Volage of he heaing device Oupu air emperaure Dae of aking daa by he auhor: July 2006

58 Chaper 4: Idenificaion Tes Design and Daa Prereamen Wha is he Dryer The dryer sysem is one of he mos enduringly popular and imporan laboraory models for eaching conrol sysem engineering. The dryer sysem is widely used because i is very simple o undersand as a sysem, and ye he conrol echniques ha can be sudied i cover many imporan classical and modern design mehods. I has a very imporan propery, i is open loop sable. Figure 4.: Schemaic of dryer sysem Figure 4.2a: 3D image of dryer sysem

59 Chaper 4: Idenificaion Tes Design and Daa Prereamen 58 Figure 4.2b: The indusrial image of dryer sysem The sysem shown in figure. 4. is very simple; a power supply is used o supply he Heaer. So he inpu of he sysem is volage. We are abou o measure he air emperaure afer he heaer. So we use a special sensor o measure he emperaure ha is called emperaure sensor. The conrol job is o auomaically regulae emperaure of he oupu air on by changing he inpu volage of he sysem. This is a difficul conrol ask. In conrol echnology he sysem is open loop sable because he sysem oupu he oupu air emperaure increases wih limi for a fixed inpu volage of he heaing device. I is noiceable ha a dryer is no used in indusry plans individually. Bu i is a par of a plan. As shown in figure 4. he main pars of his plan are: A moor ha drives he air supply fan A heaer ha does he air heaing ask A dryer ha he heaed air wih heaing sysem is is inpu. Noice ha here is no necessiy o consider he we produc in as a new inpu because is assumed ha he sysem is SISO. The cyclone uni The dried produc ou ha is no necessiy o consider i as anoher oupu because he sysem is SISO. For saring idenificaion of his sysem, we represen he signal waveform of inpu and oupu daa and heir signal frequency conens. The inpu variable for dryer idenificaion is: Volage of heaing device u Also he oupu variable for dryer idenificaion is: Temperaure of oupu air y