13. Artificial Neural Networks for Function Approximation

Transcription

1 Lecture 7 3. Artfcal eural etworks for Fucto Approxmato Motvato. A typcal cotrol desg process starts wth modelg, whch s bascally the process of costructg a mathematcal descrpto (such as a set of ODE-s) for the physcal system to be cotrolled. ote that more accurate models are ot always better. hey may requre uecessarly complex cotrol desg ad aalyss ad more demadg computato. he key here s to model essetal effects the system dyamcs the operatg rage of terest. I addto, a good model should also provde some characterzato of the model ucertates the so-called ukow ukows, whch ca be used for robust desg, adaptve desg, or merely smulato ad system testg, (such as Mote Carlo rus). Model ucertates are the dffereces betwee the model ad the real physcal process. Ucertates parameters are called parametrc, whle the others are called o-parametrc ucertates. Example 3. For the model of a cotrolled mass mx u the ucertaty m s parametrc, whle the eglected motor dyamcs, measuremet ose, sesor dyamcs represet the oparametrc ucertates. Example 3. Cosder the scalar model wth ucerta dyamcs kow. Suppose that f x x x x x Parametrc Ucertaty x f x u, where f o-parametrc x s ot that s suppose that the ukow fucto f x ca be approxmated by a lear combato of kow bass fuctos x ad ukow costat parameters. he approxmato error x s the o-parametrc ucertaty, whle the ukow costat parameters represet the parametrc ucertaty the system dyamcs. I order to characterze the latter, oe eeds to be able to fd a good set of bass fuctos x such that the approxmato error x becomes small o a compact x doma. Polyomals, Fourer seres expasos, sples ad feedforward eural etworks ca be used to approxmate fuctos o compact domas. I what follows, we show how to adapt to parametrc ucertates, whle mata robustess the presece of o-parametrc ucertates. Defto 3. 57

2 Artfcal Feedforward eural etworks are mult-put-mult-output systems composed of may ter-coected olear processg elemets (euros) operatg parallel. Fgures 3. ad 3. show sketches of two feedforward -s. 4 etwork Iput 5 etwork Output Iput ode 3 6 Output Frst Hdde Layer Secod Hdde Layer Fgure 3.: Feedforward eural etwork wth hdde layers ad 6 euros Fgure 3.: Feedforward eural etwork wth hdde layer ad 5 euros As see from these two examples, a artfcal feedforward eural etwork cossts of euros ad ther coectos. A block-dagram of a euro s show below. y x Fgure 3.3: Artfcal euro Block-Dagram 58

3 euros, the basc processg elemets of -s, have two ma compoets: a weghted summer a olear actvato fucto he actvato fuctos of terest are adal Bass Fuctos or rdge fuctos, (ofte called the sgmods). Defto 3. A adal Bass Fucto (BF) s defed as a Gaussa: xxc Wxxc xxc W xx, e e (3.) I (3.), x s the put, xc c s the ceter, ad W W 0 s a postve-defte symmetrc matrx of weghts. Most ofte we wll wrte x x x ad deote a BF whch s cetered at the th ceter., c to abbrevate emark 3. Other deftos of a BF are avalable. Ofte the lterature, a BF s defed as x x c W, where x x W x deotes the weghted Eucldea orm of a vector x. I addto, t s requred that W x be tegrable o., ad xdx 0 Bascally, ths partcular BF depeds oly o the weghted dstace r x xc W betwee ts curret put x ad the ceter x c. he Gaussa BF (3.) s a example of ths type of actvato fucto. Others clude: Multquadrcs r r c, c 0 Iverse multquadrcs r, c 0 r c Mcchell s heorem Let r be the Gaussa, the multquadrcs, or the verse multquadrcs x be a set of dstct pots. he the th matrx, whose, x x, s osgular. fucto. Let j elemet s j j terpolato emark 3. here s a large class of BF-s that s covered by Mcchell s theorem. he theorem provdes theoretcal bass for BF based fucto approxmato problems. I other words, usg a BF r ad a fte set of pots x, t s always 59

4 possble to approxmate a large class of fuctos that f x ˆ f x, for all x. such f x wth fˆ x xx Defto 3.3 A rdge fucto / sgmod s a olear fucto of the form: w x b (3.) where w deotes the vector of weghts, b s a scalar called the threshold, ad s s a olear fucto (ot ecessarly cotuous), defed wth the followg propertes: lm s s (3.3) lm s s he two most commo examples of a rdge fucto are: he logstc sgmod s (3.4) s e he hyperbolc taget s e s (3.5) s e A feedforward wth euros ts hdde layer s show Fgure 3.4. Hdde Layer of euros Iput x hreshold y Fgure 3.4: Sgle-hdde-layer feedforward wth euros m Formally speakg, a feedforward maps to, that s: m y x, x, y (3.6) Defto 3.4 A sgmodal feedforward s: Output Bas 60

5 x W V x b m where W s the matrx of the outer-layer weghts, x V x V x (3.7) s the vector of sgmods, V s the matrx of the er-layer syaptc weghts wth ts th m colum deoted by V, s the vector of thresholds, ad b deotes the bas vector. Defto 3.5 A feedforward BF s: x C x W x b b x x x C W m where b s the vector of weghts, receptve feld, W W 0 s the orm weghtg matrx, b x x x C x (3.8) s the ceter of the th m s the bas, ad s the so-called regressor vector, whose compoets are the bass actvato fuctos x x C fucto. ad the uty W emark 3.3 Ofte practcal applcatos, the symmetrc postve-defte matrx W (3.8) s chose to be dagoal ad the form: W,,, where represets the wdth of the th Gaussa fucto, that s: x e xc becomes the th compoet of the regressor vector x (3.8). Most ofte, the compoets of the regressor are costructed usg the sotropc Gaussa fucto: xc d max x e whose stadard devato (.e., wdth) s fxed accordg to the spread of the ceters C, s the umber of ceters, ad d max s the maxmum dstace betwee the chose ceters. I ths case, the stadard devato of all the sotropc Gaussa BF compoets s fxed at 6

6 dmax hs formula esures that the dvdual BF-s are ot too peaked or too flat. Both of these two extreme codtos should be avoded. Feedforward -s have bee show to be capable of approxmatg geerc classes of fuctos, cludg cotuous ad tegrable oes, o a compact doma ad to wth ay tolerace. hs property of feedforward -s s ofte referred to as the Uversal Approxmato property, whle the -s themselves are ofte called the uversal approxmators. wo related theorems are gve below. Uversal Approxmato heorem for Sgmodal -s, (G. Cybeko, 989) Ay cotuous fucto f x: ca be uformly approxmated by a sglehdde-layer wth a bouded mootoe-creasg cotuous actvato fucto ad o a compact doma X, that s: 0 WbV,,,, xx W V xb fx (3.9) emark 3.4 he uversal approxmato theorem exteds to the class of L fuctos o a compact doma. I that case, t s assumed that the actvato fucto s a bouded measurable sgmod ad the approxmato s uderstood terms of the L orm. ate of Approxmato heorem for Sgmodal -s, (. Barro, 993) Cosder a class of fuctos f x o for whch there s a Fourer represetato of the form x f x e f d for some complex-valued fucto f f x for whch f C f d s tegrable, ad defe he for every fucto f x wth C f fte, ad every, there exsts a sgmodal of the form (3.7), such that rcf f xx f x x dx L emark 3.5 x r 6

7 d Fuctos wth C f fte are cotuously dfferetable o. Moreover, the approxmato error s measured by the tegrated squared error, ( L orm), o the ball of radus r. Uversal Approxmato heorem for BF -s, (Park ad Sadberg, 99) Let x: be a tegrable bouded cotuous fucto ad assume that he for ay cotuous fucto euros, a set of ceters ˆ such that x dx 0 f x ad ay 0 there s a BF wth C, ad a commo wdth 0 x C f x x x f xx f x x dx O L x r Comparso of sgmodal ad BF -s BF ad sgmodal -s are uversal approxmators. BF depeds the Eucldea dstaces betwee the put vector x ad the ceters C. Meawhle, sgmodal -s deped o the er product of the put vector x wth the syaptc weght vectors V ad based by. Sgmodal -s provde O rate of approxmato whch does ot explctly deped o the dmeso of x. O the other had, the rate of approxmato for the BF -s s of order O ad, cosequetly t decreases expoetally as the dmeso of the put vector x creases. hs pheomeo s called the Curse of Dmesoalty, (due to. Bellma). A BF has local support whle a sgmod does ot. he local support mples learg ad adaptato ablty of BF -s. Sgmodal -s adapt but do t lear. Wth specfc referece to -s cotrol, t s ther ablty to represet olear mappgs, ad hece to model olear systems, whch s the feature to be most readly exploted the sythess of olear cotrollers. eadg materal / efereces: 63

8 ) F. Scarsell, C. so, Uversal approxmato usg feedforward eural etworks: A survey of some exstg methods, ad some ew results, eural etworks, vol., o., pp. 5-37, 998. ) K.J. Hut, D. Sbarbaro,. Zbkowsk, P.J. Gawthrop, eural etworks for Cotrol Systems A Survey, Automatca, vol. 8, o. 6., pp. 083-, 99. 3) G. Cybeko, Approxmato by superposto of a sgmodal fucto, Math. Cotrol Sgals Systems, vol., pp , ) J. Park, I.W. Sadberg, Uversal approxmato usg radal-bass-fucto etworks, eural Computato, vol. 3, o., pp ) C. Mcchell, Iterpolato of scattered data: Dstace matrces ad codtoally postve defte fuctos, Costructve Approxmato, vol., pp. -,