Solvng Nonlnear Dfferental Equatons by a Neural Network Method Luce P. Aarts and Peter Van der Veer Delft Unversty of Technology, Faculty of Cvlengneerng and Geoscences, Secton of Cvlengneerng Informatcs, Stevnweg, 68 CN Delft, The Netherlands l.aarts@ctg.tudelft.nl, p.vdveer@ct.tudelft.nl Abstract. In ths paper we demonstrate a neural network method to solve nonlnear dfferental equatons and ts boundary condtons. The dea of our method s to ncorporate knowledge about the dfferental equaton and ts boundary condtons nto neural networks and the tranng sets. Hereby we obtan specfcally structured neural networks. To solve the nonlnear dfferental equaton and ts boundary condtons we have to tran all obtaned neural networks smultaneously. Ths s realzed by applyng an evolutonary algorthm. Introducton In ths paper we present a neural network method to solve a nonlnear dfferental equaton and ts boundary condtons. In [] we have alrea demonstrated how we could solve lnear dfferental and lnear partal dfferental equatons by our neural network method. In [] we showed how to use our neural network method to solve systems of coupled frst order lnear dfferental equatons. In ths paper we demonstrate how we ncorporate knowledge about the nonlnear dfferental equaton and ts boundary condtons nto the structure of the neural networks and the tranng sets. Tranng the obtaned neural networks smultaneously now solves the nonlnear dfferental equaton and ts boundary condtons. Snce several of the obtaned neural networks are specfcally structured, the tranng of the networks s accomplshed by applyng an evolutonary algorthm. An evolutonary algorthm tres to fnd the mnmum of a gven functon. Normally one deals wth an evolutonary algorthm workng on a sngle populaton,.e. a set of elements of the soluton space. We however use an evolutonary algorthm workng on multple subpopulatons to obtan results more effcently. At last we graphcally llustrate the obtaned results of solvng the nonlnear dfferental equaton and ts boundary condtons by our neural network method. V.N. Alexandrov et al. (Eds.): ICCS, LNCS 74, pp. 8 89,. Sprnger-Verlag Berln Hedelberg
8 L.P. Aarts and P. Van der Veer Problem Statement Many of the general laws of nature, lke n physcs, chemstry, bology and astronomy, fnd ther most natural expresson n the language of dfferental equatons. Applcatons also abound n mathematcs tself, especally n geometry, and n engneerng, economcs, and many other felds of appled scence. In [] one derves the followng nonlnear dfferental equaton and ts boundary condtons for the descrpton of the problem of fndng the shape assumed by a flexble chan suspended between two ponts and hangng under ts own weght. Further the y -axs pass through the lowest pont of the chan. d y = +, () y ( ) =, () ( ) = Here the lnear densty of the chan s assumed to be a constant value. In [] the analytcal soluton s derved for system (), () and (3) and s gven by y x) = x -x ( e + e ) (. In ths paper we consder the system (), () and (3) on the nterval [-, ]. x. (3) (4) 3 Outlne of the Method By knowng the analytcal soluton of (), () and (3), we may assume that y(x) and ts frst two dervatves are contnuous mappngs. Further we defne the logsgmod functon f as f ( x) + exp( -x) =. By results n [8] we can fnd such real values of a, w and b that for a certan natural number m the followng mappngs (5)
Solvng Nonlnear Dfferental Equatons by a Neural Network Method 83 m j, ( x ) = a f ( w x + b ) = (6) dj ( x) = m = a w df ( w x + b ), (7) m d j d f ( x) = a w = ( w x + b ), (8) respectvely approxmate y x), ( and d y arbtrarly well. The networks represented by (6), (7) and (8) have one hdden layer contanng m neurons and a lnear output layer. Further we defne the DE-neural network of system (), () and (3) as the not fully connected neural network whch s constructed as follows. The output of the g ( x) = x as network represented by (7) s the nput of a layer havng the functon transfer functon. The layer contans one neuron and has no bas. The connecton weght between the network represented by (7) and the layer s. The output of the layer s the nput of a layer wth the functon h ( x) = x as transfer functon. The layer contans one neuron and has a bas wth value. The connecton weght between the two layers s. x d j + - dj g h g ( x) = x, h( x) = x Fg.. The DE-neural network for system (), () and (3)
84 L.P. Aarts and P. Van der Veer The output of the last layer s subtracted from the output of the network represented by (8). A sketch of the DE-neural network of system (), () and (3) s gven n Fg.. Snce the learnablty of neural networks to smultaneously approxmate a gven functon and ts unknown dervatves s made plausble n [5], we observe the followng. Assume that we have found such values of the weghts that the networks represented by (6), (7) and (8) respectvely approxmate y (x) and ts frst two dervatves arbtrarly well on a certan nterval. By consderng the nonlnear dfferental equaton gven by () t then follows that the DE-neural network must have a number arbtrarly close to zero as output for any nput of the nterval. In [6] t s alrea stated that any network sutably traned to approxmate a mappng satsfyng some nonlnear partal dfferental equatons wll have an output functon that tself approxmately satsfes the partal dfferental equatons by vrtue of ts approxmaton of the mappng s dervatves. Further the network represented by (6) must have for nput x = an output arbtrarly close to one and the network represented by (7) must gve for the same nput an output arbtrarly close to zero. The dea of our neural network method s based on the observaton that f we want to fulfl a system lke (), () and (3) the DE-neural network should have zero as output for any nput of the consdered nterval[ -,]. Therefore we tran the DE-neural network to have zero as output for any nput of a tranng set wth nputs x [-, ]. Further we have the followng restrctons on the values of the weghts. The neural network represented by (6) must be traned to have one as output for nput x = and for the same nput the neural network represented by (7) must be traned to have zero as output. If the tranng of the three networks has well succeeded the mappng j and ts frst two dervatves should respectvely approxmate y and ts frst two dervatves. Note that we stll have to choose the number of neurons n the hdden layers of the networks represented by (6), (7) and (8),.e. the natural number m by tral and error. The three neural networks have to be traned smultaneously as a consequence of ther nter-relatonshps. It s a specfc pont of attenton how to adjust the values of the weghts of the DE-neural network. The weghts of the DE-neural network are hghly correlated. In [] t s stated that an evolutonary algorthm makes t easer to generate neural networks wth some specal characterstcs. Therefore we use an evolutonary algorthm to adjust smultaneously the weghts of the three neural networks. Before we descrbe how we manage ths, we gve a short outlne of what an evolutonary algorthm s. 4 Evolutonary Algorthms wth Multple Subpopulatons Evolutonary algorthms work on a set of elements of the soluton space of the functon we would lke to mnmze. The set of elements s called a populaton and the elements of the set are called ndvduals. The man dea of evolutonary algorthms s that they explore all regons of the soluton space and explot promsng areas through applyng recombnaton, mutaton, selecton and renserton operatons to the ndvdu-
Solvng Nonlnear Dfferental Equatons by a Neural Network Method 85 als of a populaton. In ths way one hopefully fnds the mnmum of the gven functon. Every tme all procedures are appled to a populaton, a new generaton s created. Normally one works wth a sngle populaton. In [9] Pohlhem however states that results are more effcently obtaned when we are workng wth multple subpopulatons nstead of just a sngle populaton. Every subpopulaton evolves over a few generatons solated (lke wth a sngle populaton evolutonary algorthm) before one or more ndvduals are exchanged between the subpopulatons. To apply an evolutonary algorthm n our case, we defne e, e and e 3 as the means of the sum-ofsquares error on the tranng sets of respectvely the DE-neural network, the network represented by (6) and the network gven by (7). Here we mean by the mean of the sum-of-squares error on the tranng set of a certan network, that the square of the dfference between the target and the output of the network s summed for all nputs and that ths sum s dvded by the number of nputs. To smultaneously tran the DE- dj we mnmze the expres- neural network and the networks represented by j and son e + e e3 +, (9) by usng an evolutonary algorthm. Here equaton (9) s a functon of the varables a, w and b. 5 Results In ths secton we show the results of applyng our neural network method to the system (), () and (3). Some practcal aspects of tranng neural networks that are well known n lterature also hold for our method. In e.g. [3] and [7], t s stated that f we want to approxmate an arbtrary mappng wth a neural network represented by (6), t s advantageous for the tranng of the neural networks to scale the nputs and targets so that they fall wthn a specfed range. In ths way we can mpose fxed lmts on the values of the weghts. Ths prevents that we get stuck too far away from a good optmum durng the tranng process. By tranng the networks wth scaled data all weghts can reman n small predctable ranges. In [] more can be found about scalng the varable where the unknown of the dfferental equaton depend on and scalng the functon values of the unknown of the dfferental equaton, to mprove the tranng process of the neural networks. To make sure that n our case the weghts of the networks can reman n small predctable ranges, we scale the functon values of the unknown of the nonlnear dfferental equaton. Snce we normally do not know much about the functon values of the unknown we have to guess a good scalng of the functon values of the unknown. For solvng the system (), () and (3) on the consdered nterval [-,] we decde to scale y n the followng way:
86 L.P. Aarts and P. Van der Veer y y M =. () Hereby the system (), () and (3) becomes d y M = + 4 M M, () M ( ) = y, () M ( ) =. We now solve the system (), () and (3) by applyng the neural network method descrbed n Sect. 3. A sketch of the DE-neural network for system (), () and (3) s gven n Fg.. (3) x d j + - dj g h 4 g ( x) = x, h( x) = x Fg.. The DE-neural network for system (), () and (3) We mplemented the neural networks by usng the Neural Network Toolbox of Matlab 5.3 ([4]). Further we used the evolutonary algorthm mplemented n the GEATbx toolbox ([9]). When we work wth the evolutonary algorthms mplemented n the GEATbx toolbox, the values of the unknown varables a, w and b have to fall wthn a specfed range. By experments we notced that we obtan good results f we restrct the values of the varables a, w and b to the nterval [- 5,5]. The DE- x -,-.9,..,.9, and neural network s traned by a tranng set wth nputs { }
Solvng Nonlnear Dfferental Equatons by a Neural Network Method 87 the correspondng targets of all nputs are zero. Further we have to tran the neural network represented by j to have one as output for nput x = and for the same dj nput the neural network represented by must have zero as output. The number of neurons n the hdden layer of the neural networks represented by (6), (7) and (8) s taken equal to 6. Therefore the number of varables whch have to be adapted s equal to 8. After runnng the chosen evolutonary algorthm for 5 generatons wth 6 ndvduals dvded over 8 subpopulatons we take the set x {-, -.95,.9,..,.95,} as nput to compute the output of the neural networks dj d j represented by j, and. We also compute the analytcal soluton of x -,-.95,.9,..,.95,. By (), () and (3) and ts frst two dervatves for { } comparng the results we conclude that the approxmaton of y and ts frst two dervatves by respectvely j and ts frst two dervatves are very good. Both the neural network method soluton of (), () and (3) and ts frst two dervatves as the analytcal soluton of (), () and (3) and ts frst two dervatves are graphcally llustrated n Fg. 3 and Fg. 4. The errors between the neural network method soluton of (), () and (3) and ts frst two dervatves on the one hand and the analytcal soluton of (), () and (3) and ts frst two dervatves on the other hand are graphcally llustrated n Fg. 5. The dfference between the target of the DE-neural network of the system (), () and (3),.e. zero for any nput x of the set {-, -.95,.9,..,.95,} and ts actual output s also llustrated n Fg. 5. Consderng Fg. 5, we can conclude that the approxmatons of the soluton of (), () and (3) and ts frst dervatve are somewhat better than the approxmaton of the second dervatve of the soluton of (), () and (3). Snce we are however n most numercal solvng methods for dfferental equatons nterested n the approxmaton of just the soluton tself, our results are really satsfyng. 6 Concludng Remarks In ths paper we used our neural network method to solve a system consstng of a nonlnear dfferental equaton and ts two boundary condtons. The obtaned results are very promsng and the concept of the method appears to be feasble. In further research more attenton wll be pad to practcal aspects lke the choce of the evolutonary algorthm that s used to tran the networks smultaneously. We wll also do more extensve experments on scalng ssues n practcal stuatons, especally the scalng of the varable where the unknown of the dfferental equaton depends on.
88 L.P. Aarts and P. Van der Veer References. Aarts, L.P., Van der Veer, P.: Neural Network Method for Solvng Partal Dfferental Equatons. Accepted for publcaton n Neural Processng Letters (?). Aarts, L.P., Van der Veer, P.: Solvng Systems of Frst Order Lnear Dfferental Equatons by a Neural Network Method. Submtted for publcaton December 3. Bshop, C.M.: Neural Networks for Pattern Recognton. Clarendon Press, Oxford (995) 4. Demuth, H., Beale, M.: Neural Networks Toolbox For Use wth Matlab, User s Gude Verson 3. The Math Works, Inc., Natck Ma (998) 5. Gallant, R.A., Whte H.: On Learnng the Dervatves of an Unknown Mappng Wth Multlayer Feedforward Networks. Neural Networks 5 (99) 9-38 6. Hornk, K., Stnchcombe, M., Whte, H.: Unversal Approxmaton of an Unknown Mappng and Its Dervatves Usng Multlayer Feedforward Networks. Neural Networks 3 (99) 55-56 7. Masters, T.: Practcal Neural Networks Recpes n C++. Academc Press, Inc. San Dego (993) 8. L, X.: Smultaneous approxmatons of multvarate functons and ther dervatves by neural networks wth one hdden layer. Neurocomputng (996), 37-343 9. Pohlhem, H., Documentaton for Genetc and Evolutonary Algorthm Toolbox for use wth Matlab (GEATbx): verson.9, more nformaton on http://www.geatbx.com (999). Smmons, G.F.: Dfferental equatons wth applcatons and hstorcal notes. nd ed. McGraw-Hll, Inc., New York (99). Yao, X.: Evolvng Artfcal Neural Networks. Proceedngs of the IEEE 87(9) (999) 43 447 4 y(x) 3.5 3 o=neural network *=analytcal -> y.5.5 - -.5.5.5 Fg.3. The soluton of system (), () and (3)
Solvng Nonlnear Dfferental Equatons by a Neural Network Method 89 4 /(x) -> / o=neural network *=analytcal -> d y/ - - -.5.5.5 4 d y/ (x) 3 o=neural network *=analytcal - -.5.5.5 Fg.4. The frst two dervatves of the soluton of system (), () and (3). analytcal - nn y(x) analytcal - nn /(x) -> error -. -. -.3 -> error -. -. -.3 -.4 - analytcal - nn d y/ (x).5 -.4 - output DE-neural network (), (), (3)..5 -> error -.5 -. -> output -.5 -.5 - -. - Fg. 5. The errors between the analytcal solutons of (), () and (3) and ts frst two dervatves on the one hand and the neural network method soluton of (), () and (3) and ts frst two dervatves on the other hand. Also the output of the DE-neural network of (), () and (3) s llustrated