W considr a wight runing schm summarid as follows: th ntwork is traind to a minimum of th larning rror, th wight that would caus th smallst incras in

Size: px

Start display at page:

Download "W considr a wight runing schm summarid as follows: th ntwork is traind to a minimum of th larning rror, th wight that would caus th smallst incras in"

Gyles Woods
6 years ago
Views:

1 Univrsal Distribution of Salincis for Pruning in Layrd Nural Ntworks J. Gorodkin y, L. K. Hansn, B. Lautru and S. A. Solla x CONNECT, Th Nils Bohr Institut, Blgdamsvj 7, Conhagn, Dnmark. lautru@connct.nbi.dk, solla@nbi.dk y Cntr for Biological Squnc Analysis, Tchnical Univrsity of Dnmark, Building 6, 8 Lyngby, Dnmark. gorodkin@cbs.dtu.dk CONNECT, Dartmnt of Mathmatical Modlling, Tchnical Univrsity of Dnmark, Building 35, 8 Lyngby, Dnmark. lkhansn@i.dtu.dk Abstract A bttr undrstanding of runing mthods basd on a ranking of wights according to thir salincy in a traind ntwork rquirs furthr information on th statistical rortis of such salincis. W focus on two-layr ntworks with ithr a linar or nonlinar outut unit, and obtain analytic xrssions for th distribution of salincis and thir logarithms. Our rsults rval unxctd univrsal rortis of th log-salincy distribution and suggst a novl algorithm for salincy-basd wight ranking that avoids th numrical cost of scond drivativ valuations. Introduction Th roblm of survisd larning in layrd nural ntworks is a two stag rocss. Achoic of architctur lads to th imlicit dnition of an associatd aramtr sac f ~wg, which rrsnts th nsmbl of wights whos valus nd to b dtrmind in ordr to fully scify th ntwork. This aramtr sac is thn sarchd so as to idntify scic aramtr valus ~w. Th goal is to obtain a ntwork with low gnraliation rror E G, a quantity that masurs th dirnc btwn th inut-outut ma imlmntd by th ntwork and th targt ma. Th training of fd-forward ntworks is usually formulatd as an otimiation roblm: valus for th aramtrs ~w ar chosn so as to minimi a larning rror E L, dnd as a sum ovr a st of training xamls givn in th form of inut-outut airs. Th xtnt to which low larning rror rsults in low gnraliation rror is controlld by th rior choic of ntwork architctur th ossibility ofusing th information rovidd by thdata to guid this choic has bn xlord in a varity of larning algorithms. Hr w focus on th family of runing algorithms, basd on th limination of rdundant wights and/or nurons during th training rocss. A varity of such mthods has bn introducd in rcnt yars, som basd on th rmoval of nurons 3 4 and som on th rmoval of individual wights Th goal is in ithr cas to control th si of th ntwork so as to obtain th smallst ossibl ntwork comatibl with larning th training st. Caacity argumnts indicat that imrovd gnraliation should rsult from this rduction in ntwork si for numrical xrimnts that conrm this rdiction s for xaml. Traind ntworks of minimal si that imlmnt a targt ma ar also of intrst as a otntial tool for comaring th intrinsic comlxity of dirnt tasks, and oftn rovid an intrrtabl rndition of th comutational stratgy through which th ma is imlmntabl in a nural ntwork rrsntation. x Prmannt addrss: Dartmnt ofphysiology and Institut for Nuroscinc, Northwstrn Univrsity Mdical School, Chicago, IL 66 and Dartmnt of Physics and Astronomy, Northwstrn Univrsity, Evanston, IL 68, USA.

2 W considr a wight runing schm summarid as follows: th ntwork is traind to a minimum of th larning rror, th wight that would caus th smallst incras in larning rror if rmovd is idntid and rmovd, and th rsulting smallr ntwork is rtraind to rach a nw minimum of th larning rror W follow th aroach of L Cun t al. 5, who roos a scond ordr stimat of th salincy of ach individual wight, dnd as th incras in larning rror that would rsult from its rmoval. Our goal is to charactri th xctd distribution of salincis ovr th wights of a two-layr ntwork, an architctur that has bn shown to rovid a univrsal aroximator for th imlmntation of functions from an N-dimnsional inut sac onto a scalar outut. 3 4 Gnral assumtions about th statistical rortis of th inut data and th wights of th traind ntwork allow us to calculat th distribution of th logarithm of th salincis, to nd an unxctd rsult: that th distribution is univrsal xct for a translation that contains all dndnc on th training data. W rsnt th analytic drivation of this rsult for two fundamntal tys of two-layr ntworks 5 : on with a linar outut unit and traind through a quadratic rror function, and anothr on with a sigmoidal outut unit and traind through a logarithmic rror function of th Kullback-Liblr ty. Rsults for a singl-layr linar ntwork ar includd for comarison. Numrical simulations ar usd to illustrat th validity of our assumtions about th statistical distribution of th wights in a traind ntwork, and to vrify our rdictions about th univrsal form of th salincy distribution. An intriguing consqunc of our calculations is a novl runing algorithm that artially justis and asily xtnds th simlst form of runing in which th salincy of a wight is assumd to b dtrmind only by its magnitud. Th salincis Th salincy of a wight inalayrd nural ntwork is dnd as th incras in larning rror that would rsult from its rmoval. L Cun t al. 5 hav roosd a scond ordr mthod to stimat this incras for ntworks that hav bn traind to a minimum of th larning rror. Th rst drivativ of th larning rror E L is ro at th minimum th dominant contribution to th salincy thus coms from th scond drivativs. Highr ordr contributions ar nglctd, and th assumtion that only on wight is rmovd at a tim is usd to nglct th o-diagonal trms in th matrix of scond drivativs, to obtain E k E k ~w w k () for th incras in larning rror associatd with th rmoval of th kth wight. Th salincy s k of th kth wight is dnd as this incras: s k EL k, and it is in this aroximation fully dtrmind by th magnitud of th wight w k and th corrsonding diagonal lmnt of th Hssian matrix of scond drivativs of th larning rror valuatd at th minimum: ~w = ~w. W ar intrstd in layrd ntworks that imlmnt mas from an N-dimnsional inut onto a scalar outut w thus rstrict ourslvs to two-layr ntworks with a singl hiddn layr of sigmoidal units connctd to on outut unit, which hav bn shown to b univrsal aroximators for th imlmntation of such functions. 3 4 W driv rsults for a linar outut unit and rsnt thir xtnsion to th cas of a nonlinar outut unit. Rsults for a singl-layr linar ntwork ar includd for comarison. Th outut O of a two-layr ntwork with a linar outut unit undr rsntation of inut

3 ~x is givn by: O = KX W i tanh N i= j= w ij x A j ; W () whr x = for all, w i is th thrshold of th ith hiddn unit, and W is th thrshold of th outut unit. Th quantity w ij rfrs to th wight from th jth inut unit to th ith hiddn unit and W i rfrs to th wight from th ith hiddn unit to th outut unit. Th numbr of inuts is N and th numbr of hiddn units is K. Th quivalnt xrssion for th cas of a nonlinar outut unit is: O =tanh K i= W i tanh N j= w ij x j A ; W A : (3) W concntrat hr on th salincis for th inut-to-hiddn wights th salincis of th hiddn-to-outut wights ar asily found following a similar rocdur. 6 For a linar outut P unit w us a quadratic rror function E L = = (y ;O ) to masur th distanc btwn targt oututs y and actual oututs O. Th corrsonding salincis tak thform : s ij = w ij W i = = h ; tanh (h i ) i x j (4) whr th notation h i = P N w j= ijx j is introducd to indicat th activation of th ith hiddn unit undr rsntation of th th xaml. For a nonlinar outut unit w us th Kullback- Liblr ntroy 7 as th rror function for outut units connd to th [ {, +] intrval through P a nonlinarity of th \tanh" ty, th rror is writtn as: E L = = h( + y ) log +y +O +(; y )log ;y ;O Th corrsonding salincis tak th form 6 : s ij = h i w ijw i ; tanh (H ) ih ; tanh (h i ) x j (5) whr H is th activation of th outut unit undr th rsntation of th th xaml, a quantity qual to th outut O of th ntwork with a linar outut unit, Equation (). It is th logarithm of ths salincis that w now valuat. 3 Th distribution of salincis and thir logarithms Our analysis of th statistical rortis of th salincis of a traind layrd ntwork is basd on siml assumtions about th statistical rortis of th corrsonding wights. A simultanous sign chang of all wights in ithr ntwork () or ntwork (3) lavs th outut invariant this symmtry lads to th assumtion that th robability distribution for individual wights is symmtric around ro and has ro man. W choos a Gaussian aroximation to th distribution of wights in th traind ntwork. This furthr assumtion is wll suortd by numrical vidnc obtaind through th training of layrd ntworks architcturally too larg for th imlmntation of th targt ma. Considr th sha of th surfac E L ( ~w) nar th minimum at ~w = ~w? for such a ntwork: th surfac rmains ssntially at in a signicant intrval around ~w? along thos dirctions corrsonding to rdundant aramtrs. Gradint-dscnt wight udats in th vicinity of th minimum thus rsult in stochastic variations, which ar almost uncorrlatd from on tim st to th nxt and add u to wights 3

4 that ar normally distributd. Numrical xrimnts suort this ictur and indicat that th Gaussian charactr of th wight distribution incrass with incrasing larning tim in th vicinity of th minimum. If th wight distribution has nit covarianc matrix, and th individual comonnts x j of th inut attrns ar indndntly drawn from a distribution P (x), th cntral limit thorm can b invokd to argu that th activation h i of th ith hiddn unit undr rsntation of th th attrn is a normally distributd random variabl with ro man and varianc h =(N +) wx, whr w is th varianc of th inut-to-hiddn wights and x stands for (N +) P N j=(x j ), indndnt of. Corrlations <h i x j > vanish for wights w ij that ar stochastic variabls with ro man. W now concntrat on th cas of a two-layr ntwork with a linar outut unit and comut th salincis in Equation (4) and th distribution of thir logarithms. W hav alid similar argumnts to analy th cas ofatwo-layr ntwork with a nonlinar outut unit th corrsonding rsults ar summarid as w go along. In th cas of a linar outut unit, considr th quantity Q = = [ ; tanh (h )] (x ) : (6) Th statistical rortis of this stochastic variabl ar indndnt ofi and j th uctuations of this singl stochastic variabl assign dirnt valus of Q to dirnt inut-to-hiddn wights. It is usful to writ Q = (=) P = v, and not that dirnt trms v ar uncorrlatd, as th statistical indndnc of th inut attrns guarants that both th x's and th h's ar uncorrlatd from xaml to xaml. It thn follows that hqi = hvi and Q =(=) v. Th rlativ varianc Q hqi! as! : (7) As th numbr of xamls grows th uctuations of Q around its man hqi bcom ngligibl in this limit th stochastic variabl Q bcoms slf-avraging and it can b rlacd by hqi. Th salincis in Equation (4) can thn b writtn as s ij = w ijw i hqi : (8) Th sam argumnt alis to th cas of a nonlinar outut unit it sucs to rdn Q = = [ ; tanh (H )][ ; tanh (h )] (x ) (9) to show that th salincis in Equation (5) can also b writtn in th form of Equation (8). Th Equation (8) thus rovids a comact xrssion for th inut-to-hiddn salincis s ij for two-layr ntworks with ithr a linar or nonlinar outut unit. Sinc th valu of hqi is indndnt ofi and j, all th information ndd to rank ths wights by ordr of incrasing salincy is containd in th roduct of th magnitud of two wights: w ij and W i. This rsult rvals a siml way of imlmnting OBD in a two-layr ntwork it imrovs uon siml runing schms basd on ranking wights according to only thir own magnitud whil it avoids th numrical cost associatd with th comutation of scond drivativs. Hiddn-to-outut salincis s i, labld only by th indx i of th corrsonding wight W i, ar asily shown to b 6 roortional to Wi, with a slf-avraging roortionality factor that 4

5 is indndnt ofi. Pruning according to a salincy-basd ranking thus rducs for th hiddnto-outut wights of a two-layr ntwork to a siml magnitud-basd runing schm. Th R valuation of th salincis in Equation (8) rquirs th comutation of th aramtr hqi = dq Q P (Q), which contains xlicit information about th data. In th cas of a linar outut unit, th rtinnt distribution is P (Q) = = dh dx P (h )P (x Q ; = h i ; tanh (h ) (x ) A () with P (h) = " # x ; (h ) h h : () Th corrsonding valu of hqi can b asily calculatd in th rgims h and h (s Andix A), to obtain (x)=(3 h ) and x =, rsctivly. A similar analysis yilds an xrssion for hqi in th cas of a nonlinar outut unit (s Andix A) it dnds not only on th varianc h for th activation of th hiddn units but also on th varianc H for th activation of th outut unit. Asymtotic rsults ar (x)=(3 h H ) in th larg varianc limit and x= in th small varianc limit. W now turn to calculating th distribution for th logarithm of th salincis th rason for focusing on th logarithm of th salincis rathr than th salincis thmslvs will bcom clar in th rocss. Considr th logarithm of th salincis in Equation (8) = log s = log w W hqi () whr indics ij idntifying a scic inut-to-hiddn wight ar omittd for simlicity. W ar intrstd in th distribution P () = with P W and P w givn by dw dw ( ; log w W hqi) P w (w) P W (W ) (3) P w (w) = x ;w =( w) w P W (W )= x ;W =( W ) W : (4) Th rsulting Gaussian intgrals can b rformd (s Andix B) to obtain 6 : P () = () K (()) (5) whr K is th modid Bssl function of th scond kind of ordr ro and s () = = hqi w W q x ; ; log w W hqi : (6) This rsult follows from Equation (8) and is thus valid for a two-layr ntwork with ithr a linar or nonlinar outut unit th charactr of th outut unit simly slcts th aroriat form for hqi, as discussd in andix A. Not that a chang in hqi only contributs in a translation along th -axis th distribution for th logarithm of th salincis has a sha which isunivrsal in that it is indndnt of th 5

6 P () Figur : Distribution of th logarithm of th salincis for th inut-to-hiddn wights of a two-layr ntwork with a linar outut unit. P () is shown for h, with w = W = : and x =, as is th cas for binary inut comonnts data and thus of th task that th ntwork is bing traind for! Th sha of th distribution is shown in Figur. Th distribution of salincis can b found through a similar rocdur, to obtain 6 : P (s) = (s)k ( (s)) (7) with (s) = [ w W hqi s] and (s) = ( W = w ) s= hqi. Not that a chang in hqi rsults in a chang of th actual sha of th salincy distribution. For comarison w quot hr th corrsonding rsults for a singl-layr linar ntwork with no hiddn units. A similar but simlr calculation 6 lads to rsults for th distribution of th salincis s and thir logarithms : P () = () x(; ()) (8) with () = s x q = x( ; log x w) (9) w and P (s) = (s) x(; (s)) () with (s) =[ w s] and (s) =(= w ) s=. For this siml architctur it is ncssary to considr th ossibility hwi 6= th corrsonding calculations can b carrid through 6, and th rsults rval a scaling of both hight and width of th distribution P (s) withhwi. 4 Numrical rsults W now rsnt numrical xrimnts on two-layr ntworks that justify our assumtion of ro-man normally distributd traind wights. 6 Th salincis s associatd with th various wights ar comutd for th traind ntwork following th original rscrition by L Cun t al. as summarid in Equation (). Th corrsonding distribution for th logarithms = log s of th salincis is found to b in good agrmnt with our thortical rdictions. Numrical xrimnts rortd hr ar for th contiguity roblm 8 9, in which strings of N binary comonnts ar classid into catgoris according to th numbr of contiguous clums of +'s rsnt in th attrn. Th roblm is simlid into a dichotomy by focusing on attrns that contain ithr only two such clums (to b mad onto an outut of {) or thr such clums (to b mad onto an outut of +). Hr w considr N = out of 6

7 w =: P (w) P () (a) (b) w w =:4 P (w) P () (c) (d) w w =:3 P (w) P () () (f) w Figur : Th wight distribution P (w) and log-salincy distribution P () for th inut-to-hiddn wights of a two-layr nural ntwork with N = inut units and K = 4 hiddn units traind on th contiguity roblm. Histograms basd on data for all 4 inut-to-hiddn wights ar shown itrations aftr raching th minimum of th larning rror (gurs a and b), 4 itrations aftr raching th minimum (gurs c and d), and itrations aftr raching th minimum (gurs and f). Th varianc w of th traind wight distribution is indicatd in ach cas. 7

8 a total of 4 ossibl attrns w only considr 79, of which 33 blong to th two-clum catgory and 46 to th thr-clum catgory. W randomly slct a training st of si = 79 and us it to train a two-layr nonlinar ntwork as dscribd by Equation (3), with K = 4 hiddn units and 48 aramtrs to b dtrmind by training. Larning rocds by gradint-dscnt on an rror function of th Kullback-Liblr ty. Targt valus ar chosn at :9 instad of to avoid saturation cts that consir against th validity of th scond ordr aroximation of Equation (). Numrical rsults ar shown in Figur for a training sssion in which wights wr initially drawn from a uniform distribution in th intrval [= N = N], with hwi = and w =:33. Th wight distribution P (w) and log-salincy distribution P () wr masurd uon raching th minimum of th larning rror (gurs a and b), aftr 4 additional itrations of th gradint-dscnt algorithm (gurs c and d), and again aftr additional itrations (gurs and f). No wights wr rund th histograms shown in Figur gathr th information containd in all 4 inut-to-hiddn wights. Similar rsults ar obtaind if th initial wights ar drawn from a ro-man Gaussian as oosd to a uniform distribution. 6 Numrical rsults shown in Figur 3 rval a linar corrlation btwn th initial and nal valus of w. nal nal (a) (b) init init Figur 3: Th varianc of th nal wight distribution as a function of th varianc of th initial wight distribution in two rgims: (a) init [ :] and (b) init [ ]. Data for th traind ntwork has bn takn itrations aftr raching th minimum of th larning rror. 5 Summary Siml assumtions about th statistical rortis of wights in a traind two-layr ntwork ar suortd by numrical vidnc and usd hr to rdict th xctd distribution of salincis. Ranking of wights according to thir ost-training salincy is a crucial ingrdint of runing algorithms such as Otimal Brain Damag for th cas of a two-layr ntwork our rsults rovid a siml algorithm for imlmnting this rscrition without th comutational cost associatd with scond-drivativ valuations. An unxctd outcom of our calculations is th nding of a univrsal sha for th distribution of th logarithms of th salincis. Th discovry of a task-indndnt rol for P () lavs on th qustion of how to utili th information containd in th data to formulat a stoing critrion for th runing rocss. 8

9 Acknowldgmnts W thank Andrs Krogh, Claus Svarr, and Ol Winthr for usful discussions. This rsarch was suortd by th Danish Rsarch Councils for th Natural and Tchnical Scincs through th Danish Comutational Nural Ntwork Cntr (CONNECT) and th Danish National Rsarch Foundation through th Danish Cntr for Biological Squnc Analysis. A Th man hqi R W now calculat hqi = dqqp(q), as ndd to valuat th salincis in Equation (8). In th cas of a linar outut unit: hqi = = = dq Q = = = = x with x = R = = dx P (x ) dx P (x ) dh dx P (h )P (x Q ; dx P (x ) dh ; h = = = h h dx P (x) x, as bfor. dh P (h ) dh P (h ) Z (x ) = = dq Q ; = h ; tanh (h)i h i ; tanh (h ) (x ) A = h i ; tanh (h ) (x ) A h ; tanh (h )i (x ) () dh P (h ) h i ; tanh (h ) This intgral is asily valuatd in two limits. For h, xand x[; and us ; tanh (h) =d tanh(h)=dh to obtain dh i d tanh(h) h ; tanh (h) = h dh h Z h h ] ;h =( h ), d h( ~ ; h ~ )= 4 () 3 h and hqi = 3 x + O ( h ) ;3 : (3) h For h, substitut ~ h = h= h, to obtain and d h ~ h i ; ~ h ; tanh ( h ~ Z h ) d ~ h ; ~ h = (4) hqi = x : (5) 9

10 In th cas of a nonlinar outut unit, lads to hqi = hqi = x dq Q ; = = dh dh dx P (H )P (h )P (x ) h ; tanh (H ) ih ; tanh (h ) i (x ) A (6) h ih dh dh P (H )P (h ) ; tanh (H ) ; tanh (h )i (7) whr both P (H) andp (h) ar normal distributions with ro man and variancs H and h, rsctivly. As bfor, ths intgrals ar asily valuatd in two limits. For H and h w obtain whil for H and h th rsult is again hqi = x (8) 3 H h hqi = x : (9) B Th distribution P () W now calculat th distribution P () for th logarithm =logs of th salincis, P () = dw dw ( ; log w W hqi)p w (w) P W (W ) : (3) In ordr to rform th intgral ovr w w not that th ros of th argumnt of th dlta function occur at w = w o,withw o = =(W hqi). Thus and ( ; log w W hqi) = w o [(w + w o)+(w ; w o )] (3) P () = dw w o P W (W ) dw P w (w)[(w + w o )+(w ; w o )] s s! dw = hqi jw j P W (W )P w (3) W hqi basd on th arity P w (w) =P w (;w). Sinc P W (W )isalsoanvn function, s s! dw P () = hqi W P W (W )P w W hqi s " = dw hqi w W W x ; w W hqi + W W!# :

11 Th chang of variabls W = = hqi( W = w )V lads to P () = () dv V x ; () V + V (33) with () = s hqi w W : (34) Th intgral is rwrittn in trms of u = log V, P () = () and idntid as a modid Bssl function of th scond kind: to obtain Rfrncs K (x) = du x [;() cosh(u)] (35) du ;x cosh(u) (36) P () = () K (()) : (37) [] D. MacKay, \Baysian introlation," Nural Comutation 4, 45{447 (99). [] Y. Chauvin, \A back{roagation algorithm with otimal us of hiddn units," in Advancs in Nural Information Procssing Systms (Dnvr, 988) d. D. S. Tourtky,. 59{ 56. [3] S. Ramachandran and L. Y. Pratt, \Information masur basd skltoniation," in Advancs in Nural Information Procssing Systms 4 (Dnvr, 99) ds. R. P. Limann, J. E. Moody and D. S. Tourtky,. 8{87. [4] M. C. Mor and P. Smolnsky, \Skltoniation: a tchniqu for trimming th fat from a ntwork via rlvanc assssmnt," in Advancs in Nural Information Procssing Systms (Dnvr, 988) d. D. S. Tourtky,. 7{5. [5] Y. L Cun, J. S. Dnkr and S. A. Solla, \Otimal brain damag," in Advancs in Nural Information Procssing Systms (Dnvr, 989) d. D. S. Tourtky,. 598{65. [6] B. Hassibi and D. Stork, \Scond ordr drivativs for ntwork runing: otimal brain surgon," in Advancs in Nural Information Procssing Systms 5 (Dnvr, 99) ds. S. J. Hanson, J. D. Cowan and C. L. Gils, [7] A. S. Wignd, D. E. Rumlhart and B. A. Hubrman, \Gnraliation by wight{ limination with alication to forcasting," in Advancs in Nural Information Procssing Systms 3 (Dnvr, 99) ds. R. P. Limann, J. E. Moody and D. S. Tourtky,. 875{ 88.

12 [8] H. H. Thodbrg, \Imroving gnraliation of nural ntworks through runing," IJNS, 37{36 (99). [9] V. Trs, R. Nunir and H. G. Zimmrmann, \Early brain damag," in Advancs in Nural Information Procssing Systms 9 (Dnvr, 996) ds. M. C. Mor, M. I. Jordan and T. Ptsch,. 669{675. [] V. Vanik, \Princils of risk minimiation for larning thory," in Advancs in Nural Information Procssing Systms 4 (Dnvr, 99) ds. J. E. Moody, S. J. Hanson and R. P. Limann,. 83{838. [] J. Gorodkin, L. K. Hansn, A. Krogh, C. Svarr and O. Winthr, \A quantitativ study of runing by otimal brain damag," IJNS 4, 59{69 (993). [] J. Gorodkin, A. Srnsn and O. Winthr, \Nural ntworks and cllular automata comlxity," Comlx Systms 7, {3 (993). [3] G. Cybnko, \Aroximation by surositions of a sigmoidal function," Mathmatics of Control, Signal, and Systms, 33{34 (989). [4] K. Hornik, M. Stinchcomb and H. Whit, \Multilayr fdforward ntworks ar univrsal aroximators," Nural Ntworks, 359{366 (989). [5] S. A. Solla, E. Lvin and M. Flishr, \Acclratd larning in layrd nural ntworks," Comlx Systms, 65{64 (988). [6] J. Gorodkin, \Architcturs and comlxity of layrd nural ntworks," Cand. Scint. Thsis, CONNECT, Th Nils Bohr Institut, 994. [7] T. M. Covr and J. A. Thomas, Elmnts of Information Thory, (Wily, USA, 99),. 8. [8] J. Dnkr, D. Schwart, B. Wittnr, S. A. Solla, R. Howard, L. Jackl and J. Hold, \Automatic larning, rul xtraction, and gnraliation," Comlx Systms, 877{9 (987). [9] S. A. Solla, \Larning and gnraliation in layrd nural ntworks: th contiguity roblm," in Nural Ntworks: from Modls to Alications (Paris, 988) ds. L. Prsonna and G. Dryfus,. 68{77.

The Application of Phase Type Distributions for Modelling Queuing Systems

The Application of Phase Type Distributions for Modelling Queuing Systems Th Alication of Phas Ty Distributions for Modlling Quuing Systms Eimutis VAAKEVICIUS Dartmnt of Mathmatical Rsarch in Systms Kaunas Univrsity of Tchnology Kaunas, T - 568, ithuania ABSTRACT Quuing modls