Levenberg-Marquardt-based OBS Algorithm using Adaptive Pruning Interval for System Identification with Dynamic Neural Networks

Transcription

1 Proceedins of the 29 IEEE International Conference on Systems, Man, and Cybernetics San Antonio, X, USA - October 29 evenber-marquardt-based OBS Alorithm usin Adaptive Prunin Interval for System Identification with ynamic Neural Networs Christian Endisch, Peter Stolze, Peter Endisch, Christoph Hacl and Ralph Kennel Institute for Electrical rive Systems and Power Electronics echnische Universität München 8333 München, Germany Abstract his paper presents a prunin alorithm usin adaptive prunin interval for system identification with eneral dynamic neural networs (GNN). GNNs are artificial neural networs with internal dynamics. All layers have feedbac connections with time delays to the same and to all other layers. he parameters are trained with the evenber-marquardt (M) optimization alorithm. herefore the Jacobian matrix is required. he Jacobian is calculated by real time recurrent learnin (RR). As both M and OBS need Hessian information, computin time can be saved, if OBS uses the scaled inverse Hessian already calculated for the M alorithm. his paper discusses the effect of usin the scaled Hessian instead of the real Hessian in the OBS prunin approach. In addition to that an adaptive prunin interval is introduced. ue to prunin the structure of the identification model is chaned drastically. So the parameter optimization tas between the prunin steps becomes more or less complex. o uarantee that the parameter optimization alorithm has enouh time to cope with the structural chanes in the GNN-model, it is suested to adapt the prunin interval durin the identification process. he proposed alorithm is verified simulatively for two standard identification examples. Index erms System identification, dynamic neural networ, recurrent neural networ, GNN, optimization, evenber- Marquardt, real time recurrent learnin, networ prunin, OBS. I. INROUCION A crucial tas in system identification with recurrent neural networs is the selection of an appropriate networ architecture. It is very difficult to decide a priori which architecture and size are adequate for an identification tas. here are no eneral rules for choosin the structure before identification. A common approach is to try different confiurations until one is found that wors well. his approach can be very time consumin if many topoloies have to be tested. In this paper we start with an oversized eneral dynamic neural networ (GNN, see Fi. ) and remove unnecessary parts. In GNN all layers have feedbac connections with many time delays. he output depends not only on the current input, but also on previous inputs and previous states of the networ. urin the identification process the networ architecture is reduced to find a model for the plant as simple as possible. For architecture reduction in static neural networs several prunin alorithms are nown [],[6],[],[6]. wo well nown methods in feedforward neural networs are optimal brain damae (OB) [] and optimal brain sureon (OBS) [6]. Both methods are based on weiht ranin due to the saliency, which is defined as the chane in the output error usin Hessian information. he OB method calculates the saliency only with the pivot elements of the Hessian without retrainin after the prunin step. he OBS uses the complete Hessian information to calculate the saliency, which is rearded as a continuation of the OB method. Contrary to OB the OBS alorithm includes a retrainin part for the remainin weihts. In [3] it is shown that networ prunin with OBS can also be applied to dynamic neural networs. In OBS prunin the calculation of the inverse Hessian causes a reat amount of computation. o overcome this disadvantae of OBS it is suested to use the OBS prunin alorithm in conjunction with the M trainin alorithm [3],[2]. hus OBS ets the inverse Hessian for free. he structure of the GNN is chaned drastically durin the prunin and identification process. Because of this the influence of one prunin step on the identification system can be quite different. he deletion of one weiht in a very lare neural networ usually does not influence the identification system very much and the error after prunin can be reduced quite fast, whereas the deletion of one weiht in a small neural networ can influence the system extremely leadin to hue model errors. In the latter case the identification system needs more time to cope with the structural chanes in the GNN. In this paper the time between two prunin steps, the so-called prunin interval, is adapted online. he interval adaption maes use of a scalin alorithm by evaluatin the mean error between successive prunin steps. he next section presents the recurrent neural networ used in this paper. Administration matrices are introduced to manae the prunin process. Section III deals with the parameter optimization method used throuhout this paper. In section IV the M-based OBS alorithm is discussed and the adaptive prunin interval approach is presented. Identification examples are shown in section V. Finally, in section VI we summarize the results. II. GENER YNAMIC NEUR NEWORK (GNN) e Jesus described in his doctoral thesis [9] a broad class of dynamic networs, he called the framewor layered diital /9/$ IEEE 352

2 dynamic networ (N). he sophisticated formulations and notations of the N allow an efficient computation of the Jacobian matrix usin RR [4]. herefore we follow these conventions suested by e Jesus. In [7]-[] the optimal networ topoloy is assumed to be nown. In this paper the networ topoloy is unnown and so we choose an oversized networ for identification. In GNN all feedbac connections exist with a complete tapped delay line (from a first-order time delay element z up to the maximum order time delay element z dmax ). he output of a tapped delay line () is a vector containin delayed values of the input. Also the networ inputs have a. Fi. shows a three-layer GNN. he simulation equation for layer m is n m (t) = m,l (d) a l (t d)+ W l f m d m,l l I m d I m,l IW m,l (d) p l (t d)+b m () n m (t) is the summation output of layer m, p l (t) is the l- th input to the networ, IWm,l is the input weiht matrix between input l and layer m, W m,l is the layer weiht matrix between layer l and layer m, b m is the bias vector of layer m, m,l is the set of all delays in the tapped delay line between layer l and layer m, I m,l is the set of all input delays in the tapped delay line between input l and layer m, I m is the set of indices of input vectors that connect to layer m, f m is the set of indices of layers that directly connect forward to layer m. he output of layer m is a m (t) =f m (n m (t)) (2) where f m ( ) are nonlinear activation functions. In this paper we use tanh-functions in the hidden layers and linear activation functions in the output layer. At each point of time the equations () and (2) are iterated forward throuh the layers. ime is incremented from t =to t = Q. (See[9]fora full description of the notation used here). In Fi. below the matrix-boxes and below the arrows the dimensions are shown. R m and S m respectively indicate the dimension of the input and the number of neurons in layer m. ŷ is the output of the GNN. urin the identification process the optimal networ architecture should be found. Administration matrices show which weihts are valid or not. A. Administration Matrices he layer weiht administration matrices m,l (d) have the same dimensions as the layer weiht matrices W m,l (d) of the GNN, the input weiht administration matrices AI m,l (d) have the same dimensions as the input weiht matrices IW m,l (d) and the bias weiht administration vectors Ab m have the same dimensions as the bias weiht vectors b m.he elements of the administration matrices can have the boolean values or, indicatin if a weiht is valid or not. If e.. the layer weiht lw m,l,i (d) =[W m,l (d)],i from neuron i of layer l to neuron of layer m with a dth-order time-delay is valid, then [ m,l (d) ],i = αlm,l,i (d) =. If the element in the administration matrix equals to zero the correspondin weiht has no influence on the GNN. With these definitions the th output of layer m can be computed by n m (t) = l f d m m,l + l I m d I m,l + b m αbm S l i= R l i= lw m,l,i (d) αlm,l,i (d) al i iw m,l,i (d) αim,l,i (d) pl i (t d) (t d) a m (t) =f m (nm (t)) (3) where S l is the number of neurons in layer l and R l is the dimension of the lth input. able I shows an example of administration matrices for a three-layer GNN with 3 neurons in each hidden layer and d max = 2 at the beinnin of a prunin process. Only the weiht l 2,2,3 () from neuron 3 of layer 2 to neuron of the same layer with first-order timedelay is deleted, because αl 2,2,3 () =. All other weihts are valid. ABE I EXAMPE OF AMINISRAION MARICES FOR A HREE-AYER GNN, 3 NEURONS IN HE HIEN AYERS AN d max =2 ayer 2 3 Administration Matrices AI, () AI, (2), (), (2),2 (),2 (2),3 (),3 (2)Ab 2, () 2,2 () 2,2 (2) 2,3 () 2,3 ()Ab 2 3,2 () 3,3 () 3,3 (2)Ab 3 B. Implementation For the simulations throuhout this paper the raphical prorammin lanuae Simulin (Matlab) was used. GNN, Jacobian calculation, optimization alorithm and prunin were implemented as S-function in C. III. PARAMEER OPIMIZAION First of all a quantitative measure of the networ performance has to be defined. In the followin we use the squared error E(w )= Q 2 (y q ŷ q (w )) (y q ŷ q (w )) q= (4) = Q 2 e q (w ) e q (w ) q= 353

3 p (t) a (t) a 2 (t) a 3 (t) =ŷ(t) f 2, f 2 3,2 IW, R Σ S Σ S 2 Σ f W 3 W S 3 S R S 2 S S 3 S 2 b b 2 b 3 z S S 2 S 3 W,3 S S 3 W,2 S S 2 2,3 3,3 W W S 2 S 3 S 3 S 3 W 2,2 S 2 S 2 z 2 W 2,2 () W 2,2 (2) Σ W, S S z dmax W 2,2 (dmax) ayer ayer 2 ayer 3 Fi.. hree-layer GNN (two hidden layers) where q denotes one pattern in the trainin set, y q and ŷ q (w ) are, respectively, the desired taret and actual model output of the q-th pattern. he vector w is composed of all weihts in the GNN. he cost function E(w ) is small if the trainin (and prunin) process performs well and lare if the it performs poorly. he cost function forms an error surface in a (n +)- dimensional space, where n is equal to the number of weihts in the GNN. In the next step this space has to be searched in order to reduce the cost function. A. evenber-marquardt Alorithm All Newton methods are based on the second-order aylor series expansion about the old weiht vector w : E(w + ) = E(w +Δw ) (5) = E(w )+ Δw + 2 Δw H Δw If a minimum on the error surface is found, the radient of the expansion (5) with respect to Δw is zero: E(w + ) = + H Δw = (6) Solvin (6) for Δw ives the Newton method Δw = H w + = w H (7) he vector H is nown as the Newton direction, which is a descent direction, if the Hessian matrix is positive definite. here are several difficulties with the direct H application of Newton s method. One problem is that the optimization step may move to a maximum or saddle point if the Hessian is not positive definite and the alorithm could become unstable. here are two possibilities to solve this problem. Either the alorithm uses a line search routine (e.. Quasi-Newton) or the alorithm uses a scalin factor (e.. M). he direct evaluation of the Hessian matrix is computationally demandin. Hence the Quasi-Newton approach (e.. BFGS formula) builds up an increasinly accurate term for the inverse Hessian matrix iteratively, usin first derivatives of the cost function only. he Gauss-Newton and M approach approximate the Hessian matrix by [5] H J (w ) J(w ) (8) and it can be shown that = J (w ) e(w ) (9) where J(w ) is the Jacobian matrix e (w ) e (w ) w w 2 e 2(w ) e 2(w ) J(w )= w w e Q(w ) e Q(w ) w w 2 e (w ) w n e 2(w ) w n e Q(w ) w n () which includes first derivatives only. n is the number of all weihts in the neural networ and Q is the number of time steps evaluated. With (7), (8) and (9) the Gauss-Newton method can be written as w + = w [ J (w ) J(w ) ] J (w ) e(w ) () Now the M method can be expressed with the scalin factor μ w + = w [ J (w ) J(w )+μ Ĩ ] J (w ) e(w ) (2) where Ĩ is the identity matrix. As the M alorithm is the best optimization method for small and moderate networs (up to a few hundred weihts), this alorithm is used for all simulations in this paper. M optimization normally is offline. his means that all trainin patterns have to be available before trainin is started. In order to et an online trainin alorithm we use a slidin time window that includes the information of the last Q time steps. With the last Q errors the Jacobian matrix J(w ) from equation () is calculated quasi-online. In every time step the oldest trainin pattern drops out of the time window and a new one (from the current time step) is added just lie a first in first out (FIFO) buffer. If the time window is lare enouh it can be assumed that the information content of the trainin data is constant. With this simple method we are able 354

4 to implement the M alorithm online. As this quasi-online optimization method wors surprisinly well it is not necessary to use a recurrent approach lie [5]. For the simulations in this paper the window size is set to Q = 5 (usin a samplin time of. sec.). B. Jacobian Calculations o create the Jacobian matrix, the derivatives of the errors have to be computed, see (). he GNN has feedbac elements and internal delays, so that the Jacobian cannot be calculated by the standard bacpropaation alorithm. here are two eneral approaches to calculate the Jacobian matrix for dynamic systems: By bacpropaation throuh time (BP) [8] or by real time recurrent learnin (RR) [9]. For Jacobian calculations the RR alorithm is more efficient than the BP alorithm []. Accordin to this the RR alorithm is used in this paper. herefore we mae use of the developed formulas of the layered diital dynamic networ. he interested reader is referred to [7]-[],[3] for further details. IV. OBS PRUNING IN GNNS he oal of OBS prunin is to set one of the GNN-weihts to zero while the cost function iven in (4) is minimized. his particular weiht is denoted as w,z. In the oriinal OBS alorithm proposed by Hassibi [6] all weihts in the model are evaluated by the saliency w 2,z S,z = 2 [ ] (3) H z,z he weiht with the smallest saliency is deleted. he contributed weiht update can be calculated by Δw = w,z [ H ] z,z i H z (4) where i z is the unit vector with only the z th element equal to. A. M-based OBS he OBS in (3) and (4) requires the complex calculation of the inverse Hessian. In subsection III-A the M optimization method made H an approximation of this calculation extended with the M-scalin factor μ = H ( J(w ) J(w )+μ Ĩ ) (5) Usin this scaled and approximated inverse Hessian matrix the OBS alorithm accordin to (3) and (4) can be rewritten H as w 2,z S,z = 2 (w [(J ) J(w )+μ Ĩ) (6) ] z,z with the optimum chane of the weihts Δw = w,z (J (w ) J(w )+μ Ĩ) i z (w [(J ) J(w )+μ Ĩ) (7) ] z,z hus the OBS approach is obtained with low computational cost. B. Effect of the scaled Hessian in OBS Now the question is: How does the M-scalin factor μ in (6) and (7) influence the OBS alorithm? First of all, let us have a loo at the trusted reion method overned by the M-scalin factor μ. For small values of μ the M optimization (2) wors as Newton method (7). In this case the second order aylor series expansion (5) (with and ) ives a ood approximation for the real cost function (4). H By contrast, lare values of μ lead to simple Gradient escent (G) optimization with step lenth μ, see (2). If so, the second order series expansion (with and ) is not suitable to approximate the real cost function. In M H the aylor series expansion provides a model for the real cost function, but the model is only trusted within some reion around the current search point w overned by the M-scalin factor μ. Next we consider the M-based prunin approach defined by (6) and (7). For small values of μ the expression J(w ) J(w )+μ Ĩ ives a ood approximation for the real Hessian matrix and we recover the oriinal OBS-formulas: μ : S,z S,z Δw Δw (8) More interestin is the case for lare values of μ. hen the second order series expansion (5) (with and ) is not suitable to approximate the real cost function (4). H hus the expression J(w ) J(w )+μ Ĩ does not represent the real Hessian and is not suitable for OBS-calculation. For lare values of μ the M-based OBS iven by (6) and (7) can be reduced to the form: μ : S,z μ 2 w2,z (9) Δw w,z i z (2) (9) and (2) comply with a very simple prunin concept (in the followin referred to as Manitude Prunin) [2]: It is supposed that small weihts in the neural networ are less important than lare weihts. he squared manitude of the weiht value is used as saliency, see (9). his prunin approach is not the best (even small weihts can play an important role in a neural networ). However, it is a ood alternative prunin method, because for lare values of μ we do not have Hessian information in order to apply the much more efficient OBS prunin. With (2) only the particular weiht with the smallest manitude is set to zero, no additional weiht update is done, the remainin weihts are ept constant. his feature is also desirable, since we do not have Hessian information for adequate weiht adaption. he effect of usin the scaled Hessian in OBS can be summarized as follows: For small values of μ the Hessian information is accurate and the standard OBS is applied. For lare M-scalin factors no suitable Hessian information is available and Manitude Prunin is obtained as an alternative to OBS. he M-based OBS switches smoothly between the two prunin methods OBS and Manitude Prunin. 355

5 C. Control on Prunin Success Every Δ p time steps one prunin step is executed. he time constant Δ p is called prunin interval. Every prunin step starts with the evaluation of the last prunin step. herefore the cost function value E p before prunin is stored in order to jude the success of the prunin operation. he parameter ΔE max defines the maximum relative error increase. his user defined parameter limits the error increase durin the prunin process. he last prunin step is canceled if the current cost function value E(w ) is too hih compared to the value before prunin (see Fi. 2): if E(w ) > ΔE max E p =ΔE max E(w Δp ) undo last prunin step (2) herefore also the GNN weihts have to be stored before prunin.. Adaptive Prunin Interval Usually the cost function increases after a prunin step.he parameter optimization procedure between the prunin steps tries to reduce the error. It can be observed that the deletion of one weiht in an oversized GNN does not have too much influence on the identification process. here are many redundant weihts available. By contrast, the smaller the networ, the more complex the retrainin process after a prunin step. For this reason the prunin operation can be improved, if the prunin interval Δ p is adapted online. It must be assured that the optimization alorithm is able to cope with the structural chanes in the GNN-model within the prunin interval Δ p. First of all we have to find a measure if the prunin interval Δ p is accurately chosen. herefore we recommend to averae over the last Δ p cost function values between the prunin steps (see Fi. 2 and 3): E mean = (E(w )+E(w )+...+ E(w ) Δp+) Δ p Cost Function E E p E(w ) Δ p Prunin Δ p E mean < ΔE max E p ΔE max E p E mean Prunin Iteration Prunin Fi. 2. Example of an adequate prunin interval with Δ p =. (22) his mean cost function indicates if the current prunin interval Δ p is suitable. If the mean error E mean is hih, i. e. the optimization alorithm cannot cope with the new optimization tas within the time Δ p, the prunin interval should be increased, see Fi. 3. If the mean error E mean is small, the prunin process could be sped up by decreasin the prunin interval. Fi. 2 depicts an example for an adequately chosen prunin interval. he prunin interval Δ p is adapted by the followin scalin alorithm: if E mean > ΔE max E p Δ p = ϑ Δ p else Δ p = ϑ Δ p (23) with the user defined scalin factor ϑ >. he followin identification examples in section V show the adaption of the prunin interval Δ p durin the prunin and parameter optimization process. Cost Function E E(w ) E p Δ p Prunin Fi. 3. Δ p E mean > ΔE max E p E mean ΔE max E p Prunin Prunin interval is too short Iteration Prunin V. IENIFICAION In this section two identification examples are presented usin the M-based prunin alorithm suested in (6) and (7) with adaptive prunin interval proposed in (22) and (23), Jacobian calculation with RR and the M alorithm (2). he first example to be identified is a simple linear P 2 system. his example is chosen to test the architecture selection ability of the suested prunin approach. his test is possible for linear systems because we now the required difference equation. If the prunin alorithm with adaptive prunin interval succeeds in deletin the riht weihts, the final GNN-model has to consist of the parameters of the difference equation. he second example is a more complex one. A nonlinear dynamic system has to be identified. A. Excitation Sinal In system identification it is a very important tas to use an appropriate excitation sinal. A nonlinear dynamic system requires that all combinations of frequencies and amplitudes (in the system s operatin rane) are represented in the sinal. In this paper an APRBS-sinal (Amplitude Modulated Pseudo Random Binary Sequence) is applied, uniformly distributed from - to + in amplitudes and from ms to 25ms in hold time. For more information the reader is referred to [4]. B. Identification of P 2 plant he simple P 2 -plant. s 2 +s+ is identified by a threelayer GNN (with three neurons in the hidden layers and d max =2 n =93). his networ architecture is obviously larer than necessary. If the continuous-time P 2 -plant is 356

6 discretized usin zero-order hold [7] with a sample time of ms the discrete model can be expressed as y() =.48 u( ) +.46 u( 2) +.8 y( ).948 y( 2) (24) he initial prunin interval is set to Δ p =5, the maximum error increase is ΔE max = and the scalin factor ϑ is defined to.. he prunin process starts after iteration Q = 5, when the slidin time window is filled up. Fi. Squared Error E(w ) n =93 n = Identification ime [sec] Fi. 4. Squared Error E(w ) and number of weihts n for P 2-identification example (at the beinnin: n =93, at the end: n =7) 4 shows the weiht reduction from n =93to n =7and the identification error, which is also reduced althouh the model is becomin smaller. After about 2 seconds a lot of prunin steps are canceled due to (2). able II summarizes ABE II EXAMPE OF AMINISRAION MARICES FOR A HREE-AYER GNN, 3 NEURONS IN HE HIEN AYERS AN d max =2 ayer Number of Weihts n Administration Matrices AI, () AI, (2), (), (2),2 (),2 (2),3 (),3 (2)Ab 2, () 2,2 () 2,2 (2) 2,3 () 2,3 ()Ab 2 3,2 () 3,3 () 3,3 (2)Ab 3 the administration matrices after the identification and prunin process. Most of the weihts are deleted. Alon with the weiht vector w =46 = [ ], (25) we are able to draw the final GNN-model, see Fi. 5. Each circle in the sinal flow raph includes a summin junction and an activation function. he input and the output of the model have two time delays. ue to the very small input weiht.27 all the tanh-activation functions can be nelected (only the linear reion near the oriin with slope is used) and p Fi. 5. z z z.85 z Identification result: GNN-model of P 2 the example we et the followin difference equation for the final GNNmodel: ŷ() =.48 u( ) +.46 u( 2) +.85 y( ).948 y( 2) (26) his equation loos similar to the real difference equation of the system (24). Fi. 6 depicts another P 2 -identification usin the same initial prunin interval Δ p =5but no adaption is conducted. he optimization alorithm cannot cope with the structural chanes and the identification error increases. Fi. 7 Squared Error E(w ) Identification ime [sec] Fi. 6. Constant prunin interval is too small: he Squared Error E(w ) increases. shows the output of the P 2 -plant y (solid ray line) and the output of the GNN-model ŷ (thic dashed blac line) for the first 3 seconds (sample time: ms). y and ŷ Identification ime [sec] Fi. 7. Outputs of the P 2-plant y (solid ray line) and of the GNN-model ŷ (thic dashed blac line) for the first 3 seconds. C. Identification of a nonlinear plant he first example is chosen to underpin the weiht reduction ability of the proposed prunin alorithm. In this subsection ŷ Number of Weihts n 357

7 a complex nonlinear dynamic system presented by Narendra [3] is considered: y( ) y( 2) y( 3) u( 2) [y( 3) ] + u( ) y() = +y 2 ( 2) + y 2 ( 3) For this identification example a three-layer GNN (with four neurons in the hidden layers and d max =3 n = 22) is used. he initial prunin interval is Δ p =, the maximum error increase is set to ΔE max =2and the scalin factor ϑ is defined to.. Fi. 8 depicts the weiht reduction from n = Squared Error E(w ) Identification ime [sec] Fi. 8. Squared Error E(w ) and number of weihts n for nonlinear identification example (at the beinnin: n = 22, at the end: n =57) 22 to n =57and the identification error for 2 seconds. he prunin process is stopped after 82 seconds due to the abrupt rise of the cost function. he last prunin step is undone. Fi. y, ŷ and p Identification ime [sec] Fi. 9. Output of the nonlinear plant y (solid ray line), output of GNNmodel ŷ (thic dashed blac line) and APRBS excitation sinal p (thin dashed blac line) for the first 3 seconds. 9 shows the output of the nonlinear plant y (solid ray line), the output of GNN-model ŷ (thic dashed blac line) and the APRBS excitation sinal (thin dashed blac line) durin the first 3 seconds (sample time: ms). VI. CONCUSION his paper uses the M-based OBS approach for system identification with GNNs. he advantae of this prunin method is that the OBS-method exploits Hessian information already calculated for M parameter optimization. he Mbased OBS switches smoothly between the two prunin methods OBS and Manitude Prunin overned by the M-scalin factor. o manae the prunin process in GNN administration matrices are introduced. hese matrices indicate which weihts still exist and which are already deleted. In every prunin step the success of the last prunin step is checed. If the increase 2 5 Number of Weihts n in the error is too hih, the last prunin step is canceled. For this revision the weihts have to be stored. he structure of the GNN is chaned drastically durin the prunin and identification process. So the influence of one prunin step to the identification system can be quite different. o uarantee that the parameter optimization alorithm has enouh time to cope with the structural chanes in the GNN-model, this paper suests an adaptive prunin interval. We defined the mean cost function value to measure the success of the current prunin interval. ependin on this value the prunin interval is adapted by a scalin alorithm. he proposed prunin alorithm is verified by one simple linear and one complex nonlinear dynamic system identification example. Networ prunin does not only wor for static neural networs. hese structure findin alorithms offer a very interestin application in system identification with dynamic neural networs. REFERENCES [] M. Atti,. Bourain, F. Alexandre, Optimal Brain Sureon Variants For Feature Selection, in IEEE Proc. of the Intern. Joint Conf. on Neural Networs, pp , 24. [2] C. M. Bishop, Neural Networs for Pattern Reconition. Oxford University Press, 995. [3] C. Endisch, C. Hacl and. Schröder, Optimal Brain Sureon For General ynamic Neural Networs, in ecture Notes in Artificial Intellience (NAI) 4874, Spriner-Verla Berlin, pp. 5-28, 27. [4] C. Endisch, P. Stolze, C. Hacl and. Schröder, Comments on Bacpropaation Alorithms for a Broad Class of ynamic Networs, IEEE rans. on Neural Networs, vol. 2, no. 3, pp , 29. [5] M. Haan, B. M. Mohammed, rainin Feedforward Networs with the Marquardt Alorithm, IEEE rans. on Neural Networs, vol. 5, no. 6, pp , 994. [6] B. Hassibi,. G. Stor, G. J. Wolff, Optimal Brain Sureon and General Networ Prunin, IEEE Intern. Conf. on Neural Networs, vol., pp , 993. [7] O. e Jesús, M. Haan, Bacpropaation Alorithms hrouh ime for a General Class of Recurrent Networ, IEEE Int. Joint Conf. Neural Networ, Washinton, pp , 2. [8] O. e Jesús, M. Haan, Forward Perturbation Alorithm For a General Class of Recurrent Networ, IEEE Int. Joint Conf. Neural Networ, Washinton, pp , 2. [9] O. e Jesús, rainin General ynamic Neural Networs, Ph.. dissertation, Olahoma State University, Stillwater, OK, 22. [] O. e Jesús, M. Haan, Bacpropaation Alorithms for a Broad Class of ynamic Networs, IEEE rans. on Neural Networs, vol. 8, no., pp. 4 27, 27. [] Y. e Cun, J. S. ener, S. A. Solla, Optimal Brain amae, in. S. ouretzy, Advances in Neural Information Processin Systems, Moran Kaufmann, pp , 99. [2] J. Men, Penalty OBS Scheme for Feedforward Neural Networ, Proceedins of the 7th IEEE Intern. Conf. on ools with Artificial Intellience (ICAI 5), pp , 25. [3] K. S. Narendra, K. Parthasarathy, Identification and Control of ynamical Systems Usin Neural Networs, IEEE rans. on Neural Networs, vol., no., pp. 4 27, 99. [4] O. Nelles, Nonlinear System Identification. Berlin Heidelber New Yor: Spriner-Verla, 2. [5].S.H. Nia, J. Sjöber, Efficient rainin of Neural Nets for Nonlinear Adaptive Filterin Usin a Recursive evenber-marquardt Alorithm, IEEE rans. on Sinal Processin, vol. 48, no. 7, pp , 2. [6] R. Reed, Prunin Alorithms A Survey, IEEE rans. on Neural Networs, vol. 4, no. 5, pp , 993. [7]. Schröder, Eletrische Antriebe - Reelun von Antriebssystemen. 2nd edn., Berlin Heidelber New Yor: Spriner-Verla, 2. [8] P.J. Werbos, Bacpropaation hrouh ime: What it is and how to do it, Proc. IEEE, vol. 78, no., pp , 99. [9] R.J. Williams,. Zipser, A earnin Alorithm for Continually Runnin Fully Recurrent Neural Networs, Neural Computin, vol., pp ,