Training Algorithm for Extra Reduced Size Lattice Ladder Multilayer Perceptrons

Transcription

1 Training Agorithm for Extra Reduced Size Lattice Ladder Mutiayer Perceptrons Daius Navakauskas Division of Automatic Contro Department of Eectrica Engineering Linköpings universitet, SE Linköping, Sweden WWW: E-mai: March 5, 2003 AUTOMATIC CONTROL COMMUNICATION SYSTEMS LINKÖPING Report no.: LiTH-ISY-R-2499 Submitted to journa Informatica ISSN Technica reports from the Contro & Communication group in Linköping are avaiabe at

2

3 Training Agorithm for Extra Reduced Size Lattice Ladder Mutiayer Perceptrons Daius Navakauskas Radioeectronics Department, Eectronics Facuty Vinius Gediminas Technica University Naugarduko 41, 2600 Vinius, Lithuania E-mai: March 5, 2003 Abstract A quick gradient training agorithm for a specific neura network structure caed an extra reduced size attice adder mutiayer perceptron is introduced. Presented derivation of the agorithm utiizes recenty found by author simpest way of exact computation of gradients for rotation parameters of attice adder fiter. Deveoped neura network training agorithm is optima in terms of minima number of constants, mutipication and addition operations, whie the reguarity of the structure is aso preserved. Keywords: attice adder fiter, attice adder mutiayer perceptron, adaptation, gradient adaptive attice agorithms. 1 Introduction During the ast decade a ot of new structures for artificia neura networks (ANN) were introduced fufiing the need to have modes for noninear processing of time-varying signas [6, 7, 16, 17]. One of many and perhaps the most straightforward way to insert tempora behaviour into ANNs is to use digita fiters in a pace of synapses of mutiayer perceptron (MLP). Foowing that way, a timedeay neura network [18], FIR/IIR MLPs [1, 20], gamma MLP [8], a cascade neura network [4] to name a few ANN architectures, were deveoped. A attice adder reaization of IIR fiters incorporated as MLP synapses forms a structure of attice adder mutiayer perceptron (LLMLP) firsty introduced by [2] and foowed by severa simpified versions proposed by the author [9, 11]. A LLMLP is an appeaing structure for advanced signa processing, however, even moderate impementation of its training hinders the fact that a ot of storage and computation power must be aocated [10]. Currenty with Automatic Contro Division at ISY at Linköping University. 1

4 We known neura network training agorithms such as backpropagation and its modifications, conjugate-gradients, Quasi-Newton, Levenberg-Marquardt, etc. or their adaptive counterparts ike tempora-backpropagation [19], IIR MLP training agorithm [3], recursive Levenberg-Marquardt [13], etc., essentiay are based on the use of gradients (aso caed sensitivity functions) partia derivatives of the cost function with respect to current weights. Here we are going to show how the exporation of specific to attice adder fiter (LLF) computation of gradients eads to efficient reaization of overa famiy of LLMLP training agorithms. Whie doing so, we present quickest in terms of number of constants, mutipication and addition operations training agorithm for an extra reduced size LLMLP (XLLMLP). 2 Towards simpified computations In order not to obscure main ideas, we wi first work ony with one LLF and afterwards in Section 4 generaize and utiize resuts in a deveopment of training agorithm for a specific XLLMLP structure. Athough, the fina LLF training agorithm is fairy simpe, many intermediate steps invoved in the derivation coud be confusing. Thus, here we wi present previous resuts on cacuation of LLF gradients, introduce the simpifications we have chosen, and ony in next Section 3 actuay derive a simpified expressions. Consider one attice adder fiter (see Figure 1 in next page) used in XLLMLP structure, when its computations are expressed by [ ] fj 1 (n) b j (n) [ cos Θj sin Θ j sin Θ j M s out (n) v j b j (n), j0 ][ ] fj (n), j 1,2,...,M; (1a) zb j 1 (n) (1b) with boundary conditions b 0 (n) f 0 (n); f M (n) s in (n). (1c) Here we used the foowing notations: s in (n) and s out (n) are signas at input and output of Mth order LLF, f j (n) and b j (n) are forward and backward signas foating in jth section of LLF, Θ j and v j are attice and adder coefficients correspondingy, whie z is a deay operator such that zb j (n) b j (n 1). It coud be shown (see, for exampe, [5]) that gradient expressions for the cacuation of LLF coefficients require M recursions, yieding a training agorithm with tota compexity proportiona to M 2. One possibe way to simpify the LLF gradients cacuation was presented in [15]. We assume that the concept of fowgraph transposition is aready known (if not consut, for exampe, [14, pages ]). Appying the fowgraph transposition rues to the LLF equations (1a) and (1b), we obtain the LLF transpose reaization. The resuting system gives rise to the foowing recurrent 2

5 repacemen s in (n) cos Θ M Lattice f 1 (n) cos Θ 1 f 0 (n) sinθ M sin Θ M sinθ 1 sin Θ 1 z z cos Θ M cos Θ 1 b M (n) b 1 (n) b 0 (n) v M v 1 v 0 Ladder s out (n) Figure 1: A attice adder fiter of order M. Input signa s in (n) is processed according to (1) using attice Θ x and adder v x coefficients and as a resut output signa s out (n) is obtained. Some forward f x (n) and backward b x (n) signas foating inside of LLF are shown whie by z we indicate a deay operator. reation [ ] [ gj (n) 1 t j 1 (n) z [ 0 + ] [ cos Θj sin Θ j sin Θ j ], j M,...,2,1, v j 1 ] [ ] gj 1 (n) t j (n) (2a) with boundary conditions t M (n) v M, g 0 (n) t 0 (n), g M (n) s out (n). (2b) Here by g j (n) and t j (n) we indicated forward and backward signas foating in jth section of transposed LLF. After simpe re-arrangements (see, for exampe, [10, pages 51 54]) it coud be shown that fitered regressor components aternativey coud be expressed as Θ j (n) f j(n)t j (n) b j (n)g j (n) e(n). (3) Here e(n) is a LLF error. The main idea given by J. R.-Fonoosa and E. Masgrau towards simpifying the gradient computations for attice parameters is to find a recurrence reation that reaizes the mapping [ ] fj 1 (n)t j 1 (n) b j 1 (n)g j 1 (n) [ ] fj (n)t j (n), (4) b j (n)g j (n) in such a way that a the necessary transfer functions may be obtained from one singe fiter resuting in an agorithm with tota compexity proportiona to M. 3

6 3 Derivation of simpified cacuation of LLF gradients There are in tota 14 possibe and impementabe ways of reaization of mapping (4). The optima way in a sense of minima number of constants, addition and deay operations invoved in the cacuations of the gradients for LLF, and aso reguarity of the regressor attice structure, was reported in [12]. It forms the basis for new training agorithm to be deveoped in next section. Thus, et us show in the step-by-step fashion the re-arrangements invoved in the derivation of optima reaization of mapping (4). Accordingy, we are seeking a simpe reaization for the foowing optima augmented mapping f j 1 (n)g j 1 (n) f j (n)g j (n) b j 1 (n)g j 1 (n) f j (n)t j (n) b j (n)g j (n) f j 1 (n)t j 1 (n), (5) b j (n)t j (n) b j 1 (n)t j 1 (n) whie the overa system describing singe section of regressor attice is expressed by [ ] [ ][ ] fj 1 (n)g j 1 (n) cos Θj sin Θ j f j (n)g j 1 (n) ; (6a) b j (n)g j 1 (n) sin Θ j zb j 1 (n)g j 1 (n) [ ] [ ][ ] fj 1 (n)t j 1 (n) cos Θj sin Θ j f j (n)t j 1 (n) ; (6b) b j (n)t j 1 (n) sin Θ j zb j 1 (n)t j 1 (n) [ ] [ ][ ][ ] fj (n)g j (n) 1 cos Θj sin Θ j fj (n)g j 1 (n) (6c) f j (n)t j 1 (n) z sin Θ j f j (n)t j (n) [ ] 0 + f v j (n); j 1 [ ] [ ][ ][ ] bj (n)g j (n) 1 cos Θj sin Θ j bj (n)g j 1 (n) (6d) b j (n)t j 1 (n) z sin Θ j b j (n)t j (n) [ ] 0 + b v j (n). j 1 In order to achieve mapping (5) two information fow directions in (6) must be reversed: now f j (n)g j 1 (n) must be computed based on f j 1 (n)g j 1 (n), and b j 1 (n) t j 1 (n) must be computed based on b j (n)t j 1 (n). For the first re-direction to be fufied we take (6a) and re-arrange it as foows [ ] [ ] fj (n)g j 1 (n) 1 0 b j (n)g j 1 (n) 0 z 1 1 sin Θ j sin Θ j 1 Simiary, taking (6b) and re-arranging we get 1 [ ] [ ] fj 1 (n)t j 1 (n) 1 0 b j 1 (n)t j 1 (n) 0 z 1 sin Θ j sin Θ j 1 [ ] fj 1 (n)g j 1 (n). (7a) b j 1 (n)g j 1 (n) [ ] fj (n)t j 1 (n). (7b) b j (n)t j 1 (n) 4

7 For the convenience we rewrite unchanged expressions (6c) and (6d) here [ ] [ fj (n)g j (n) 1 f j (n)t j 1 (n) z [ 0 + [ ] [ bj (n)g j (n) 1 b j (n)t j 1 (n) z [ 0 + ][ cos Θj sin Θ j sin Θ j ] f j (n); ][ cos Θj sin Θ j sin Θ j ] b j (n). v j 1 v j 1 ][ ] fj (n)g j 1 (n) f j (n)t j (n) ][ ] bj (n)g j 1 (n) b j (n)t j (n) (7c) (7d) Now, (7) describes a system that has no conficting information fow directions. However, there are two advance operations to be performed in (7a) and in (7b), making the system non-causa, hence un-impementabe. There is, however, more to this computation than first meets the eye. Notice first, the mapping (5) needs ony four expressions to be specified. Thus a simpification of (7) becomes pausibe. It coud be shown that (7) coud be simpified drasticay and finay expressed by f j (n)g j (n) b j (n)g j (n) f j 1 (n)t j 1 (n) z 0 b j 1 (n)t j 1 (n) sin Θ j I sin Θ j sin Θ j sin Θ j f j 1 (n)g j 1 (n) 0 0 z 0 0 b j 1 (n)g j 1 (n) f j (n)t j (n) + v 0 j 1 f j 1 (n) b j (n)t j (n) b j 1 (n) (8a) The simpified system of singe section of regressor attice is aready causa. Moreover, its impementation requires ony 2 deay operators, 3 constants (sinθ j, sin Θ j and v j 1 ) and 8 addition operations (more evident from pictoria representation of (8) that we skiped to save space). In order to finish the derivation, the boundary conditions when M such systems are cascaded must be considered. Based on (1c) and (2b) we get such new boundary conditions f 0 (n)g 0 (n) f 0 (n)t 0 (n); b 0 (n)g 0 (n) b 0 (n)t 0 (n); (8b) f M (n)t M (n) v M ; b M (n)t M (n) v M b M (n). 4 Simpified training agorithm for extra reduced size LLMLP One way of LLMLP structure reduction coud be achieved by restricting each neuron to have ony one output attice adder fiter whie connecting ayers through conventiona synaptic coefficients. It yieds an extra reduced size 5

8 s 1 1 (n) LLF 1 s 1(n) ŝ 1(n) Φ s 1(n) s 1 i (n) LLF i s i (n) ŝ h (n) Φ s h (n) s 1 N 1 (n) LLF N 1 s N 1 (n) ŝ N (n) Φ s N (n) {Θ,V } [N 1,M ] W [N 1,N ] Φ ( ) [N ] Figure 2: A ayer of XLLMLP. Input signas sx 1 (n) from a previous ayer are processed according to (9) firsty by a group of attice adder fiters LLF x (see Figure 1 for more detaied treatment), then by static synapses wx,x, finay by neuron activation functions Φ ( ) forming current ayer output signas s x(n). Emphasized intermediate signas: LLF output signas s x(n) and signas before activation functions ŝ x(n). For the purpose of cearness size of main matrices (shown in brackets) are reveaed at the bottom. attice adder mutiayer perceptron structure (XLLMLP) thats pictoria representation is given on next page in Figure 2 and definition foows. Definition 1 ([11]) A XLLMLP of size L ayers, {N 0, N 1,...,N L } nodes and fiter orders {M 1,M 2,...,M L } is defined by { N 1 s h(n) Φ i1 } M vij b ij(n), h 1,...,N, 1,...,L, (9a) wih j0 } {{ } } s i(n) {{ } ŝ h(n) when oca fow of information in the attice part of fiters is defined by [ ] [ ] [ ] f i,j 1 (n) cos Θ b ij (n) ij sin Θ ij f ij (n) sinθ ij cos Θ ij zb i,j 1 (n), j 1,2,...,M, (9b) with initia and boundary conditions b i0(n) f i0(n); f i,m+1(n) s 1 i (n). (9c) 6

9 Here s h (n) is an output signa of the output neuron; s i (n) is an output signa of the fiter that is connected to i input neuron; wih represents singe (static) weight connecting two neurons in a ayer; Θ ij and v ij are weights of fiter s attice and adder parts correspondingy; b ij (n) and f ij (n) are forward and backward prediction errors of the fiter; i, j, h and index inputs, fiter coefficients, outputs and ayers respectivey. Let us consider cacuation of sensitivity functions using backpropagation agorithm for XLLMLP th hidden ayer neurons. It coud be shown [11] that sensitivity functions for XLLMLP coud be expressed by w ih(n) s i(n) δ h(n); N vij(n) b ij(n) δh(n); h1 Θ ij(n) vir(n) M r0 Θ ij N b ir(n) wihδ h(n). h1 (10a) (10b) (10c) Note, that these expressions are simiar to standard LLF gradient expressions in a way that here we used additiona indexes to state LLF position in the whoe XLLMLP architecture (being precise, we showed expressions for sensitivity functions for jth coefficients of LLF connecting ith input neuron with hth output neuron in th ayer of LLMLP) and repaced output error e(n) term by the generaized oca instantaneous error, i.e., δh (n) E(n)/ ŝ h (n), that coud be expicity expressed by e h (n)φ L{ ŝ h (n)}, L, δh(n) Φ { ŝ h (n)} M v +1 hj γ+1 hj (n)n w +1 hp δ+1 p (n), L, j0 p1 (10d) with γ +1 hj [ ϕ +1 h,j 1 (n) γ +1 hj (n) (n) and its companion ϕ+1 hj (n) by ] [ ][ cos Θ +1 hj sin Θ +1 hj sin Θ +1 hj cos Θ +1 hj ϕ +1 hj (n) zγ +1 h,j 1 (n) ]. (10e) Aiming to present simpified training of XLLMLP we repace (10c) with Θ ij(n) f ij (n)t ij (n) b ij (n)g ij (n) cos Θ ij }{{} Θ ij(n) wihδ h(n), (11) N h1 7

10 where order recursion computation is done in a way of (8) by f ij (n)g ij (n) b ij (n)g ij (n) f i,j 1 (n)t i,j 1 (n) b i,j 1 (n)t i,j 1 (n) 1 z sin Θ ij sin Θ ij 0 sinθ ij z 0 sinθ ij z sinθ ij 0 z z sinθ ij 0 z sin Θ ij sin Θ ij 1 fi,j 1 (n)g i,j 1 (n) 0 b i,j 1 (n)g i,j 1 (n) + v 0 i,j 1, f ij (n)t ij (n) b ij (n)t ij (n) f i,j 1 (n) b i,j 1 (n) (12a) with boundary conditions: f i0(n)g i0(n) f i0(n)t i0(n); b i0(n)g i0(n) b i0(n)t i0(n); (12b) f im (n)t im (n) v im ; b im (n)t im (n) b im (n)v im. (12c) Fu XLLMLP training agorithm is presented at the end of the artice. Let us here ony summarize main steps of it: 1. XLLMLP reca. Given input pattern s 0 i (n) is presented to the input ayer (see ine 1 of the agorithm). Afterwards it is processed through XLLMLP in a ayer by ayer fashion (ines 2 14): first by corresponding attice adder fiters (ines 3 9), then by static synapses (ines 10 13). In that way XLLMLP output pattern s L h (n) is obtained. 2. Node errors. Using rezuts of previous cacuations, errors of output ayer neurons are determined (ine 15). Then, again in a ayer by ayer fashion however this time in the backward direction these errors are processed through compete XLLMLP (ines 16 23). Foowing that way errors of hidden ayer neurons δh (n) are cacuated. 3. Gradient terms. Simpified cacuation of gradient terms for the attice weight updates is done as in (12). Let us here be more specific. Working in a ayer by ayer fashion and taking fiter one by one (ines 24 38; note however that processing order here is insignificant), agorithm proceeds as foow: at ine 25 initia conditions are assigned as in (12c); in ines two ower equations from (12a) are evauated for a sections of the fiter; at ine 30 eft boundary conditions are fufied as in (12b); for a sections of the fiter (ines 31 37) by ines remaining upper equations from (12a) and in ine 36 aso Θ ij (n) from (11) are evauated. 4. Weight updates. Dynamic synapses (LLFs) are updated at ines 41 and 42 reaizing expressions (10b) and (11), whie static ones at ine 44 reaizing (10a) and taking into account their previous vaues, training parameter µ, node errors and gradient terms. 5. Stabiity test. New parameter vaues of attice fiters are checked if they satisfy stabiity requirement (see ines 46 48), if some of them do not satisfy od parameter vaues are substituted at ine 47. 8

11 5 Concusions In this paper we deat with the probem of computationa efficiency of attice adder mutiayer perceptrons training agorithms that are based on the computation of gradients, for exampe, backpropagation, conjugate-gradient or Levenberg- Marquardt. Here we expored cacuations that are most computationay demanding and specific to attice-adder fiter computation of gradients for attice (rotation) parameters. The optima way in a sense of minima number of constants, addition and deay operations invoved in aforementioned computations for singe attice-adder fiter, and aso reguarity of the regressor structure, was found recenty by the author. Based on it quick training agorithm for the extra reduced size attice adder mutiayer perceptron was here derived. Not surprisingy, presented agorithm requires approximatey M times (where M is the order of fiter) ess computations whie it foows exact gradient path (because of the fact that derivations do not invoved any approximations) when coefficients of fiters are assumed to be stationary. More importanty, incorporated in the agorithm impementation of a singe section of regressor attice wi require ony 2 deay eements, 3 constants and 8 addition operations. Experimenta study of the proposed agorithm was not considered because of the fact that computations of gradients are exact and possibe comparison inherenty wi be biased depending on the chosen way of impementation. Acknowedgements Research was supported by The Roya Swedish Academy of Sciences and The Swedish Institute New Visby project Ref. No. 2473/2002(381/T81). References [1] A. D. Back and A. C. Tsoi. FIR and IIR synapses, a new neura network architecture for time series modeing. Neura Computation, 3: , [2] A. D. Back and A. C. Tsoi. An adaptive attice architecture for dynamic mutiayer perceptrons. Neura Computation, 4: , [3] A. D. Back and A. C. Tsoi. A simpified gradient agorithm for IIR synapse mutiayer perceptrons. Neura Computation, 5: , [4] A. D. Back and A. C. Tsoi. A cascade neura network mode with noninear poes and zeros. In Proceedings of 1996 Internationa Conference on Neura Information Processing, voume 1, pages IEEE Press, [5] S. Haykin. Adaptive Fiter Theory. Prentice-Ha Internationa, Inc., 3rd edition, [6] S. Haykin. Neura Networks: A Comprehensive Foundation. Prentice-Ha, Upper Sadde River, NJ, 2nd edition,

12 [7] A. Juditsky, H. Hjamarsson, A. Benveniste, B. Deyon, L. Ljung, J. Sjöberg, and Q. Zhang. Noninear back-box modeing in system identification: Mathematica foundations. Automatica, 31: , [8] S. Lawrence, A. D. Back, A. C. Tsoi, and C. L. Gies. The Gamma MLP - using mutipe tempora resoutions for improved cassification. In Neura Networks for Signa Processing 7, pages IEEE Press, [9] D. Navakauskas. A reduced size attice-adder neura network. In Signa Processing Society Workshop on Neura Networks for Signa Processing, pages , Cambridge, Engand, IEEE Press. [10] D. Navakauskas. Artificia Neura Network for the Restoration of Noise Distorted Songs Audio Records. Doctora dissertation, Vinius Gediminas Technica University, No. 434, September [11] D. Navakauskas. Reducing impementation input of attice-adder mutiayer perceptrons. In Proceedings of the 15th European Conference on Circuit Theory and Design, voume 3, pages , Espoo, Finand, Hesinki University of Technoogy. [12] D. Navakauskas. Speeding up the training of attice-adder mutiayer perceptrons. Technica Report LiTH-ISY-R-2417, Department of Eectrica Engineering, Linköping University, SE Linköping, Sweden, [13] L. S. H. Ngia and J. Sjöberg. Efficient training of neura nets for noninear adaptive fitering using a recursive Levenberg-Marquardt agorithm. IEEE Transactions on Signa Processing, 48(7): , [14] P. A. Regaia. Adaptive IIR Fitering in Signa Processing and Contro. Marce Dekker, Inc., [15] J. A. Rodriguez-Fonoosa and E. Masgrau. Simpified gradient cacuation in adaptive IIR attice fiters. IEEE Transactions on Signa Processing, 39(7): , [16] J. Sjöberg, Q. Zhang, L. Ljung, A. Benveniste, B. Deyon, P.-Y. Gorennec, H. Hjamarsson, and A. Juditsky. Noninear back-box modeing in system identification: A unified overview. Automatica, 31: , [17] A. C. Tsoi and A. D. Back. Discrete time recurrent neura network architectures: A unifying review. Neurocomputing, 15: , [18] A. Waibe, T. Hanazawa, G. Hinton, K. Shikano, and K. Lang. Phoneme recognition using time-deay neura networks. IEEE Transactions on Acoustics, Speech, and Signa Processing, 37(3): , [19] E. A. Wan. Tempora backpropagation for FIR neura networks. In Proceedings of Internationa Joint Conference on Neura Networks, pages IEEE Press, [20] E. A. Wan. Finite Impuse Response Neura Networks with Appications in the Time Series Prediction. Doctora dissertation, Department of Eectrica Engineering, Stanford University, USA, November

13 XLLMLP Training Agorithm Avaiabe at time n. New input pattern: s 0 i (n), i 1,2,...,N0 ; Desired output pattern: d L h (n), h 1,2,...,NL ; Static coefficients: wih (n), h 1,2,...,N ; Ladder coefficients: vij (n), j 0,1,...,M ; Lattice coefficients: Θ ij (n), j 1,2,...,M ; Backward & forward signas: b ij (n), f ij (n), j 0,1,...,M ; Gradients w.r.t. attice weights: Θ ij (n 1), j 1,2,...,M ; Regressor attices signas: bgij (n 1), ft ij (n 1), fg ij, bt ij, j 0,...,M ; Tempora variabe: temp; Backward & forward gradients:ϕ ij (n 1), γ ij (n 1), j 0,2,...,M. 1. XLLMLP Reca. 1 Let f 1 i,m 1 (n) s 0 i (n). 2 for 1,2,...,L do 3 for i 1,2,...,N 1 do 4 for j M,...,2,1 do [ ] [ f i,j 1 (n) cos Θ 5 b ij (n) ij (n) sin Θ ij (n) ][ sinθ ij (n) cos Θ ij (n) 6 end for j 7 b i0 (n) f i0 (n); 8 s M+1 i (n) vij (n)b ij (n). 9 end for i j0 10 for h 1,2,...,N do 11 ŝ h N+1 (n) w ih s i (n); i1 12 s h (n) Φ { ŝ h (n)}. 13 end for h 14 end for b i,j 1 fij (n) ]. (n 1) 2. Node Errors. 15 e L h (n) d h(n) s L h (n), h 1,2,...,NL. 16 for L 1,L 2,...,1 and h 1,2,...,N do 17 Let γ h,m (n) for j M,...,2,1 do [ ] [ ϕ h,j 1 (n) cos Θ 19 γhj (n) hj (n) sin Θ hj (n) ][ sin Θ hj (n) cos Θ hj (n) 20 end for j ϕ hj (n) ]. (n 1) γ h,j 1 11

14 21 ϕ h0 (n) γ h0 (n). e h (n)φ L{ ŝ h (n)}, L, 22 δh (n) Φ { ŝ h (n)} M v +1 hj (n)n w +1 hp δ+1 p (n), L. 23 end for h, 3. Gradient Terms. j0 hj γ+1 p1 24 for 1,2,...,L and i 1,2,...,N 1 do 25 Let bt i,m v i,m (n)b ij (n), ft i,m s 1 i (n)v i,m (n). 26 for j M,...,2,1 do 27 bt i,j 1 bt ij [ ft ij (n) + bg i,j 1 (n 1)] sinθ ij (n) +v i,j 1 (n)b i,j 1 (n); 28 ft i,j 1 (n) ft i,j (n 1) + v i,j 1 (n)f i,j 1 (n); 29 end for j 30 Let temp bt i0, fg i0 ft i0 (n 1). 31 for j 1,2,...,M do 32 ft i,j 1 (n) ft ij (n 1) [ fg i,j 1 + bt ij] sinθ ij (n); 33 fg ij fg i,j 1 + [ bg i,j 1 (n 1) + ft ij (n)] sin Θ ij (n); 34 bgi,j 1 (n) temp; 35 temp bg i,j 1 (n 1) + [ fg i,j 1 + bt ij] sin Θ ij (n); 36 Θ ij (n) [ ft ij (n) temp] /cos Θ ij (n). 37 end for j 38 end for i, 4. Weight Updates. 39 for 1,2,...,L and i 1,2,...,N 1 do 40 for j 1,2,...,M do 41 vij (n + 1) v ij (n) + N µb ij (n) δh (n); h1 42 Θ ij (n + 1) Θ ij (n) + µ Θ N ij (n) wih δ h (n). 43 end for j h1 44 for h 1,2,...,N do w ih (n + 1) w ih (n) + µδ h (n) s i (n). 45 end for i, 5. Stabiity Test. 46 for 1,2,...,L and i 1,2,...,N 1 and j 1,2,...,M do 47 if Θ ij (n + 1) > π/2 then Θ ij (n + 1) Θ ij (n). 48 end for j,i, 12